github-actions[bot] commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4609330871
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle,
+ d.isSetFilterBson() ? d.getFilterBson().length : 0,
+ d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
Review Comment:
[P2] Keep pool waiters from blocking sibling range scanners
With a workload group whose remote scanner min/max are both 1,
`mongodb.pool_max_clients=1`, and two file scanners (e.g.
`max_file_scanners_concurrency=2,min_file_scanners_concurrency=2,enable_adaptive_scan=false`),
this new multi-range path can stall one query until timeout. Scanner A leases
the client and yields after a non-EOF block, retaining the lease. The
ThreadPool scheduler already queued the next context; scanner B can then occupy
the sole worker inside `NativeMongoStream::open`'s try-pop/sleep loop. A cannot
resume to release the client. The file-scanner cap uses the global worker count
rather than the workload group's cap, so it does not prevent this ordering.
Please make checkout cooperative or otherwise ensure lease holders can resume
before dispatching competing range streams. The new pool fixture uses the
single strategy and misses this case.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -136,14 +254,26 @@ public Map<String, String>
getScanNodeProperties(ConnectorSession session, Conne
policies.getOrDefault("mongodb_read_preference", "inherit"));
var snapshot = MongoSnapshot.prepare(session, properties, table,
client, concern, preference,
1000L * bounded(policies, "query_timeout", 300, 1,
Integer.MAX_VALUE / 1000));
+ var pushed = predicate(session, table, filter);
+ var pruned = projection(session, table, columns);
+ MongoSplitPlan plan = rangePlanner.plan(session, table, snapshot,
preference, new ConnectorScanContext(),
Review Comment:
[P2] Connect properties-phase probes to query cancellation
The engine normally calls `getScanNodeProperties` during scan
initialization, so this fresh context drives the actual first range probe. It
is never registered with the statement's cancellation machinery. `KILL QUERY`
cancels registered split contexts/task groups but does not interrupt this
synchronous metadata call; after a delayed listIndexes/sample batch returns,
`checkProbe` still sees an uncancelled context and can issue further getMore
commands. With `mongodb_split_plan_timeout_ms=300000`, cancelled queries can
keep doing FE/source work until that deadline. This is separate from the
documented limitation of interrupting one in-flight C-driver connection call.
Please use the statement's cancellation signal here or defer probing to the
phase that supplies an engine-owned scan context; cover cancellation after the
first FE sample batch.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,312 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+ private final Supplier<MongoCatalogClient> client;
+
+ MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+ this.client = client;
+ }
+
+ MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table,
TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, long
descriptorBytes) {
+ context.checkCancelled();
+ var options = session.getSessionProperties();
+ String strategy = options.getOrDefault("mongodb_split_strategy",
"single");
+ if (!Set.of("single", "auto", "range").contains(strategy)) {
+ throw new IllegalArgumentException("Invalid
mongodb_split_strategy");
+ }
+ int requested = number(options, "mongodb_split_count", 0, 0, 256);
+ int maximum = number(options, "mongodb_split_max_count", 256, 1, 256);
+ int rows = number(options, "mongodb_split_sample_rows", 4096, 1, 4096);
+ int timeout = number(options, "mongodb_split_plan_timeout_ms", 2000,
1, 300000);
+ int targetBytes = number(options, "mongodb_split_target_bytes",
268435456, 1, Integer.MAX_VALUE);
+ if (requested > maximum) {
+ throw new IllegalArgumentException("mongodb_split_count exceeds
mongodb_split_max_count");
+ }
+ if ("single".equals(strategy) || requested == 1) {
+ return MongoSplitPlan.single(Math.max(1, requested), 1,
+ requested == 1 ? "requested_one" : "single_strategy");
+ }
+ String field = options.getOrDefault("mongodb_split_field", "");
+ boolean custom = !field.isEmpty();
+ if (!custom) {
+ field = "_id";
+ }
+ if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() ||
field.length() > 256) {
+ throw new IllegalArgumentException("MongoDB split field must be an
unambiguous top-level field");
+ }
+ String selectedField = field;
+ Budget budget =
session.getStatementScope().computeIfAbsent("mongodb.split.budget",
Budget::new);
+ // A provider may be reused: records and another_records must not
share boundaries or admission.
+ String tableKey = memoKey + ":" + session.getCatalogId() + ":" +
table.getSchema().getVersion();
+ // Count fallback scans too: an auto view scan must not make a global
custom field appear
+ // unambiguous when another MongoDB relation also participates in the
statement.
+ if (custom) {
+ budget.customField(tableKey);
+ }
+ if (!"collection".equals(table.getObjectKind())) {
+ return fallback(strategy, requested, "unsupported_object");
+ }
+ if (snapshot == null || session.getStatementScope() ==
ConnectorStatementScope.NONE) {
+ return fallback(strategy, requested, "snapshot_required");
+ }
+ Memo memo = session.getStatementScope().computeIfAbsent(tableKey,
Memo::new);
+ return memo.get(session, context, () -> load(session, table, snapshot,
preference, context, strategy,
+ selectedField, requested, maximum, rows, timeout, targetBytes,
descriptorBytes, budget));
+ }
+
+ private MongoSplitPlan load(ConnectorSession session, MongoTableHandle
table, TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, String strategy,
String field, int requested,
+ int maximum, int rows, int timeout, int targetBytes, long
descriptorBytes, Budget budget) {
+ int descriptorMaximum = (int) Math.min(maximum, (8L * 1024 * 1024) /
descriptorBytes);
+ if (descriptorMaximum == 0) {
+ throw new DorisConnectorException("MongoDB split descriptor
exceeds planning byte budget");
+ }
+ String source = snapshot.getSourceKind() + ":" +
snapshot.getSourceId();
+ if (descriptorMaximum == 1) {
+ budget.admit(source, 1, descriptorBytes);
+ return MongoSplitPlan.single(Math.max(1, requested), 1,
"descriptor_budget")
+ .withDescriptorBudget(descriptorBytes);
+ }
+ long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+ Math.min(timeout, MongoSnapshot.remainingMillis(session)));
+ MongoSplitProbe probe;
+ try {
+ // Waiting behind two active probes consumes the same optional
budget as source I/O.
+ // Keep acquisition inside the fallback boundary, but never
release a slot we did not acquire.
+ budget.acquireProbe(session, context, deadline);
+ try {
+ context.checkCancelled();
+ probe = client.get().probeSplits(table, field, preference,
snapshot, rows,
+ remaining(session, context, deadline), context);
+ budget.sampled(probe.sampleBytes());
+ } finally {
+ budget.probes.release();
+ }
+ } catch (PlanningLimit limit) {
+ context.checkCancelled();
+ MongoSnapshot.remainingMillis(session);
+ // Only this optional, pre-publication tuning budget may trigger
auto fallback.
+ // Source authorization/identity/history errors and query
cancellation remain failures.
+ return fallback(strategy, requested, "probe_budget");
+ }
+ context.checkCancelled();
+ MongoSnapshot.remainingMillis(session);
+ if (!probe.indexed()) {
+ return fallback(strategy, requested, "missing_index");
+ }
+ String family = family(probe.samples());
+ if (family.isEmpty()) {
+ return probe.samples().isEmpty() ?
MongoSplitPlan.single(Math.max(1, requested), 1, "empty_sample")
Review Comment:
[P2] Admit fallback descriptors against the shared planning budgets
This `empty_sample` return, and the auto fallbacks above/below it, occur
before `budget.admit` and return a plan with `descriptorBudget=0`. They still
publish a full descriptor and source scan, so opted-in statements evade the D08
source/statement partition and byte limits. For example, a UNION ALL of 257
scans of the same indexed empty collection with snapshot/range/count=8 produces
257 `empty_sample` single descriptors on one pinned source despite the
256-partition cap. Repeated schemas/candidates also escape the 32/64 MiB
accounting and post-freeze size guard. Please reserve one partition and its
descriptor bytes for successful fallbacks inside the opted-in planner, while
keeping the default `single` strategy exemption.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle,
+ d.isSetFilterBson() ? d.getFilterBson().length : 0,
+ d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
+ java.util.Objects.checkIndex(index, plan.size());
+ request.getScanContext().checkCancelled();
+ TMongoScanDesc range = d.deepCopy();
+ TMongoSplit split = new TMongoSplit();
+ split.setStrategy(plan.family().isEmpty() ? "single" :
"range");
+ split.setField(plan.field());
+ split.setTypeFamily(plan.family());
+ split.setIndex(index);
+ split.setRequested(plan.requested());
+ split.setAdmitted(plan.admitted());
+ split.setActual(plan.size());
+ split.setReason(plan.reason());
+ if (!plan.family().isEmpty()) {
+ range.setVersion(2);
+
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+ }
+ range.setSplit(split);
+ if (snapshot != null) {
+ range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+ }
+ if (plan.descriptorBudget() > 0 && descriptorSize >
plan.descriptorBudget()) {
+ try {
+ // EXPLAIN may freeze a plan before a later caller
supplies a larger candidate
+ // or projection. Never let that RPC silently exceed
the frozen reservation.
+ int actual = new org.apache.thrift.TSerializer(
+ new
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+ if (actual > plan.descriptorBudget()) {
Review Comment:
[P2] Reserve descriptor bytes after normal predicates are attached
Normal SQL planning reaches this guard without metadata changing. For
`Filter(text = '<60000-character literal>') -> FileScan(records)` with a STRING
binding, `mongodb.type_mismatch='null'`, snapshot concern and range count 8,
`PhysicalPlanTranslator` first calls the child scan's `init()`.
`initSchemaParams -> getPathPartitionKeys -> getScanNodeProperties` freezes the
memo's byte reservation before the filter is attached. Final planning adds the
supported BSON candidate, and its actual size exceeds that initial reservation,
raising `MongoDB descriptor grew after split plan freeze` even though eight
such descriptors are far below 8 MiB. Please defer admission until the final
candidate/projection are available, or update the reservation safely before
first publication while retaining T and boundaries. The existing
metadata-growth unit test asserts this rejection but reproduces routine engine
staging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]