github-actions[bot] commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4609330871


##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle,
+                d.isSetFilterBson() ? d.getFilterBson().length : 0,
+                d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {

Review Comment:
   [P2] Keep pool waiters from blocking sibling range scanners
   
   With a workload group whose remote scanner min/max are both 1, 
`mongodb.pool_max_clients=1`, and two file scanners (e.g. 
`max_file_scanners_concurrency=2,min_file_scanners_concurrency=2,enable_adaptive_scan=false`),
 this new multi-range path can stall one query until timeout. Scanner A leases 
the client and yields after a non-EOF block, retaining the lease. The 
ThreadPool scheduler already queued the next context; scanner B can then occupy 
the sole worker inside `NativeMongoStream::open`'s try-pop/sleep loop. A cannot 
resume to release the client. The file-scanner cap uses the global worker count 
rather than the workload group's cap, so it does not prevent this ordering. 
Please make checkout cooperative or otherwise ensure lease holders can resume 
before dispatching competing range streams. The new pool fixture uses the 
single strategy and misses this case.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -136,14 +254,26 @@ public Map<String, String> 
getScanNodeProperties(ConnectorSession session, Conne
                 policies.getOrDefault("mongodb_read_preference", "inherit"));
         var snapshot = MongoSnapshot.prepare(session, properties, table, 
client, concern, preference,
                 1000L * bounded(policies, "query_timeout", 300, 1, 
Integer.MAX_VALUE / 1000));
+        var pushed = predicate(session, table, filter);
+        var pruned = projection(session, table, columns);
+        MongoSplitPlan plan = rangePlanner.plan(session, table, snapshot, 
preference, new ConnectorScanContext(),

Review Comment:
   [P2] Connect properties-phase probes to query cancellation
   
   The engine normally calls `getScanNodeProperties` during scan 
initialization, so this fresh context drives the actual first range probe. It 
is never registered with the statement's cancellation machinery. `KILL QUERY` 
cancels registered split contexts/task groups but does not interrupt this 
synchronous metadata call; after a delayed listIndexes/sample batch returns, 
`checkProbe` still sees an uncancelled context and can issue further getMore 
commands. With `mongodb_split_plan_timeout_ms=300000`, cancelled queries can 
keep doing FE/source work until that deadline. This is separate from the 
documented limitation of interrupting one in-flight C-driver connection call. 
Please use the statement's cancellation signal here or defer probing to the 
phase that supplies an engine-owned scan context; cover cancellation after the 
first FE sample batch.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,312 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that 
plan or its failure. */
+final class MongoRangePlanner {
+    private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+    private final Supplier<MongoCatalogClient> client;
+
+    MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+        this.client = client;
+    }
+
+    MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table, 
TMongoSnapshot snapshot,
+            String preference, ConnectorScanContext context, long 
descriptorBytes) {
+        context.checkCancelled();
+        var options = session.getSessionProperties();
+        String strategy = options.getOrDefault("mongodb_split_strategy", 
"single");
+        if (!Set.of("single", "auto", "range").contains(strategy)) {
+            throw new IllegalArgumentException("Invalid 
mongodb_split_strategy");
+        }
+        int requested = number(options, "mongodb_split_count", 0, 0, 256);
+        int maximum = number(options, "mongodb_split_max_count", 256, 1, 256);
+        int rows = number(options, "mongodb_split_sample_rows", 4096, 1, 4096);
+        int timeout = number(options, "mongodb_split_plan_timeout_ms", 2000, 
1, 300000);
+        int targetBytes = number(options, "mongodb_split_target_bytes", 
268435456, 1, Integer.MAX_VALUE);
+        if (requested > maximum) {
+            throw new IllegalArgumentException("mongodb_split_count exceeds 
mongodb_split_max_count");
+        }
+        if ("single".equals(strategy) || requested == 1) {
+            return MongoSplitPlan.single(Math.max(1, requested), 1,
+                    requested == 1 ? "requested_one" : "single_strategy");
+        }
+        String field = options.getOrDefault("mongodb_split_field", "");
+        boolean custom = !field.isEmpty();
+        if (!custom) {
+            field = "_id";
+        }
+        if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() || 
field.length() > 256) {
+            throw new IllegalArgumentException("MongoDB split field must be an 
unambiguous top-level field");
+        }
+        String selectedField = field;
+        Budget budget = 
session.getStatementScope().computeIfAbsent("mongodb.split.budget", 
Budget::new);
+        // A provider may be reused: records and another_records must not 
share boundaries or admission.
+        String tableKey = memoKey + ":" + session.getCatalogId() + ":" + 
table.getSchema().getVersion();
+        // Count fallback scans too: an auto view scan must not make a global 
custom field appear
+        // unambiguous when another MongoDB relation also participates in the 
statement.
+        if (custom) {
+            budget.customField(tableKey);
+        }
+        if (!"collection".equals(table.getObjectKind())) {
+            return fallback(strategy, requested, "unsupported_object");
+        }
+        if (snapshot == null || session.getStatementScope() == 
ConnectorStatementScope.NONE) {
+            return fallback(strategy, requested, "snapshot_required");
+        }
+        Memo memo = session.getStatementScope().computeIfAbsent(tableKey, 
Memo::new);
+        return memo.get(session, context, () -> load(session, table, snapshot, 
preference, context, strategy,
+                selectedField, requested, maximum, rows, timeout, targetBytes, 
descriptorBytes, budget));
+    }
+
+    private MongoSplitPlan load(ConnectorSession session, MongoTableHandle 
table, TMongoSnapshot snapshot,
+            String preference, ConnectorScanContext context, String strategy, 
String field, int requested,
+            int maximum, int rows, int timeout, int targetBytes, long 
descriptorBytes, Budget budget) {
+        int descriptorMaximum = (int) Math.min(maximum, (8L * 1024 * 1024) / 
descriptorBytes);
+        if (descriptorMaximum == 0) {
+            throw new DorisConnectorException("MongoDB split descriptor 
exceeds planning byte budget");
+        }
+        String source = snapshot.getSourceKind() + ":" + 
snapshot.getSourceId();
+        if (descriptorMaximum == 1) {
+            budget.admit(source, 1, descriptorBytes);
+            return MongoSplitPlan.single(Math.max(1, requested), 1, 
"descriptor_budget")
+                    .withDescriptorBudget(descriptorBytes);
+        }
+        long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+                Math.min(timeout, MongoSnapshot.remainingMillis(session)));
+        MongoSplitProbe probe;
+        try {
+            // Waiting behind two active probes consumes the same optional 
budget as source I/O.
+            // Keep acquisition inside the fallback boundary, but never 
release a slot we did not acquire.
+            budget.acquireProbe(session, context, deadline);
+            try {
+                context.checkCancelled();
+                probe = client.get().probeSplits(table, field, preference, 
snapshot, rows,
+                        remaining(session, context, deadline), context);
+                budget.sampled(probe.sampleBytes());
+            } finally {
+                budget.probes.release();
+            }
+        } catch (PlanningLimit limit) {
+            context.checkCancelled();
+            MongoSnapshot.remainingMillis(session);
+            // Only this optional, pre-publication tuning budget may trigger 
auto fallback.
+            // Source authorization/identity/history errors and query 
cancellation remain failures.
+            return fallback(strategy, requested, "probe_budget");
+        }
+        context.checkCancelled();
+        MongoSnapshot.remainingMillis(session);
+        if (!probe.indexed()) {
+            return fallback(strategy, requested, "missing_index");
+        }
+        String family = family(probe.samples());
+        if (family.isEmpty()) {
+            return probe.samples().isEmpty() ? 
MongoSplitPlan.single(Math.max(1, requested), 1, "empty_sample")

Review Comment:
   [P2] Admit fallback descriptors against the shared planning budgets
   
   This `empty_sample` return, and the auto fallbacks above/below it, occur 
before `budget.admit` and return a plan with `descriptorBudget=0`. They still 
publish a full descriptor and source scan, so opted-in statements evade the D08 
source/statement partition and byte limits. For example, a UNION ALL of 257 
scans of the same indexed empty collection with snapshot/range/count=8 produces 
257 `empty_sample` single descriptors on one pinned source despite the 
256-partition cap. Repeated schemas/candidates also escape the 32/64 MiB 
accounting and post-freeze size guard. Please reserve one partition and its 
descriptor bytes for successful fallbacks inside the opted-in planner, while 
keeping the default `single` strategy exemption.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle,
+                d.isSetFilterBson() ? d.getFilterBson().length : 0,
+                d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {
+                java.util.Objects.checkIndex(index, plan.size());
+                request.getScanContext().checkCancelled();
+                TMongoScanDesc range = d.deepCopy();
+                TMongoSplit split = new TMongoSplit();
+                split.setStrategy(plan.family().isEmpty() ? "single" : 
"range");
+                split.setField(plan.field());
+                split.setTypeFamily(plan.family());
+                split.setIndex(index);
+                split.setRequested(plan.requested());
+                split.setAdmitted(plan.admitted());
+                split.setActual(plan.size());
+                split.setReason(plan.reason());
+                if (!plan.family().isEmpty()) {
+                    range.setVersion(2);
+                    
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+                }
+                range.setSplit(split);
+                if (snapshot != null) {
+                    range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+                }
+                if (plan.descriptorBudget() > 0 && descriptorSize > 
plan.descriptorBudget()) {
+                    try {
+                        // EXPLAIN may freeze a plan before a later caller 
supplies a larger candidate
+                        // or projection. Never let that RPC silently exceed 
the frozen reservation.
+                        int actual = new org.apache.thrift.TSerializer(
+                                new 
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+                        if (actual > plan.descriptorBudget()) {

Review Comment:
   [P2] Reserve descriptor bytes after normal predicates are attached
   
   Normal SQL planning reaches this guard without metadata changing. For 
`Filter(text = '<60000-character literal>') -> FileScan(records)` with a STRING 
binding, `mongodb.type_mismatch='null'`, snapshot concern and range count 8, 
`PhysicalPlanTranslator` first calls the child scan's `init()`. 
`initSchemaParams -> getPathPartitionKeys -> getScanNodeProperties` freezes the 
memo's byte reservation before the filter is attached. Final planning adds the 
supported BSON candidate, and its actual size exceeds that initial reservation, 
raising `MongoDB descriptor grew after split plan freeze` even though eight 
such descriptors are far below 8 MiB. Please defer admission until the final 
candidate/projection are available, or update the reservation safely before 
first publication while retaining T and boundaries. The existing 
metadata-growth unit test asserts this rejection but reproduces routine engine 
staging.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to