Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4610810903


##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle,
+                d.isSetFilterBson() ? d.getFilterBson().length : 0,
+                d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {

Review Comment:
   Pool checkout now tries once and yields NEED_SEND_AGAIN to the scanner 
scheduler when busy. The same reader retains its split and original deadline; 
it neither closes the pending stream nor starts a replacement scan. Scheduler 
handling retains an accounted empty block and re-admits the scanner through its 
normal queue, allowing a queued lease holder to resume on the sole worker.
   
   Coverage includes native deadline/pending-reader UTs, a one-worker 
scanner-chain UT, and a source-backed 32769-row case with pool=1, 
workload-group remote min/max=1 and two verified competing scanners in both 
synchronous and streaming modes. The old binary timed out on that case. Fixed 
in 5ab8d8a18a1caef2a4cb401490be378c197be738.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle,
+                d.isSetFilterBson() ? d.getFilterBson().length : 0,
+                d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {
+                java.util.Objects.checkIndex(index, plan.size());
+                request.getScanContext().checkCancelled();
+                TMongoScanDesc range = d.deepCopy();
+                TMongoSplit split = new TMongoSplit();
+                split.setStrategy(plan.family().isEmpty() ? "single" : 
"range");
+                split.setField(plan.field());
+                split.setTypeFamily(plan.family());
+                split.setIndex(index);
+                split.setRequested(plan.requested());
+                split.setAdmitted(plan.admitted());
+                split.setActual(plan.size());
+                split.setReason(plan.reason());
+                if (!plan.family().isEmpty()) {
+                    range.setVersion(2);
+                    
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+                }
+                range.setSplit(split);
+                if (snapshot != null) {
+                    range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+                }
+                if (plan.descriptorBudget() > 0 && descriptorSize > 
plan.descriptorBudget()) {
+                    try {
+                        // EXPLAIN may freeze a plan before a later caller 
supplies a larger candidate
+                        // or projection. Never let that RPC silently exceed 
the frozen reservation.
+                        int actual = new org.apache.thrift.TSerializer(
+                                new 
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+                        if (actual > plan.descriptorBudget()) {

Review Comment:
   Properties initialization no longer selects a snapshot, probes MongoDB, or 
reserves descriptor bytes. Admission runs only from the final scan request, 
after its normal predicate/projection are attached. Cached EXPLAIN properties 
read the completed plan lazily without more source I/O; the guard still rejects 
genuine descriptor growth after publication.
   
   Added FE and retained SQL regressions for the normal 60000-character 
supported predicate. The old SQL path failed with descriptor growth; both final 
planning modes now return the expected count/sum. Fixed in 
5ab8d8a18a1caef2a4cb401490be378c197be738.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,312 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that 
plan or its failure. */
+final class MongoRangePlanner {
+    private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+    private final Supplier<MongoCatalogClient> client;
+
+    MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+        this.client = client;
+    }
+
+    MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table, 
TMongoSnapshot snapshot,
+            String preference, ConnectorScanContext context, long 
descriptorBytes) {
+        context.checkCancelled();
+        var options = session.getSessionProperties();
+        String strategy = options.getOrDefault("mongodb_split_strategy", 
"single");
+        if (!Set.of("single", "auto", "range").contains(strategy)) {
+            throw new IllegalArgumentException("Invalid 
mongodb_split_strategy");
+        }
+        int requested = number(options, "mongodb_split_count", 0, 0, 256);
+        int maximum = number(options, "mongodb_split_max_count", 256, 1, 256);
+        int rows = number(options, "mongodb_split_sample_rows", 4096, 1, 4096);
+        int timeout = number(options, "mongodb_split_plan_timeout_ms", 2000, 
1, 300000);
+        int targetBytes = number(options, "mongodb_split_target_bytes", 
268435456, 1, Integer.MAX_VALUE);
+        if (requested > maximum) {
+            throw new IllegalArgumentException("mongodb_split_count exceeds 
mongodb_split_max_count");
+        }
+        if ("single".equals(strategy) || requested == 1) {
+            return MongoSplitPlan.single(Math.max(1, requested), 1,
+                    requested == 1 ? "requested_one" : "single_strategy");
+        }
+        String field = options.getOrDefault("mongodb_split_field", "");
+        boolean custom = !field.isEmpty();
+        if (!custom) {
+            field = "_id";
+        }
+        if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() || 
field.length() > 256) {
+            throw new IllegalArgumentException("MongoDB split field must be an 
unambiguous top-level field");
+        }
+        String selectedField = field;
+        Budget budget = 
session.getStatementScope().computeIfAbsent("mongodb.split.budget", 
Budget::new);
+        // A provider may be reused: records and another_records must not 
share boundaries or admission.
+        String tableKey = memoKey + ":" + session.getCatalogId() + ":" + 
table.getSchema().getVersion();
+        // Count fallback scans too: an auto view scan must not make a global 
custom field appear
+        // unambiguous when another MongoDB relation also participates in the 
statement.
+        if (custom) {
+            budget.customField(tableKey);
+        }
+        if (!"collection".equals(table.getObjectKind())) {
+            return fallback(strategy, requested, "unsupported_object");
+        }
+        if (snapshot == null || session.getStatementScope() == 
ConnectorStatementScope.NONE) {
+            return fallback(strategy, requested, "snapshot_required");
+        }
+        Memo memo = session.getStatementScope().computeIfAbsent(tableKey, 
Memo::new);
+        return memo.get(session, context, () -> load(session, table, snapshot, 
preference, context, strategy,
+                selectedField, requested, maximum, rows, timeout, targetBytes, 
descriptorBytes, budget));
+    }
+
+    private MongoSplitPlan load(ConnectorSession session, MongoTableHandle 
table, TMongoSnapshot snapshot,
+            String preference, ConnectorScanContext context, String strategy, 
String field, int requested,
+            int maximum, int rows, int timeout, int targetBytes, long 
descriptorBytes, Budget budget) {
+        int descriptorMaximum = (int) Math.min(maximum, (8L * 1024 * 1024) / 
descriptorBytes);
+        if (descriptorMaximum == 0) {
+            throw new DorisConnectorException("MongoDB split descriptor 
exceeds planning byte budget");
+        }
+        String source = snapshot.getSourceKind() + ":" + 
snapshot.getSourceId();
+        if (descriptorMaximum == 1) {
+            budget.admit(source, 1, descriptorBytes);
+            return MongoSplitPlan.single(Math.max(1, requested), 1, 
"descriptor_budget")
+                    .withDescriptorBudget(descriptorBytes);
+        }
+        long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+                Math.min(timeout, MongoSnapshot.remainingMillis(session)));
+        MongoSplitProbe probe;
+        try {
+            // Waiting behind two active probes consumes the same optional 
budget as source I/O.
+            // Keep acquisition inside the fallback boundary, but never 
release a slot we did not acquire.
+            budget.acquireProbe(session, context, deadline);
+            try {
+                context.checkCancelled();
+                probe = client.get().probeSplits(table, field, preference, 
snapshot, rows,
+                        remaining(session, context, deadline), context);
+                budget.sampled(probe.sampleBytes());
+            } finally {
+                budget.probes.release();
+            }
+        } catch (PlanningLimit limit) {
+            context.checkCancelled();
+            MongoSnapshot.remainingMillis(session);
+            // Only this optional, pre-publication tuning budget may trigger 
auto fallback.
+            // Source authorization/identity/history errors and query 
cancellation remain failures.
+            return fallback(strategy, requested, "probe_budget");
+        }
+        context.checkCancelled();
+        MongoSnapshot.remainingMillis(session);
+        if (!probe.indexed()) {
+            return fallback(strategy, requested, "missing_index");
+        }
+        String family = family(probe.samples());
+        if (family.isEmpty()) {
+            return probe.samples().isEmpty() ? 
MongoSplitPlan.single(Math.max(1, requested), 1, "empty_sample")

Review Comment:
   Every successful opted-in single fallback now goes through the shared 
count/byte admission helper, records its descriptor reservation and is 
memoized. This includes empty samples, missing indexes, 
unsupported/unknown-cost auto paths and probe-budget fallbacks. The default 
single strategy retains its prior behavior.
   
   New UTs cover 257 empty scans, aggregate fallback descriptor bytes, and 
non-snapshot auto admission. A real 257-branch empty-collection query also 
fails at the source partition budget. Fixed in 
5ab8d8a18a1caef2a4cb401490be378c197be738.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -136,14 +254,26 @@ public Map<String, String> 
getScanNodeProperties(ConnectorSession session, Conne
                 policies.getOrDefault("mongodb_read_preference", "inherit"));
         var snapshot = MongoSnapshot.prepare(session, properties, table, 
client, concern, preference,
                 1000L * bounded(policies, "query_timeout", 300, 1, 
Integer.MAX_VALUE / 1000));
+        var pushed = predicate(session, table, filter);
+        var pruned = projection(session, table, columns);
+        MongoSplitPlan plan = rangePlanner.plan(session, table, snapshot, 
preference, new ConnectorScanContext(),

Review Comment:
   All source work has moved out of properties into final planning. Probe 
checks now observe both the scan context and the captured query task-group 
lifetime, including synchronous requests whose context is unregistered. KILL 
cannot publish the cancelled plan or advance another batch after cancellation 
is observed; in-flight calls remain bounded by driver transport/deadline 
settings.
   
   A real cancellation test also exposed driver5.14 attaching unsupported 
snapshot readConcern to killCursors. Fixed-width sampling now requests one 
bounded singleBatch, leaving no sampling cursor behind. UTs cover task-group 
closure before another advance; real sync/stream KILL tests assert the wire 
flag, no sampling getMore and zero retained cursors. Fixed in 
5ab8d8a18a1caef2a4cb401490be378c197be738.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to