Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4640768837


##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +121,120 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle, d);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {
+                java.util.Objects.checkIndex(index, plan.size());
+                MongoRangePlanner.checkCancelled(session, 
request.getScanContext());
+                TMongoScanDesc range = d.deepCopy();
+                TMongoSplit split = new TMongoSplit();
+                split.setStrategy(plan.family().isEmpty() ? "single" : 
"range");
+                split.setField(plan.field());
+                split.setTypeFamily(plan.family());
+                split.setIndex(index);
+                split.setRequested(plan.requested());
+                split.setAdmitted(plan.admitted());
+                split.setActual(plan.size());
+                split.setReason(plan.reason());
+                if (!plan.family().isEmpty()) {
+                    range.setVersion(2);
+                    split.setFilterBson(plan.partitionBytes(index));
+                }
+                range.setSplit(split);
+                if (snapshot != null) {
+                    range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+                }
+                if (plan.descriptorBudget() > 0 && descriptorSize > 
plan.descriptorBudget()) {

Review Comment:
   Agreed. A real scan node owns its provider and supplies its final request 
once, so this post-publication growth guard was unreachable. Removed it and the 
unused per-plan budget wrapper. Final predicates/projections still participate 
in shared count/byte admission before publication. Replaced the artificial 
reused-provider rejection test with independent scan-node admission, including 
a 60K-character literal. Fixed in `54ff217bdb74`; all 164 Mongo unit tests pass.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -132,19 +256,47 @@ public Map<String, String> 
getScanNodeProperties(ConnectorSession session, Conne
         Map<String, String> result = new LinkedHashMap<>();
         var policies = session.getSessionProperties();
         String concern = 
properties.resolveReadConcern(policies.getOrDefault("mongodb_read_concern", 
"inherit"));
-        String preference = properties.resolveReadPreference(
-                policies.getOrDefault("mongodb_read_preference", "inherit"));
-        var snapshot = MongoSnapshot.prepare(session, properties, table, 
client, concern, preference,
-                1000L * bounded(policies, "query_timeout", 300, 1, 
Integer.MAX_VALUE / 1000));
+        var pushed = predicate(session, table, filter);
+        var pruned = projection(session, table, columns);
         result.put("read_concern", concern);
-        result.put("snapshot", snapshot == null ? "none" : 
snapshot.getTimestampSeconds()
-                + ":" + snapshot.getTimestampIncrement());
         result.put("object_kind", table.getObjectKind());
         result.put("schema_source", table.getSchema().getSource());
         result.put("schema_version", table.getSchema().getVersion());
-        result.put("predicate_pushdown", predicate(session, table, 
filter).kind().name());
-        result.put("projection_pushdown", projection(session, table, 
columns).isEmpty() ? "false" : "true");
-        return result;
+        result.put("predicate_pushdown", pushed.kind().name());
+        result.put("projection_pushdown", pruned.isEmpty() ? "false" : "true");
+        // initSchemaParams reads properties BEFORE WHERE/projection are 
attached. Source probes
+        // and byte admission here would reject a normal large predicate and 
ignore KILL QUERY.
+        // Resolve diagnostics lazily from the final cancellable plan, without 
doing any source I/O.
+        return new AbstractMap<>() {

Review Comment:
   Checked both the engine call chain and a running baseline FE. EXPLAIN 
finalizes the scan through FileScanNode/getSplits -> planScan before rendering; 
baseline output already shows actual 8/8/8 range plans and 8/8/1 
insufficient_boundaries. The eager predicate/projection flags and repeated 
lazy-map reads were real issues. In `54ff217bdb74`, diagnostics come from the 
final request, and appendExplainInfo takes one entrySet snapshot per render. 
The lazy wrapper remains because the engine caches it during initialization. 
Baseline tests reproduce the stale flags and mixed-generation reads; the fixes 
pass. Retained SQL EXPLAIN assertions and hinted read-policy checks also pass 
on the installed candidate.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,370 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that 
plan or its failure. */
+final class MongoRangePlanner {
+    static final int SAMPLE_ROWS = 4096;
+    static final int PROBE_TIMEOUT_MS = 2000;

Review Comment:
   Documented the fixed optional-probe budget and its trade-off in 
`54ff217bdb74`. Index discovery, stats, sampling and probe-slot waiting share 
two seconds; statement/transport limits may shorten it. Increasing 
query_timeout cannot raise it. Large-source sampling may exhaust it: auto falls 
back before publication, whereas explicit range reports probe_budget. Single 
avoids the probe. Keeping a bounded optional planning cost preserves the 
existing shared-controls design without adding another user knob.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,370 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that 
plan or its failure. */
+final class MongoRangePlanner {
+    static final int SAMPLE_ROWS = 4096;
+    static final int PROBE_TIMEOUT_MS = 2000;
+    private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+    private final Supplier<MongoCatalogClient> client;
+
+    MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+        this.client = client;
+    }
+
+    MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table, 
TMongoSnapshot snapshot,
+            String preference, ConnectorScanContext context, long 
descriptorBytes) {
+        checkCancelled(session, context);
+        var options = session.getSessionProperties();
+        String strategy = options.getOrDefault("mongodb_split_strategy", 
"single");
+        if (!Set.of("single", "auto", "range").contains(strategy)) {
+            throw new IllegalArgumentException("Invalid 
mongodb_split_strategy");
+        }
+        if ("single".equals(strategy)) {
+            return MongoSplitPlan.single(1, 1, "single_strategy");
+        }
+        String field = options.getOrDefault("mongodb_split_field", "");
+        boolean custom = !field.isEmpty();
+        if (!custom) {
+            field = "_id";
+        }
+        if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() || 
field.length() > 256) {
+            throw new IllegalArgumentException("MongoDB split field must be an 
unambiguous top-level field");
+        }
+        String selectedField = field;
+        Budget budget = 
session.getStatementScope().computeIfAbsent("mongodb.split.budget", 
Budget::new);
+        String tableKey = tableKey(session, table);
+        // Count fallback scans too: a view must not make a global custom 
field look unambiguous.
+        if (custom) {
+            budget.customField(tableKey);
+        }
+        Memo memo = session.getStatementScope().computeIfAbsent(tableKey, 
Memo::new);
+        return memo.get(session, context, () -> load(session, table, snapshot, 
preference, context,
+                strategy, selectedField, descriptorBytes, budget));
+    }
+
+    Optional<MongoSplitPlan> planned(ConnectorSession session, 
MongoTableHandle table) {
+        Memo memo = session.getStatementScope().getIfPresent(tableKey(session, 
table));
+        return memo == null ? Optional.empty() : memo.completed();
+    }
+
+    private String tableKey(ConnectorSession session, MongoTableHandle table) {
+        // One provider can see several namespaces; never reuse another 
collection's bounds/admission.
+        return memoKey + ":" + session.getCatalogId() + ":" + 
table.getSchema().getVersion();
+    }
+
+    private MongoSplitPlan load(ConnectorSession session, MongoTableHandle 
table, TMongoSnapshot snapshot,
+            String preference, ConnectorScanContext context, String strategy, 
String field,
+            long descriptorBytes, Budget budget) {
+        int maximum = splitLimit(session);
+        targetBytes(session, -1);
+        int requested = Math.min(8, maximum);
+        int descriptorMaximum = (int) Math.min(maximum, (8L * 1024 * 1024) / 
descriptorBytes);
+        if (descriptorMaximum == 0) {
+            throw new DorisConnectorException("MongoDB split descriptor 
exceeds planning byte budget");
+        }
+        String source = snapshot == null ? "catalog:" + session.getCatalogId()
+                : snapshot.getSourceKind() + ":" + snapshot.getSourceId();
+        if (!"collection".equals(table.getObjectKind())) {
+            return fallback(strategy, requested, "unsupported_object", source, 
descriptorBytes, budget);
+        }
+        if (snapshot == null || session.getStatementScope() == 
ConnectorStatementScope.NONE) {
+            return fallback(strategy, requested, "snapshot_required", source, 
descriptorBytes, budget);
+        }
+        if (descriptorMaximum == 1) {
+            return single(requested, maximum == 1 ? "split_limit" : 
"descriptor_budget",
+                    source, descriptorBytes, budget);
+        }
+        long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+                Math.min(PROBE_TIMEOUT_MS, 
MongoSnapshot.remainingMillis(session)));
+        MongoSplitProbe probe;
+        try {
+            budget.acquireProbe(session, context, deadline);
+            try {
+                checkCancelled(session, context);
+                probe = client.get().probeSplits(table, field, preference, 
snapshot, SAMPLE_ROWS,
+                        remaining(session, context, deadline), () -> 
checkCancelled(session, context));
+                budget.sampled(probe.sampleBytes());
+            } finally {
+                budget.probes.release();
+            }
+        } catch (PlanningLimit limit) {
+            checkCancelled(session, context);
+            MongoSnapshot.remainingMillis(session);
+            // Only optional pre-publication tuning can fall back. 
Cancellation, authorization,
+            // history and source errors must never be converted into a 
successful full scan.
+            return fallback(strategy, requested, "probe_budget", source, 
descriptorBytes, budget);
+        }
+        checkCancelled(session, context);
+        MongoSnapshot.remainingMillis(session);
+        long targetBytes = targetBytes(session, probe.logicalBytes());
+        long estimate = probe.logicalBytes() < 0 ? 8
+                : Math.max(1, probe.logicalBytes() / targetBytes + 
(probe.logicalBytes() % targetBytes == 0 ? 0 : 1));
+        requested = (int) Math.min(maximum, estimate);
+        if (!probe.indexed()) {
+            return fallback(strategy, requested, "missing_index", source, 
descriptorBytes, budget);
+        }
+        String family = family(probe.samples());
+        if (family.isEmpty()) {
+            return probe.samples().isEmpty()
+                    ? single(requested, "empty_sample", source, 
descriptorBytes, budget)
+                    : fallback(strategy, requested, "unsupported_type", 
source, descriptorBytes, budget);
+        }
+        if ("auto".equals(strategy) && probe.logicalBytes() < 0) {
+            return single(requested, "unknown_complement_cost", source, 
descriptorBytes, budget);
+        }
+        // Division avoids overflow when a valid shared split-size setting is 
near Long.MAX_VALUE.
+        if ("auto".equals(strategy) && probe.logicalBytes() / targetBytes < 2) 
{

Review Comment:
   Documented the custom-field amplification beside the auto behavior in 
`54ff217bdb74`: when typed indexed ranges collectively examine about N 
documents, the mandatory complement adds approximately another N; 
mixed/multikey data or different source plans can cost more, and the complement 
may dominate elapsed time. Auto uses logical-byte/split-size heuristics rather 
than source I/O costs. The docs recommend the default _id ranges where 
possible; degenerate custom plans still use one complete cursor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to