Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4610810903
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle,
+ d.isSetFilterBson() ? d.getFilterBson().length : 0,
+ d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
Review Comment:
Pool checkout now tries once and yields NEED_SEND_AGAIN to the scanner
scheduler when busy. The same reader retains its split and original deadline;
it neither closes the pending stream nor starts a replacement scan. Scheduler
handling retains an accounted empty block and re-admits the scanner through its
normal queue, allowing a queued lease holder to resume on the sole worker.
Coverage includes native deadline/pending-reader UTs, a one-worker
scanner-chain UT, and a source-backed 32769-row case with pool=1,
workload-group remote min/max=1 and two verified competing scanners in both
synchronous and streaming modes. The old binary timed out on that case. Fixed
in 5ab8d8a18a1caef2a4cb401490be378c197be738.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +120,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle,
+ d.isSetFilterBson() ? d.getFilterBson().length : 0,
+ d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
+ java.util.Objects.checkIndex(index, plan.size());
+ request.getScanContext().checkCancelled();
+ TMongoScanDesc range = d.deepCopy();
+ TMongoSplit split = new TMongoSplit();
+ split.setStrategy(plan.family().isEmpty() ? "single" :
"range");
+ split.setField(plan.field());
+ split.setTypeFamily(plan.family());
+ split.setIndex(index);
+ split.setRequested(plan.requested());
+ split.setAdmitted(plan.admitted());
+ split.setActual(plan.size());
+ split.setReason(plan.reason());
+ if (!plan.family().isEmpty()) {
+ range.setVersion(2);
+
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+ }
+ range.setSplit(split);
+ if (snapshot != null) {
+ range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+ }
+ if (plan.descriptorBudget() > 0 && descriptorSize >
plan.descriptorBudget()) {
+ try {
+ // EXPLAIN may freeze a plan before a later caller
supplies a larger candidate
+ // or projection. Never let that RPC silently exceed
the frozen reservation.
+ int actual = new org.apache.thrift.TSerializer(
+ new
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+ if (actual > plan.descriptorBudget()) {
Review Comment:
Properties initialization no longer selects a snapshot, probes MongoDB, or
reserves descriptor bytes. Admission runs only from the final scan request,
after its normal predicate/projection are attached. Cached EXPLAIN properties
read the completed plan lazily without more source I/O; the guard still rejects
genuine descriptor growth after publication.
Added FE and retained SQL regressions for the normal 60000-character
supported predicate. The old SQL path failed with descriptor growth; both final
planning modes now return the expected count/sum. Fixed in
5ab8d8a18a1caef2a4cb401490be378c197be738.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,312 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+ private final Supplier<MongoCatalogClient> client;
+
+ MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+ this.client = client;
+ }
+
+ MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table,
TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, long
descriptorBytes) {
+ context.checkCancelled();
+ var options = session.getSessionProperties();
+ String strategy = options.getOrDefault("mongodb_split_strategy",
"single");
+ if (!Set.of("single", "auto", "range").contains(strategy)) {
+ throw new IllegalArgumentException("Invalid
mongodb_split_strategy");
+ }
+ int requested = number(options, "mongodb_split_count", 0, 0, 256);
+ int maximum = number(options, "mongodb_split_max_count", 256, 1, 256);
+ int rows = number(options, "mongodb_split_sample_rows", 4096, 1, 4096);
+ int timeout = number(options, "mongodb_split_plan_timeout_ms", 2000,
1, 300000);
+ int targetBytes = number(options, "mongodb_split_target_bytes",
268435456, 1, Integer.MAX_VALUE);
+ if (requested > maximum) {
+ throw new IllegalArgumentException("mongodb_split_count exceeds
mongodb_split_max_count");
+ }
+ if ("single".equals(strategy) || requested == 1) {
+ return MongoSplitPlan.single(Math.max(1, requested), 1,
+ requested == 1 ? "requested_one" : "single_strategy");
+ }
+ String field = options.getOrDefault("mongodb_split_field", "");
+ boolean custom = !field.isEmpty();
+ if (!custom) {
+ field = "_id";
+ }
+ if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() ||
field.length() > 256) {
+ throw new IllegalArgumentException("MongoDB split field must be an
unambiguous top-level field");
+ }
+ String selectedField = field;
+ Budget budget =
session.getStatementScope().computeIfAbsent("mongodb.split.budget",
Budget::new);
+ // A provider may be reused: records and another_records must not
share boundaries or admission.
+ String tableKey = memoKey + ":" + session.getCatalogId() + ":" +
table.getSchema().getVersion();
+ // Count fallback scans too: an auto view scan must not make a global
custom field appear
+ // unambiguous when another MongoDB relation also participates in the
statement.
+ if (custom) {
+ budget.customField(tableKey);
+ }
+ if (!"collection".equals(table.getObjectKind())) {
+ return fallback(strategy, requested, "unsupported_object");
+ }
+ if (snapshot == null || session.getStatementScope() ==
ConnectorStatementScope.NONE) {
+ return fallback(strategy, requested, "snapshot_required");
+ }
+ Memo memo = session.getStatementScope().computeIfAbsent(tableKey,
Memo::new);
+ return memo.get(session, context, () -> load(session, table, snapshot,
preference, context, strategy,
+ selectedField, requested, maximum, rows, timeout, targetBytes,
descriptorBytes, budget));
+ }
+
+ private MongoSplitPlan load(ConnectorSession session, MongoTableHandle
table, TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, String strategy,
String field, int requested,
+ int maximum, int rows, int timeout, int targetBytes, long
descriptorBytes, Budget budget) {
+ int descriptorMaximum = (int) Math.min(maximum, (8L * 1024 * 1024) /
descriptorBytes);
+ if (descriptorMaximum == 0) {
+ throw new DorisConnectorException("MongoDB split descriptor
exceeds planning byte budget");
+ }
+ String source = snapshot.getSourceKind() + ":" +
snapshot.getSourceId();
+ if (descriptorMaximum == 1) {
+ budget.admit(source, 1, descriptorBytes);
+ return MongoSplitPlan.single(Math.max(1, requested), 1,
"descriptor_budget")
+ .withDescriptorBudget(descriptorBytes);
+ }
+ long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+ Math.min(timeout, MongoSnapshot.remainingMillis(session)));
+ MongoSplitProbe probe;
+ try {
+ // Waiting behind two active probes consumes the same optional
budget as source I/O.
+ // Keep acquisition inside the fallback boundary, but never
release a slot we did not acquire.
+ budget.acquireProbe(session, context, deadline);
+ try {
+ context.checkCancelled();
+ probe = client.get().probeSplits(table, field, preference,
snapshot, rows,
+ remaining(session, context, deadline), context);
+ budget.sampled(probe.sampleBytes());
+ } finally {
+ budget.probes.release();
+ }
+ } catch (PlanningLimit limit) {
+ context.checkCancelled();
+ MongoSnapshot.remainingMillis(session);
+ // Only this optional, pre-publication tuning budget may trigger
auto fallback.
+ // Source authorization/identity/history errors and query
cancellation remain failures.
+ return fallback(strategy, requested, "probe_budget");
+ }
+ context.checkCancelled();
+ MongoSnapshot.remainingMillis(session);
+ if (!probe.indexed()) {
+ return fallback(strategy, requested, "missing_index");
+ }
+ String family = family(probe.samples());
+ if (family.isEmpty()) {
+ return probe.samples().isEmpty() ?
MongoSplitPlan.single(Math.max(1, requested), 1, "empty_sample")
Review Comment:
Every successful opted-in single fallback now goes through the shared
count/byte admission helper, records its descriptor reservation and is
memoized. This includes empty samples, missing indexes,
unsupported/unknown-cost auto paths and probe-budget fallbacks. The default
single strategy retains its prior behavior.
New UTs cover 257 empty scans, aggregate fallback descriptor bytes, and
non-snapshot auto admission. A real 257-branch empty-collection query also
fails at the source partition budget. Fixed in
5ab8d8a18a1caef2a4cb401490be378c197be738.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -136,14 +254,26 @@ public Map<String, String>
getScanNodeProperties(ConnectorSession session, Conne
policies.getOrDefault("mongodb_read_preference", "inherit"));
var snapshot = MongoSnapshot.prepare(session, properties, table,
client, concern, preference,
1000L * bounded(policies, "query_timeout", 300, 1,
Integer.MAX_VALUE / 1000));
+ var pushed = predicate(session, table, filter);
+ var pruned = projection(session, table, columns);
+ MongoSplitPlan plan = rangePlanner.plan(session, table, snapshot,
preference, new ConnectorScanContext(),
Review Comment:
All source work has moved out of properties into final planning. Probe
checks now observe both the scan context and the captured query task-group
lifetime, including synchronous requests whose context is unregistered. KILL
cannot publish the cancelled plan or advance another batch after cancellation
is observed; in-flight calls remain bounded by driver transport/deadline
settings.
A real cancellation test also exposed driver5.14 attaching unsupported
snapshot readConcern to killCursors. Fixed-width sampling now requests one
bounded singleBatch, leaving no sampling cursor behind. UTs cover task-group
closure before another advance; real sync/stream KILL tests assert the wire
flag, no sampling getMore and zero retained cursors. Fixed in
5ab8d8a18a1caef2a4cb401490be378c197be738.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]