Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4642209841
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoDriverCatalogClient.java:
##########
@@ -177,6 +183,131 @@ public BsonTimestamp selectSnapshot(String database,
String collection, String p
}
}
+ @Override
+ public MongoSplitProbe probeSplits(MongoTableHandle table, String field,
String preference,
+ TMongoSnapshot snapshot, int rows, long timeoutMs, Runnable
checkCancelled) {
+ long deadline = System.nanoTime() +
TimeUnit.MILLISECONDS.toNanos(timeoutMs);
+ String index = null;
+ long bytes = 0;
+ try {
+ var collection = probeDatabase(table.getDatabase(), deadline,
checkCancelled)
+ .getCollection(table.getName());
+ try (MongoCursor<Document> indexes =
collection.listIndexes().batchSize(1)
Review Comment:
Fixed in `41a2f893579`. listIndexes now requests a bounded batch of 128,
with the existing 128-index/64-KiB enumeration guards, cursor-lifetime timeout
and cancellation checks retained. The driver protocol test reproduces the old
batch size and passes with the larger batch. The serial preset runner passes
all 169 MongoDB tests; root FE build, Checkstyle and full retained split SQL
regression also pass.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoDriverCatalogClient.java:
##########
@@ -177,6 +183,131 @@ public BsonTimestamp selectSnapshot(String database,
String collection, String p
}
}
+ @Override
+ public MongoSplitProbe probeSplits(MongoTableHandle table, String field,
String preference,
+ TMongoSnapshot snapshot, int rows, long timeoutMs, Runnable
checkCancelled) {
+ long deadline = System.nanoTime() +
TimeUnit.MILLISECONDS.toNanos(timeoutMs);
+ String index = null;
+ long bytes = 0;
+ try {
+ var collection = probeDatabase(table.getDatabase(), deadline,
checkCancelled)
+ .getCollection(table.getName());
+ try (MongoCursor<Document> indexes =
collection.listIndexes().batchSize(1)
+ .timeoutMode(TimeoutMode.CURSOR_LIFETIME).iterator()) {
+ int count = 0;
+ while (true) {
+ // hasNext may issue getMore. Cancellation must be checked
before that request,
+ // not only after it returns; the lifetime timeout also
covers all cursor batches.
+ checkProbe(deadline, checkCancelled);
+ if (!indexes.hasNext()) {
+ break;
+ }
+ checkProbe(deadline, checkCancelled);
+ Document value = indexes.next();
+ bytes +=
MongoSplitPlan.encode(value.toBsonDocument()).length;
+ if (++count > 128 || bytes > 65536) {
+ throw new MongoRangePlanner.PlanningLimit();
+ }
+ if (eligibleIndex(value, field)) {
+ index = value.getString("name");
+ break;
+ }
+ }
+ }
+ if (index == null) {
+ return new MongoSplitProbe(List.of(), -1, bytes, false);
+ }
+ Document stats = probeDatabase(table.getDatabase(), deadline,
checkCancelled)
+ .runCommand(new Document("collStats",
table.getName()).append("scale", 1));
+ Number size = stats.get("size", Number.class);
+ long logicalBytes = size == null ? -1 : size.longValue();
+ List<BsonValue> samples = new ArrayList<>();
+ // Sampling an index prefix puts almost all increasing keys into
the final range.
+ // Keep $sample first so MongoDB can use its random cursor, then
retain only bounded
+ // fixed-width keys. Statistics tune cost only; they never narrow
source coverage.
+ // Unsupported large strings/arrays become null before they can
reach the FE.
+ Document type = new Document("$type", "$" + field);
+ Document supported = new Document("$in", List.of(type,
List.of("int", "long", "objectId", "date")));
+ Document projection = new Document("_id", 0).append("value",
+ new Document("$cond", List.of(supported, "$" + field, new
Document("$literal", null))));
+ // Do not shrink small-table requests below MongoDB's 5%
random-cursor threshold.
+ // Its historical random cursor can stall on concurrently
rewritten records; a bounded
+ // full sample of a small collection remains cheap and preserves
the pinned timestamp.
+ var pipeline = List.of(new Document("$sample", new
Document("size", rows)),
+ new Document("$project", projection), new
Document("$group", new Document("_id", null)
+ .append("values", new Document("$push",
"$value"))));
+ // Grouping at most 4096 fixed-width hints produces at most one
small document.
+ // batchSize=2 exhausts that reply, avoiding remote cursor cleanup
on cancellation and
+ // driver 5.14's invalid snapshot readConcern on killCursors. No
getMore is needed.
+ var command = new Document("aggregate",
table.getName()).append("pipeline", pipeline)
+ .append("cursor", new Document("batchSize", 2))
+ .append("readConcern", new Document("level",
"snapshot").append("atClusterTime",
+ new BsonTimestamp((int)
snapshot.getTimestampSeconds(),
+ (int) snapshot.getTimestampIncrement())));
+ BsonDocument reply = probeDatabase(table.getDatabase(), deadline,
checkCancelled)
+ .runCommand(command,
properties.readPreferenceForSnapshot(preference), BsonDocument.class);
+ checkProbe(deadline, checkCancelled);
+ bytes += MongoSplitPlan.encode(reply).length;
+ if (bytes > 1024 * 1024) {
+ throw new MongoRangePlanner.PlanningLimit();
+ }
+ for (BsonValue batch :
reply.getDocument("cursor").getArray("firstBatch")) {
+ for (BsonValue value : batch.asDocument().getArray("values")) {
+ checkProbe(deadline, checkCancelled);
+ if (samples.size() >= rows) {
+ throw new MongoRangePlanner.PlanningLimit();
+ }
+ samples.add(value);
+ }
+ }
+ checkProbe(deadline, checkCancelled);
+ return new MongoSplitProbe(samples, logicalBytes, bytes, true);
+ } catch (MongoException error) {
+ checkCancelled.run();
+ // Only the optional planning deadline can trigger auto fallback.
Permission denial,
+ // expired snapshot history, replacement, and transport errors
must fail the statement.
+ // A thread can resume after its deadline while handling an
earlier BadValue/permission
+ // failure. Elapsed time alone must not disguise that source error
as optional timeout.
+ if (error instanceof MongoExecutionTimeoutException || error
instanceof MongoOperationTimeoutException) {
+ throw new MongoRangePlanner.PlanningLimit();
+ }
+ throw failure(error);
+ }
+ }
+
+ static boolean eligibleIndex(Document index, String field) {
Review Comment:
Addressed in `5d7648cc2a0` and retained in `41a2f893579`. Eligibility checks
every key component for an ordinary +/-1 B-tree direction, including the stored
_fts:text pattern, and skips wildcard keys. Later ordinary indexes remain
eligible. TEXT-only auto fallback, explicit range rejection and later
ordinary-index selection pass against authenticated MongoDB 8 replica
set/mongos in both planning modes, with baseline-failing unit coverage.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,389 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ static final int SAMPLE_ROWS = 4096;
+ static final int PROBE_TIMEOUT_MS = 2000;
+ private static final long MAX_PLAN_DESCRIPTOR_BYTES = 8L * 1024 * 1024;
+ private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+ private final Supplier<MongoCatalogClient> client;
+ private final long probeTimeoutMs;
+
+ MongoRangePlanner(Supplier<MongoCatalogClient> client, long
probeTimeoutMs) {
+ this.client = client;
+ this.probeTimeoutMs = probeTimeoutMs;
+ }
+
+ MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table,
TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, long
descriptorBytes) {
+ checkCancelled(session, context);
+ var options = session.getSessionProperties();
+ String strategy = options.getOrDefault("mongodb_split_strategy",
"single");
+ if (!Set.of("single", "auto", "range").contains(strategy)) {
+ throw new IllegalArgumentException("Invalid
mongodb_split_strategy");
+ }
+ if ("single".equals(strategy)) {
+ return MongoSplitPlan.single(1, 1, "single_strategy");
+ }
+ String field = options.getOrDefault("mongodb_split_field", "");
+ boolean custom = !field.isEmpty();
Review Comment:
Fixed in `41a2f893579`. Empty and explicit _id now share the default-field
contract; only other custom fields require one MongoDB scan. The README is
clarified. A baseline-failing FE test covers two distinct scans for auto/range.
Native verification passes 16
single/default-range/explicit-_id-range/explicit-_id-auto join combinations
across replica set/mongos and both planning modes, checking original T and
actual splits/scanners. The retained SQL suite also compares explicit/default
_id joins to the single result.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,389 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ static final int SAMPLE_ROWS = 4096;
+ static final int PROBE_TIMEOUT_MS = 2000;
+ private static final long MAX_PLAN_DESCRIPTOR_BYTES = 8L * 1024 * 1024;
+ private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+ private final Supplier<MongoCatalogClient> client;
+ private final long probeTimeoutMs;
+
+ MongoRangePlanner(Supplier<MongoCatalogClient> client, long
probeTimeoutMs) {
+ this.client = client;
+ this.probeTimeoutMs = probeTimeoutMs;
+ }
+
+ MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table,
TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, long
descriptorBytes) {
+ checkCancelled(session, context);
+ var options = session.getSessionProperties();
+ String strategy = options.getOrDefault("mongodb_split_strategy",
"single");
+ if (!Set.of("single", "auto", "range").contains(strategy)) {
+ throw new IllegalArgumentException("Invalid
mongodb_split_strategy");
+ }
+ if ("single".equals(strategy)) {
+ return MongoSplitPlan.single(1, 1, "single_strategy");
+ }
+ String field = options.getOrDefault("mongodb_split_field", "");
+ boolean custom = !field.isEmpty();
+ if (!custom) {
+ field = "_id";
+ }
+ if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() ||
field.length() > 256) {
+ throw new IllegalArgumentException("MongoDB split field must be an
unambiguous top-level field");
+ }
+ String selectedField = field;
+ Budget budget =
session.getStatementScope().computeIfAbsent("mongodb.split.budget",
Budget::new);
+ String tableKey = tableKey(session, table);
+ // Count fallback scans too: a view must not make a global custom
field look unambiguous.
+ if (custom) {
+ budget.customField(tableKey);
+ }
+ Memo memo = session.getStatementScope().computeIfAbsent(tableKey,
Memo::new);
+ return memo.get(session, context, () -> load(session, table, snapshot,
preference, context,
+ strategy, selectedField, descriptorBytes, budget));
+ }
+
+ Optional<MongoSplitPlan> planned(ConnectorSession session,
MongoTableHandle table) {
+ Memo memo = session.getStatementScope().getIfPresent(tableKey(session,
table));
+ return memo == null ? Optional.empty() : memo.completed();
+ }
+
+ void freezeDiagnostics(ConnectorSession session, MongoTableHandle table,
Map<String, String> properties) {
+ session.getStatementScope().computeIfAbsent(tableKey(session, table) +
":diagnostics",
+ () -> Map.copyOf(properties));
+ }
+
+ Map<String, String> diagnostics(ConnectorSession session, MongoTableHandle
table) {
+ Map<String, String> properties = session.getStatementScope()
+ .getIfPresent(tableKey(session, table) + ":diagnostics");
+ return properties == null ? Map.of() : properties;
+ }
+
+ private String tableKey(ConnectorSession session, MongoTableHandle table) {
+ // One provider can see several namespaces; never reuse another
collection's bounds/admission.
+ return memoKey + ":" + session.getCatalogId() + ":" +
table.getSchema().getVersion();
+ }
+
+ private MongoSplitPlan load(ConnectorSession session, MongoTableHandle
table, TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, String strategy,
String field,
+ long descriptorBytes, Budget budget) {
+ int maximum = splitLimit(session);
+ validateSplitSizes(session);
+ int requested = Math.min(8, maximum);
+ // The default single strategy exits before this path; opted-in plans
always reserve bytes.
+ assert descriptorBytes > 0 : "MongoDB range descriptor bytes must be
positive";
+ int descriptorMaximum = (int) Math.min(maximum,
MAX_PLAN_DESCRIPTOR_BYTES / descriptorBytes);
+ if (descriptorMaximum == 0) {
+ throw new DorisConnectorException("MongoDB split descriptor
exceeds planning byte budget");
+ }
+ String source = snapshot == null ? "catalog:" + session.getCatalogId()
+ : snapshot.getSourceKind() + ":" + snapshot.getSourceId();
+ if (!"collection".equals(table.getObjectKind())) {
+ return fallback(strategy, requested, "unsupported_object", source,
descriptorBytes, budget);
+ }
+ if (snapshot == null || session.getStatementScope() ==
ConnectorStatementScope.NONE) {
+ return fallback(strategy, requested, "snapshot_required", source,
descriptorBytes, budget);
+ }
+ if (descriptorMaximum == 1) {
+ return single(requested, maximum == 1 ? "split_limit" :
"descriptor_budget",
+ source, descriptorBytes, budget);
+ }
+ long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+ Math.min(probeTimeoutMs,
MongoSnapshot.remainingMillis(session)));
+ MongoSplitProbe probe;
+ try {
+ budget.acquireProbe(session, context, deadline);
+ try {
+ checkCancelled(session, context);
+ probe = client.get().probeSplits(table, field, preference,
snapshot, SAMPLE_ROWS,
+ remaining(session, context, deadline), () ->
checkCancelled(session, context));
+ budget.sampled(probe.sampleBytes());
+ } finally {
+ budget.probes.release();
+ }
+ } catch (PlanningLimit limit) {
+ checkCancelled(session, context);
+ MongoSnapshot.remainingMillis(session);
+ // Only optional pre-publication tuning can fall back.
Cancellation, authorization,
+ // history and source errors must never be converted into a
successful full scan.
+ return fallback(strategy, requested, "probe_budget", source,
descriptorBytes, budget);
+ }
+ checkCancelled(session, context);
+ MongoSnapshot.remainingMillis(session);
+ long targetBytes = targetBytes(session, probe.logicalBytes());
+ long estimate = probe.logicalBytes() < 0 ? 8
+ : Math.max(1, probe.logicalBytes() / targetBytes +
(probe.logicalBytes() % targetBytes == 0 ? 0 : 1));
+ requested = (int) Math.min(maximum, estimate);
+ if (!probe.indexed()) {
+ return fallback(strategy, requested, "missing_index", source,
descriptorBytes, budget);
+ }
+ String family = family(probe.samples());
+ if (family.isEmpty()) {
+ return probe.samples().isEmpty()
+ ? single(requested, "empty_sample", source,
descriptorBytes, budget)
+ : fallback(strategy, requested, "unsupported_type",
source, descriptorBytes, budget);
+ }
+ if ("auto".equals(strategy) && probe.logicalBytes() < 0) {
+ return single(requested, "unknown_complement_cost", source,
descriptorBytes, budget);
+ }
+ // Division avoids overflow when a valid shared split-size setting is
near Long.MAX_VALUE.
+ if ("auto".equals(strategy) && probe.logicalBytes() / targetBytes < 2)
{
+ return single(requested, "small_collection", source,
descriptorBytes, budget);
+ }
+ int admitted = budget.admit(source, Math.min(requested,
descriptorMaximum), descriptorBytes);
+ try {
+ var values = probe.samples().stream().filter(v ->
family.equals(family(v))).toList();
+ MongoSplitPlan plan = MongoSplitPlan.ranges(field, family, values,
requested, admitted);
+ budget.refund(source, admitted - plan.size(), descriptorBytes);
+ return plan;
+ } catch (RuntimeException failure) {
+ budget.refund(source, admitted, descriptorBytes);
+ throw failure;
+ }
+ }
+
+ static int splitLimit(ConnectorSession session) {
+ long limit = number(session.getSessionProperties(),
"max_file_split_num", 100000);
+ // Mongo freezes all boundaries even when transport is streaming; that
retained plan must
+ // honor the shared count cap too. Nonpositive means no user cap, as
on other file scans.
+ return limit <= 0 ? 256 : (int) Math.min(256, limit);
+ }
+
+ private static void validateSplitSizes(ConnectorSession session) {
+ targetBytes(session, -1);
+ }
+
+ static long targetBytes(ConnectorSession session, long logicalBytes) {
+ var options = session.getSessionProperties();
+ long specified = number(options, "file_split_size", 0);
Review Comment:
Fixed in `41a2f893579`. Negative file_split_size now throws
IllegalArgumentException. Zero still selects automatic sizing, positive
overrides still apply, and nonpositive split-count caps retain their existing
unlimited meaning. Added a regression that fails on the old silent fallback;
all 169 MongoDB FE tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]