Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4623526545


##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoDriverCatalogClient.java:
##########
@@ -177,6 +184,126 @@ public BsonTimestamp selectSnapshot(String database, 
String collection, String p
         }
     }
 
+    @Override
+    public MongoSplitProbe probeSplits(MongoTableHandle table, String field, 
String preference,
+            TMongoSnapshot snapshot, int rows, long timeoutMs, Runnable 
checkCancelled) {
+        long deadline = System.nanoTime() + 
TimeUnit.MILLISECONDS.toNanos(timeoutMs);
+        String index = null;
+        long bytes = 0;
+        try {
+            var collection = probeDatabase(table.getDatabase(), deadline, 
checkCancelled)
+                    .getCollection(table.getName());
+            try (MongoCursor<Document> indexes = 
collection.listIndexes().batchSize(1)
+                    .timeoutMode(TimeoutMode.CURSOR_LIFETIME).iterator()) {
+                int count = 0;
+                while (true) {
+                    // hasNext may issue getMore. Cancellation must be checked 
before that request,
+                    // not only after it returns; the lifetime timeout also 
covers all cursor batches.
+                    checkProbe(deadline, checkCancelled);
+                    if (!indexes.hasNext()) {
+                        break;
+                    }
+                    checkProbe(deadline, checkCancelled);
+                    Document value = indexes.next();
+                    bytes += 
MongoSplitPlan.encode(value.toBsonDocument()).length;
+                    if (++count > 128 || bytes > 65536) {
+                        throw new MongoRangePlanner.PlanningLimit();
+                    }
+                    if (eligibleIndex(value, field)) {
+                        index = value.getString("name");
+                        break;
+                    }
+                }
+            }
+            if (index == null) {
+                return new MongoSplitProbe(List.of(), -1, bytes, false);
+            }
+            Number size = probeDatabase(table.getDatabase(), deadline, 
checkCancelled)
+                    .runCommand(new Document("collStats", 
table.getName()).append("scale", 1))
+                    .get("size", Number.class);
+            long logicalBytes = size == null ? -1 : size.longValue();
+            List<BsonValue> samples = new ArrayList<>();
+            // Index-prefix sampling is deliberately bounded, not an unbiased 
cardinality estimate.
+            // It can produce skewed ranges, but never narrows coverage or 
removes the complement.
+            // Project only qualified fixed-width scalars: a giant 
array/string key cannot multiply
+            // the FE's retained memory by sample_rows. The remaining types 
become BsonNull hints.
+            Document type = new Document("$type", "$" + field);
+            Document supported = new Document("$in", List.of(type, 
List.of("int", "long", "objectId", "date")));
+            Document projection = new Document("_id", 0).append("value",
+                    new Document("$cond", List.of(supported, "$" + field, new 
Document("$literal", null))));
+            try (ClientSession session = 
client.startSession(ClientSessionOptions.builder()
+                    .causallyConsistent(false).snapshot(true).build())) {
+                session.setSnapshotTimestamp(new BsonTimestamp((int) 
snapshot.getTimestampSeconds(),
+                        (int) snapshot.getTimestampIncrement()));
+                var data = probeDatabase(table.getDatabase(), deadline, 
checkCancelled)
+                        .getCollection(table.getName(), BsonDocument.class)
+                        
.withReadPreference(properties.readPreferenceForSnapshot(preference));
+                // Negative batchSize requests singleBatch in the pinned 
driver. Fixed-width samples
+                // fit well below one BSON reply; no remote cursor survives 
cancellation/timeout.
+                // Driver 5.14 otherwise adds snapshot readConcern to 
killCursors, which MongoDB rejects.
+                try (MongoCursor<BsonDocument> cursor = 
data.find(session).hintString(index)
+                        .projection(projection).limit(rows).batchSize(-rows)
+                        .timeoutMode(TimeoutMode.CURSOR_LIFETIME).iterator()) {
+                    while (true) {
+                        // Do not let a per-iteration timeout reset or a late 
cancellation start
+                        // another remote batch after the optional planning 
budget has ended.
+                        checkProbe(deadline, checkCancelled);
+                        if (!cursor.hasNext()) {
+                            break;
+                        }
+                        checkProbe(deadline, checkCancelled);
+                        BsonDocument value = cursor.next();
+                        bytes += MongoSplitPlan.encode(value).length;
+                        if (samples.size() >= rows || bytes > 1024 * 1024) {
+                            throw new MongoRangePlanner.PlanningLimit();
+                        }
+                        samples.add(value.get("value", BsonNull.VALUE));
+                    }
+                }
+            }
+            checkProbe(deadline, checkCancelled);
+            return new MongoSplitProbe(samples, logicalBytes, bytes, true);
+        } catch (MongoException error) {
+            checkCancelled.run();
+            // Only the optional planning deadline can trigger auto fallback. 
Permission denial,
+            // expired snapshot history, replacement, and transport errors 
must fail the statement.
+            // A thread can resume after its deadline while handling an 
earlier BadValue/permission
+            // failure. Elapsed time alone must not disguise that source error 
as optional timeout.
+            if (error instanceof MongoExecutionTimeoutException || error 
instanceof MongoOperationTimeoutException) {
+                throw new MongoRangePlanner.PlanningLimit();
+            }
+            throw failure(error);
+        }
+    }
+
+    static boolean eligibleIndex(Document index, String field) {
+        Document keys = index.get("key", Document.class);
+        if (keys == null || keys.isEmpty() || 
!field.equals(keys.keySet().iterator().next())
+                || !(keys.get(field) instanceof Number direction) || 
Math.abs(direction.intValue()) != 1

Review Comment:
   Fixed in 8a4819d2b3a1a363792be509076446c32c788530. Index eligibility now 
checks every key component and excludes `$**` and paths ending in `.$**` before 
selecting a hint. Enumeration continues to a later ordinary compound B-tree. If 
none qualifies, `auto` keeps the single-split fallback and explicit `range` 
reports `missing_index`; source errors still propagate.
   
   Four new UT methods fail on the old implementation and pass with the fix. 
They cover ascending/descending prefixes, root/nested wildcard components, 
later ordinary-index selection, zero sampling sessions for wildcard-only 
indexes, and planner fallback/rejection. All 136 Mongo connector UTs, full FE 
Checkstyle and the FE build pass.
   
   The retained SQL split suite also passes on authenticated MongoDB 7.0.28 and 
8.0.20 replica sets and mongos, with zero failed/fatal/skipped scripts in all 
four runs. Each includes both synchronous and streaming wildcard cases, plus 
the existing date/Decimal/time-zone comparisons. The framework-generated new 
results are independently checked as 601 rows and sum 180300, with all old 
golden values unchanged. Source-only snapshot probes reproduce 291 for the 
wildcard hint and succeed with the later ordinary compound hint on all four 
environments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to