Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4641936639


##########
be/src/format_v2/table/mongodb_reader.cpp:
##########
@@ -220,16 +261,29 @@ class NativeMongoStream final : public MongoCursorStream {
 public:
     ~NativeMongoStream() override { close(); }
     Status open(const TMongoScanDesc& d, std::function<bool()> cancelled) 
override {
-        _cancelled = std::move(cancelled);
-        _deadline = Clock::now() + std::chrono::milliseconds(d.timeout_ms);
+        if (_client) {
+            return Status::InvalidArgument("MongoDB stream requires close 
before reopening");
+        }
+        if (!_pool) {
+            _cancelled = std::move(cancelled);
+            _deadline = Clock::now() + std::chrono::milliseconds(d.timeout_ms);
+            RETURN_IF_ERROR(check_deadline());
+            RETURN_IF_ERROR(validate_pushdown_bson(d));
+            RETURN_IF_ERROR(validate_snapshot_descriptor(d));
+            RETURN_IF_ERROR(acquire_pool(d, &_pool));
+        }
+        // Preserve the first deadline across cooperative retries. Sleeping 
here can occupy a
+        // workload group's only scanner worker while the client holder is 
queued to resume.
         RETURN_IF_ERROR(check_deadline());
-        RETURN_IF_ERROR(validate_pushdown_bson(d));
-        RETURN_IF_ERROR(validate_snapshot_descriptor(d));
-        RETURN_IF_ERROR(acquire_pool(d, &_pool));
-        while (!(_client = mongoc_client_pool_try_pop(_pool->pool.get()))) {
+        _client = mongoc_client_pool_try_pop(_pool->pool.get());

Review Comment:
   Fixed in `5d7648cc2a0`. PipelineTask now waits for initialized downstream 
read dependencies before opening the source, excludes the source's own data 
dependency, and refreshes dependencies after normal open. The controlled 
baseline test fails on premature source open; the new cases cover both resume 
and cancellation. The old ASAN BE also timed out on a 32769-row join with 2-KiB 
projected payloads, pool=2, two actual scanners per scan, one pipeline task and 
runtime filters off. The updated build passes all eight single/range, 
sync/streaming and replica-set/mongos combinations, with the original T and 
actual scanner/split counts verified. The pool=1/worker=1 cases, 111 related BE 
tests, and the full retained split SQL suite also pass.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoDriverCatalogClient.java:
##########
@@ -177,6 +183,131 @@ public BsonTimestamp selectSnapshot(String database, 
String collection, String p
         }
     }
 
+    @Override
+    public MongoSplitProbe probeSplits(MongoTableHandle table, String field, 
String preference,
+            TMongoSnapshot snapshot, int rows, long timeoutMs, Runnable 
checkCancelled) {
+        long deadline = System.nanoTime() + 
TimeUnit.MILLISECONDS.toNanos(timeoutMs);
+        String index = null;
+        long bytes = 0;
+        try {
+            var collection = probeDatabase(table.getDatabase(), deadline, 
checkCancelled)
+                    .getCollection(table.getName());
+            try (MongoCursor<Document> indexes = 
collection.listIndexes().batchSize(1)
+                    .timeoutMode(TimeoutMode.CURSOR_LIFETIME).iterator()) {
+                int count = 0;
+                while (true) {
+                    // hasNext may issue getMore. Cancellation must be checked 
before that request,
+                    // not only after it returns; the lifetime timeout also 
covers all cursor batches.
+                    checkProbe(deadline, checkCancelled);
+                    if (!indexes.hasNext()) {
+                        break;
+                    }
+                    checkProbe(deadline, checkCancelled);
+                    Document value = indexes.next();
+                    bytes += 
MongoSplitPlan.encode(value.toBsonDocument()).length;
+                    if (++count > 128 || bytes > 65536) {
+                        throw new MongoRangePlanner.PlanningLimit();
+                    }
+                    if (eligibleIndex(value, field)) {

Review Comment:
   Fixed in `5d7648cc2a0`. Index eligibility now checks the complete key 
pattern and accepts only ordinary numeric +/-1 B-tree components without 
wildcard keys. Unsupported families are skipped so later ordinary indexes 
remain eligible. The baseline regression rejects the previously accepted TEXT 
suffix; coverage includes the actual `_fts: text, _ftsx: 1` pattern and 
text/hashed/geo/columnstore suffixes. Authenticated MongoDB 8 replica-set and 
mongos checks pass for TEXT-only auto fallback, explicit range rejection and 
selection of a later ordinary index in both planning modes. All 165 MongoDB FE 
tests, FE Checkstyle, root FE/ASAN BE builds and the complete split SQL suite 
pass. This increment does not claim a new MongoDB 7 qualification run.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to