github-actions[bot] commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4611010746


##########
be/src/exec/scan/scanner_scheduler.cpp:
##########
@@ -271,6 +271,16 @@ void 
ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
                     // We got a new created block or a reused block.
                     status = scanner->get_block_after_projects(state, 
free_block.get(), &eos);
                     first_read = false;
+                    if (status.is<ErrorCode::NEED_SEND_AGAIN>()) {
+                        // A pool waiter must release this worker so a queued 
lease holder can
+                        // resume (including a workload group with just one 
scan thread).
+                        // The reader retains its split/deadline and produced 
no output to publish.
+                        DCHECK_EQ(free_block->rows(), 0);
+                        ctx->inc_block_usage(free_block->allocated_bytes());
+                        scan_task->cached_block = std::move(free_block);

Review Comment:
   [P2] Keep client holders reachable when scanner admission shrinks
   
   These retries become ordinary `COMPLETED` empty results, but their 
resubmission can starve the client holder. With five active scanners and 
`mongodb.pool_max_clients=1`, completed results can be `[A(holder, non-EOF), B, 
C, D, E(waiters)]`. If workload-group pressure sets sticky `low_memory_mode` 
before consumption, `can_admit_scan_task` lowers admission to four. Consuming A 
parks it because the four completed waiters occupy every slot; consuming a 
waiter pushes it above A on `_pending_tasks`' LIFO stack. Each available slot 
then admits a waiter, so A never resumes to release its client and the query 
fails at a waiter's deadline, even with idle workers. This is the new 
completed-result/resubmission path, distinct from the removed blocking-checkout 
loop. Requeue resource waits fairly or keep holders eligible, and cover an 
admission decrease while holder/waiter results are queued.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +121,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle,
+                d.isSetFilterBson() ? d.getFilterBson().length : 0,
+                d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {
+                java.util.Objects.checkIndex(index, plan.size());
+                MongoRangePlanner.checkCancelled(session, 
request.getScanContext());
+                TMongoScanDesc range = d.deepCopy();
+                TMongoSplit split = new TMongoSplit();
+                split.setStrategy(plan.family().isEmpty() ? "single" : 
"range");
+                split.setField(plan.field());
+                split.setTypeFamily(plan.family());
+                split.setIndex(index);
+                split.setRequested(plan.requested());
+                split.setAdmitted(plan.admitted());
+                split.setActual(plan.size());
+                split.setReason(plan.reason());
+                if (!plan.family().isEmpty()) {
+                    range.setVersion(2);
+                    
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+                }
+                range.setSplit(split);
+                if (snapshot != null) {
+                    range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+                }
+                if (plan.descriptorBudget() > 0 && descriptorSize > 
plan.descriptorBudget()) {
+                    try {
+                        // EXPLAIN may freeze a plan before a later caller 
supplies a larger candidate
+                        // or projection. Never let that RPC silently exceed 
the frozen reservation.
+                        int actual = new org.apache.thrift.TSerializer(
+                                new 
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+                        if (actual > plan.descriptorBudget()) {
+                            throw new DorisConnectorException("MongoDB 
descriptor grew after split plan freeze");
+                        }
+                    } catch (org.apache.thrift.TException failure) {
+                        throw new DorisConnectorException("Cannot size MongoDB 
split descriptor");
+                    }
+                }
+                return new MongoScanRange(range);
+            }
+
+            @Override
+            public int size() {
+                return plan.size();
+            }
+        };
+    }
+
+    @Override
+    public long streamingSplitEstimate(ConnectorSession session, 
ConnectorTableHandle handle,
+            Optional<ConnectorExpression> filter, boolean countPushdown) {
+        return 
"single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
 "single"))
+                ? -1 : Math.min(8, MongoRangePlanner.splitLimit(session));
+    }
+
+    @Override
+    public ConnectorSplitSource streamSplits(ConnectorSession session, 
ConnectorScanRequest request) {
+        return new ConnectorSplitSource() {
+            private Iterator<ConnectorScanRange> ranges;
+            private boolean closed;
+
+            @Override
+            public boolean hasNext() {
+                if (closed) {
+                    return false;
+                }
+                MongoRangePlanner.checkCancelled(session, 
request.getScanContext());
+                if (ranges == null) {
+                    // All probes, admission and boundary validation finish 
before the first publication.
+                    ranges = planScan(session, request).iterator();
+                }
+                return ranges.hasNext();
+            }
+
+            @Override
+            public ConnectorScanRange next() {
+                if (!hasNext()) {
+                    throw new NoSuchElementException();
+                }
+                return ranges.next();
+            }
+
+            @Override
+            public void close() {
+                closed = true;
+                ranges = Collections.emptyIterator();
+            }
+        };
+    }
+
+    private long descriptorBytes(ConnectorSession session, MongoTableHandle 
table,
+            int candidateBytes, int projectionBytes) {
+        if 
("single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
 "single"))) {
+            return 0;
+        }
+        // A count-only cap is insufficient: 100 copies of a large 
schema/credential/filter can exceed
+        // a split RPC's body limit. Include UTF-8 payloads plus room for 
Thrift fields and range BSON.
+        long bytes = 8192L + candidateBytes + projectionBytes;
+        for (String value : List.of(properties.getUri(), 
properties.getUsername(), properties.getPassword(),
+                properties.getAuthDatabase(), table.getDatabase(), 
table.getName(), table.getUuid(),
+                table.getSchema().getVersion(), 
table.getSchema().toBindingsJson())) {

Review Comment:
   [P2] Include time-series snapshot options in descriptor admission
   
   For `auto`/snapshot time-series fallbacks, `MongoSnapshot.prepare` emits 
`snapshot.timeseries_options_json`, but this estimator never counts it. BE 
accepts up to 16384 bytes, so the 8192-byte pad is insufficient. A static 
example is a 15000-character `timeField` without `metaField`, with 900 BIGINT 
aliases `c0000`..`c0899` mapped to `['v']`: bindings occupy 138627 bytes. With 
the short `mongodb://localhost:27017`, `sample.records`, default auth database 
and count projection, 228 separate scans reserve 33502320 bytes (below 32 MiB), 
while counted strings plus the timeField alone require at least 35051352 bytes. 
The long field passes MongoDB's [field 
validator](https://github.com/mongodb/mongo/blob/r7.0.28/src/mongo/db/timeseries/timeseries_global_options.h#L39-L44)
 and [no-meta default-index 
path](https://github.com/mongodb/mongo/blob/r7.0.28/src/mongo/db/catalog/create_collection.cpp#L206-L228)
 by inspection. Every fallback now invokes admission, unlike the previous 
finding; this 
 is an omitted serialized field in the estimator. The growth guard is skipped 
because the initial estimate equals its reservation. Include snapshot options, 
or size the complete descriptor, before admission and cover this 
repeated-fallback byte limit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to