Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4615278513


##########
be/src/exec/scan/scanner_scheduler.cpp:
##########
@@ -271,6 +271,16 @@ void 
ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
                     // We got a new created block or a reused block.
                     status = scanner->get_block_after_projects(state, 
free_block.get(), &eos);
                     first_read = false;
+                    if (status.is<ErrorCode::NEED_SEND_AGAIN>()) {
+                        // A pool waiter must release this worker so a queued 
lease holder can
+                        // resume (including a workload group with just one 
scan thread).
+                        // The reader retains its split/deadline and produced 
no output to publish.
+                        DCHECK_EQ(free_block->rows(), 0);
+                        ctx->inc_block_usage(free_block->allocated_bytes());
+                        scan_task->cached_block = std::move(free_block);

Review Comment:
   Fixed in 1aa44ce6949ee7d55f026ac99a4a951a39473dbe. Resource waits now enter 
the deferred end of the pending deque, while ordinary runnable scanners retain 
their previous LIFO order. Repeated waiters cannot bury the parked lease holder 
when admission shrinks. The TaskExecutor path also queues a consumed waiter 
before selecting the next task, rather than directly resuming it. The original 
reader and deadline are retained.
   
   A real one-worker scanner-chain UT first queues five holder/waiter results, 
lowers the effective cap from five to four, then requires exact rows and 
completion of all scanners. It timed out on the old scheduler and passes with 
this fix. BE_TEST disables low-memory mode, so the test applies that identical 
admission ceiling directly under the transfer lock. A separate UT covers 
TaskExecutor holder selection; all 153 related ASAN BE tests pass.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +121,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession 
session, ConnectorScan
             // Include complete source roots so pruning cannot turn malformed 
nested paths into missing values.
             d.setProjectionBson(encode(projection));
         }
-        // One cursor owns the entire logical namespace. Filters and LIMIT 
remain Doris residuals.
-        return List.of(new MongoScanRange(d));
+        long descriptorSize = descriptorBytes(session, handle,
+                d.isSetFilterBson() ? d.getFilterBson().length : 0,
+                d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+        MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot, 
d.getReadPreference(),
+                request.getScanContext(), descriptorSize);
+        // Retain the shared descriptor once; streaming creates owned RPC 
values under engine backpressure.
+        // A wide schema must not be copied 256 times merely to keep a frozen 
boundary table in FE.
+        return new AbstractList<>() {
+            @Override
+            public ConnectorScanRange get(int index) {
+                java.util.Objects.checkIndex(index, plan.size());
+                MongoRangePlanner.checkCancelled(session, 
request.getScanContext());
+                TMongoScanDesc range = d.deepCopy();
+                TMongoSplit split = new TMongoSplit();
+                split.setStrategy(plan.family().isEmpty() ? "single" : 
"range");
+                split.setField(plan.field());
+                split.setTypeFamily(plan.family());
+                split.setIndex(index);
+                split.setRequested(plan.requested());
+                split.setAdmitted(plan.admitted());
+                split.setActual(plan.size());
+                split.setReason(plan.reason());
+                if (!plan.family().isEmpty()) {
+                    range.setVersion(2);
+                    
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+                }
+                range.setSplit(split);
+                if (snapshot != null) {
+                    range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+                }
+                if (plan.descriptorBudget() > 0 && descriptorSize > 
plan.descriptorBudget()) {
+                    try {
+                        // EXPLAIN may freeze a plan before a later caller 
supplies a larger candidate
+                        // or projection. Never let that RPC silently exceed 
the frozen reservation.
+                        int actual = new org.apache.thrift.TSerializer(
+                                new 
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+                        if (actual > plan.descriptorBudget()) {
+                            throw new DorisConnectorException("MongoDB 
descriptor grew after split plan freeze");
+                        }
+                    } catch (org.apache.thrift.TException failure) {
+                        throw new DorisConnectorException("Cannot size MongoDB 
split descriptor");
+                    }
+                }
+                return new MongoScanRange(range);
+            }
+
+            @Override
+            public int size() {
+                return plan.size();
+            }
+        };
+    }
+
+    @Override
+    public long streamingSplitEstimate(ConnectorSession session, 
ConnectorTableHandle handle,
+            Optional<ConnectorExpression> filter, boolean countPushdown) {
+        return 
"single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
 "single"))
+                ? -1 : Math.min(8, MongoRangePlanner.splitLimit(session));
+    }
+
+    @Override
+    public ConnectorSplitSource streamSplits(ConnectorSession session, 
ConnectorScanRequest request) {
+        return new ConnectorSplitSource() {
+            private Iterator<ConnectorScanRange> ranges;
+            private boolean closed;
+
+            @Override
+            public boolean hasNext() {
+                if (closed) {
+                    return false;
+                }
+                MongoRangePlanner.checkCancelled(session, 
request.getScanContext());
+                if (ranges == null) {
+                    // All probes, admission and boundary validation finish 
before the first publication.
+                    ranges = planScan(session, request).iterator();
+                }
+                return ranges.hasNext();
+            }
+
+            @Override
+            public ConnectorScanRange next() {
+                if (!hasNext()) {
+                    throw new NoSuchElementException();
+                }
+                return ranges.next();
+            }
+
+            @Override
+            public void close() {
+                closed = true;
+                ranges = Collections.emptyIterator();
+            }
+        };
+    }
+
+    private long descriptorBytes(ConnectorSession session, MongoTableHandle 
table,
+            int candidateBytes, int projectionBytes) {
+        if 
("single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
 "single"))) {
+            return 0;
+        }
+        // A count-only cap is insufficient: 100 copies of a large 
schema/credential/filter can exceed
+        // a split RPC's body limit. Include UTF-8 payloads plus room for 
Thrift fields and range BSON.
+        long bytes = 8192L + candidateBytes + projectionBytes;
+        for (String value : List.of(properties.getUri(), 
properties.getUsername(), properties.getPassword(),
+                properties.getAuthDatabase(), table.getDatabase(), 
table.getName(), table.getUuid(),
+                table.getSchema().getVersion(), 
table.getSchema().toBindingsJson())) {

Review Comment:
   Fixed in 1aa44ce6949ee7d55f026ac99a4a951a39473dbe. Descriptor admission now 
reads the completed scan descriptor and explicitly counts the UTF-8 bytes in 
snapshot.timeseries_options_json before range/auto fallback admission, in 
addition to the existing filter, projection and schema/connection payloads.
   
   The new regression uses 900 aliases and a 15000-character timeField, 
independently serializes each actual TMongoScanDesc with Thrift's binary 
protocol, and checks the accumulated source payload against 32 MiB. The old 
estimate admits descriptors past the wire-byte cap; the corrected estimate 
rejects before crossing it. All 132 MongoDB FE UTs and FE Checkstyle pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to