Gabriel39 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4615278513
##########
be/src/exec/scan/scanner_scheduler.cpp:
##########
@@ -271,6 +271,16 @@ void
ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
// We got a new created block or a reused block.
status = scanner->get_block_after_projects(state,
free_block.get(), &eos);
first_read = false;
+ if (status.is<ErrorCode::NEED_SEND_AGAIN>()) {
+ // A pool waiter must release this worker so a queued
lease holder can
+ // resume (including a workload group with just one
scan thread).
+ // The reader retains its split/deadline and produced
no output to publish.
+ DCHECK_EQ(free_block->rows(), 0);
+ ctx->inc_block_usage(free_block->allocated_bytes());
+ scan_task->cached_block = std::move(free_block);
Review Comment:
Fixed in 1aa44ce6949ee7d55f026ac99a4a951a39473dbe. Resource waits now enter
the deferred end of the pending deque, while ordinary runnable scanners retain
their previous LIFO order. Repeated waiters cannot bury the parked lease holder
when admission shrinks. The TaskExecutor path also queues a consumed waiter
before selecting the next task, rather than directly resuming it. The original
reader and deadline are retained.
A real one-worker scanner-chain UT first queues five holder/waiter results,
lowers the effective cap from five to four, then requires exact rows and
completion of all scanners. It timed out on the old scheduler and passes with
this fix. BE_TEST disables low-memory mode, so the test applies that identical
admission ceiling directly under the transfer lock. A separate UT covers
TaskExecutor holder selection; all 153 related ASAN BE tests pass.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +121,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle,
+ d.isSetFilterBson() ? d.getFilterBson().length : 0,
+ d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
+ java.util.Objects.checkIndex(index, plan.size());
+ MongoRangePlanner.checkCancelled(session,
request.getScanContext());
+ TMongoScanDesc range = d.deepCopy();
+ TMongoSplit split = new TMongoSplit();
+ split.setStrategy(plan.family().isEmpty() ? "single" :
"range");
+ split.setField(plan.field());
+ split.setTypeFamily(plan.family());
+ split.setIndex(index);
+ split.setRequested(plan.requested());
+ split.setAdmitted(plan.admitted());
+ split.setActual(plan.size());
+ split.setReason(plan.reason());
+ if (!plan.family().isEmpty()) {
+ range.setVersion(2);
+
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+ }
+ range.setSplit(split);
+ if (snapshot != null) {
+ range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+ }
+ if (plan.descriptorBudget() > 0 && descriptorSize >
plan.descriptorBudget()) {
+ try {
+ // EXPLAIN may freeze a plan before a later caller
supplies a larger candidate
+ // or projection. Never let that RPC silently exceed
the frozen reservation.
+ int actual = new org.apache.thrift.TSerializer(
+ new
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+ if (actual > plan.descriptorBudget()) {
+ throw new DorisConnectorException("MongoDB
descriptor grew after split plan freeze");
+ }
+ } catch (org.apache.thrift.TException failure) {
+ throw new DorisConnectorException("Cannot size MongoDB
split descriptor");
+ }
+ }
+ return new MongoScanRange(range);
+ }
+
+ @Override
+ public int size() {
+ return plan.size();
+ }
+ };
+ }
+
+ @Override
+ public long streamingSplitEstimate(ConnectorSession session,
ConnectorTableHandle handle,
+ Optional<ConnectorExpression> filter, boolean countPushdown) {
+ return
"single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
"single"))
+ ? -1 : Math.min(8, MongoRangePlanner.splitLimit(session));
+ }
+
+ @Override
+ public ConnectorSplitSource streamSplits(ConnectorSession session,
ConnectorScanRequest request) {
+ return new ConnectorSplitSource() {
+ private Iterator<ConnectorScanRange> ranges;
+ private boolean closed;
+
+ @Override
+ public boolean hasNext() {
+ if (closed) {
+ return false;
+ }
+ MongoRangePlanner.checkCancelled(session,
request.getScanContext());
+ if (ranges == null) {
+ // All probes, admission and boundary validation finish
before the first publication.
+ ranges = planScan(session, request).iterator();
+ }
+ return ranges.hasNext();
+ }
+
+ @Override
+ public ConnectorScanRange next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ return ranges.next();
+ }
+
+ @Override
+ public void close() {
+ closed = true;
+ ranges = Collections.emptyIterator();
+ }
+ };
+ }
+
+ private long descriptorBytes(ConnectorSession session, MongoTableHandle
table,
+ int candidateBytes, int projectionBytes) {
+ if
("single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
"single"))) {
+ return 0;
+ }
+ // A count-only cap is insufficient: 100 copies of a large
schema/credential/filter can exceed
+ // a split RPC's body limit. Include UTF-8 payloads plus room for
Thrift fields and range BSON.
+ long bytes = 8192L + candidateBytes + projectionBytes;
+ for (String value : List.of(properties.getUri(),
properties.getUsername(), properties.getPassword(),
+ properties.getAuthDatabase(), table.getDatabase(),
table.getName(), table.getUuid(),
+ table.getSchema().getVersion(),
table.getSchema().toBindingsJson())) {
Review Comment:
Fixed in 1aa44ce6949ee7d55f026ac99a4a951a39473dbe. Descriptor admission now
reads the completed scan descriptor and explicitly counts the UTF-8 bytes in
snapshot.timeseries_options_json before range/auto fallback admission, in
addition to the existing filter, projection and schema/connection payloads.
The new regression uses 900 aliases and a 15000-character timeField,
independently serializes each actual TMongoScanDesc with Thrift's binary
protocol, and checks the accumulated source payload against 32 MiB. The old
estimate admits descriptors past the wire-byte cap; the corrected estimate
rejects before crossing it. All 132 MongoDB FE UTs and FE Checkstyle pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]