github-actions[bot] commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4611010746
##########
be/src/exec/scan/scanner_scheduler.cpp:
##########
@@ -271,6 +271,16 @@ void
ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
// We got a new created block or a reused block.
status = scanner->get_block_after_projects(state,
free_block.get(), &eos);
first_read = false;
+ if (status.is<ErrorCode::NEED_SEND_AGAIN>()) {
+ // A pool waiter must release this worker so a queued
lease holder can
+ // resume (including a workload group with just one
scan thread).
+ // The reader retains its split/deadline and produced
no output to publish.
+ DCHECK_EQ(free_block->rows(), 0);
+ ctx->inc_block_usage(free_block->allocated_bytes());
+ scan_task->cached_block = std::move(free_block);
Review Comment:
[P2] Keep client holders reachable when scanner admission shrinks
These retries become ordinary `COMPLETED` empty results, but their
resubmission can starve the client holder. With five active scanners and
`mongodb.pool_max_clients=1`, completed results can be `[A(holder, non-EOF), B,
C, D, E(waiters)]`. If workload-group pressure sets sticky `low_memory_mode`
before consumption, `can_admit_scan_task` lowers admission to four. Consuming A
parks it because the four completed waiters occupy every slot; consuming a
waiter pushes it above A on `_pending_tasks`' LIFO stack. Each available slot
then admits a waiter, so A never resumes to release its client and the query
fails at a waiter's deadline, even with idle workers. This is the new
completed-result/resubmission path, distinct from the removed blocking-checkout
loop. Requeue resource waits fairly or keep holders eligible, and cover an
admission decrease while holder/waiter results are queued.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +121,115 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle,
+ d.isSetFilterBson() ? d.getFilterBson().length : 0,
+ d.isSetProjectionBson() ? d.getProjectionBson().length : 0);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
+ java.util.Objects.checkIndex(index, plan.size());
+ MongoRangePlanner.checkCancelled(session,
request.getScanContext());
+ TMongoScanDesc range = d.deepCopy();
+ TMongoSplit split = new TMongoSplit();
+ split.setStrategy(plan.family().isEmpty() ? "single" :
"range");
+ split.setField(plan.field());
+ split.setTypeFamily(plan.family());
+ split.setIndex(index);
+ split.setRequested(plan.requested());
+ split.setAdmitted(plan.admitted());
+ split.setActual(plan.size());
+ split.setReason(plan.reason());
+ if (!plan.family().isEmpty()) {
+ range.setVersion(2);
+
split.setFilterBson(MongoSplitPlan.encode(plan.partition(index)));
+ }
+ range.setSplit(split);
+ if (snapshot != null) {
+ range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+ }
+ if (plan.descriptorBudget() > 0 && descriptorSize >
plan.descriptorBudget()) {
+ try {
+ // EXPLAIN may freeze a plan before a later caller
supplies a larger candidate
+ // or projection. Never let that RPC silently exceed
the frozen reservation.
+ int actual = new org.apache.thrift.TSerializer(
+ new
org.apache.thrift.protocol.TBinaryProtocol.Factory()).serialize(range).length;
+ if (actual > plan.descriptorBudget()) {
+ throw new DorisConnectorException("MongoDB
descriptor grew after split plan freeze");
+ }
+ } catch (org.apache.thrift.TException failure) {
+ throw new DorisConnectorException("Cannot size MongoDB
split descriptor");
+ }
+ }
+ return new MongoScanRange(range);
+ }
+
+ @Override
+ public int size() {
+ return plan.size();
+ }
+ };
+ }
+
+ @Override
+ public long streamingSplitEstimate(ConnectorSession session,
ConnectorTableHandle handle,
+ Optional<ConnectorExpression> filter, boolean countPushdown) {
+ return
"single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
"single"))
+ ? -1 : Math.min(8, MongoRangePlanner.splitLimit(session));
+ }
+
+ @Override
+ public ConnectorSplitSource streamSplits(ConnectorSession session,
ConnectorScanRequest request) {
+ return new ConnectorSplitSource() {
+ private Iterator<ConnectorScanRange> ranges;
+ private boolean closed;
+
+ @Override
+ public boolean hasNext() {
+ if (closed) {
+ return false;
+ }
+ MongoRangePlanner.checkCancelled(session,
request.getScanContext());
+ if (ranges == null) {
+ // All probes, admission and boundary validation finish
before the first publication.
+ ranges = planScan(session, request).iterator();
+ }
+ return ranges.hasNext();
+ }
+
+ @Override
+ public ConnectorScanRange next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ return ranges.next();
+ }
+
+ @Override
+ public void close() {
+ closed = true;
+ ranges = Collections.emptyIterator();
+ }
+ };
+ }
+
+ private long descriptorBytes(ConnectorSession session, MongoTableHandle
table,
+ int candidateBytes, int projectionBytes) {
+ if
("single".equals(session.getSessionProperties().getOrDefault("mongodb_split_strategy",
"single"))) {
+ return 0;
+ }
+ // A count-only cap is insufficient: 100 copies of a large
schema/credential/filter can exceed
+ // a split RPC's body limit. Include UTF-8 payloads plus room for
Thrift fields and range BSON.
+ long bytes = 8192L + candidateBytes + projectionBytes;
+ for (String value : List.of(properties.getUri(),
properties.getUsername(), properties.getPassword(),
+ properties.getAuthDatabase(), table.getDatabase(),
table.getName(), table.getUuid(),
+ table.getSchema().getVersion(),
table.getSchema().toBindingsJson())) {
Review Comment:
[P2] Include time-series snapshot options in descriptor admission
For `auto`/snapshot time-series fallbacks, `MongoSnapshot.prepare` emits
`snapshot.timeseries_options_json`, but this estimator never counts it. BE
accepts up to 16384 bytes, so the 8192-byte pad is insufficient. A static
example is a 15000-character `timeField` without `metaField`, with 900 BIGINT
aliases `c0000`..`c0899` mapped to `['v']`: bindings occupy 138627 bytes. With
the short `mongodb://localhost:27017`, `sample.records`, default auth database
and count projection, 228 separate scans reserve 33502320 bytes (below 32 MiB),
while counted strings plus the timeField alone require at least 35051352 bytes.
The long field passes MongoDB's [field
validator](https://github.com/mongodb/mongo/blob/r7.0.28/src/mongo/db/timeseries/timeseries_global_options.h#L39-L44)
and [no-meta default-index
path](https://github.com/mongodb/mongo/blob/r7.0.28/src/mongo/db/catalog/create_collection.cpp#L206-L228)
by inspection. Every fallback now invokes admission, unlike the previous
finding; this
is an omitted serialized field in the estimator. The growth guard is skipped
because the initial estimate equals its reservation. Include snapshot options,
or size the complete descriptor, before admission and cover this
repeated-fallback byte limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]