Gabriel39 opened a new pull request, #70708:
URL: https://github.com/apache/doris/pull/70708
### Background
The native MongoDB catalog reads logical collections through BE cursors and
leaves SQL residual evaluation in Doris. A snapshot read pins one
server-selected timestamp across participating scans. Existing scans use one
cursor per namespace; the connector contract is described in
`fe/fe-connector/fe-connector-mongodb/README.md`.
### What problem does this PR solve?
Issue Number: #70251
Related PR: #70654 (builds on common snapshots and fixes its remaining
pool-key review comment)
PR Type: Feature
Problem Summary:
D08 needs parallel ordinary-collection reads whose partitions cover every
document exactly once at the same snapshot. Runtime session settings now opt
into bounded `range` or conservative `auto` planning; `single` remains the
default. Coverage requires typed boundaries, a disjoint complement, frozen
admission and failure handling; most added lines exercise those contracts. The
pool-key follow-up also makes unequal remaining query deadlines contend for the
configured client limit.
```sql
SELECT /*+ SET_VAR(mongodb_read_concern='snapshot',
mongodb_split_strategy='range',
mongodb_split_count=8) */ count(*) FROM
mongo_native.sample.records;
```
### Details
- **Change map:** Production code +993/-36; tests and fixture documentation
+1801; generated regression output +42; connector documentation +83/-2. Read
the production change in three parts: FE planning/probes, settings and
statement budgets (about 720 lines, `MongoRangePlanner`, `MongoSplitPlan`,
`MongoDriverCatalogClient`); lazy descriptor publication and diagnostics (about
145 lines, `MongoScanPlanProvider`, `MongoScanRange`); versioned wire
validation, native predicate conjunction and stable pool settings (about 130
lines, `TMongoSplit`, `MongoCursorStream`).
- **Scope:** Ordinary collections on qualified MongoDB 7/8 replica sets and
mongos. Scalar range families are ObjectId, signed int32/int64 and BSON Date.
Other types, missing/null and arrays remain covered by the complement. Explicit
view/time-series range requests reject; their existing single-cursor contracts
remain available.
- **Design:** Implements the D08 contract in #70251. Bounded indexed-prefix
sampling supplies balance hints; full sorting/pagination would defeat the
planning budget. The complete half-open boundary/complement table and
requested/admitted/actual counts freeze before publication. Plan limits cover
probes, counts and repeated descriptor bytes at statement/source scope; a
published failure cannot repin or append a full rescan. SQL candidates remain
independent conjuncts and retain Doris residuals.
- **Usage:** The example enables snapshot range reads for one statement. Set
`mongodb_split_field` for a top-level custom field in a statement containing
exactly one MongoDB scan; otherwise `_id` is used. EXPLAIN/Profile report the
selected family, requested/admitted/actual totals and stable decision reason.
- **Default behavior:** `mongodb_split_strategy=single`, count=0,
maximum=256, target=256 MiB, sample rows=4096 and probe timeout=2000 ms. `auto`
retains single for unqualified or small/unknown-cost plans. Settings are
runtime session variables and statement hints; no restart is needed.
- **Compatibility:** Requires matching FE/BE for the new path. New optional
split field 26 and version-2 descriptors make older native readers reject
ranges instead of scanning the full namespace repeatedly. Version-1 single
reads remain valid; no persisted catalog/image/storage format changes.
Rolling-upgrade/downgrade qualification is outside this issue's agreed scope.
- **Security and privileges:** Uses the existing configured MongoDB endpoint
and credential contract. Range probes additionally require `listIndexes` and
`collStats`; missing privileges fail. Real fixture queries use read-only
identities plus the existing snapshot-identity metadata privileges. No new
privilege or outbound HTTP path is introduced.
- **Behavior change:** EXPLAIN and completed Profiles gain split
diagnostics, including on single reads. Configured connect/selection caps now
remain stable across pooled streams; socket/server/local deadlines remain per
stream. Synchronous driver connect/selection may continue for the configured
transport cap after local cancellation.
- **Follow-up plan:** General D06 resource/credential extensions and the
shared IOContext StopFlag change remain excluded. D09/D10 release
soak/performance qualification remains tracked separately in #70251.
### Testing
All runs below are local, against this candidate:
- Preset FE UT runner: 120 Mongo connector tests and 30 session/core tests
passed. Includes randomized full-domain partition checks, exact
int64/Date/ObjectId boundaries, duplicate/complement coverage, sticky failures,
probe cleanup/cancellation, descriptor growth and admission limits.
- Preset ASAN BE UT runner: 70 BSON/reader/native-stream tests passed with
leak and stack-use-after-return checks. The below-transport-cap pool regression
failed before the fix and passed afterward; cancellation after pool setup and
candidate/partition conjunction also have regression tests.
- MongoDB 7.0.28/8.0.20 authenticated replica-set and mongos fixtures:
one-BE fault/oracle runs passed. Source commands verify one T and exactly-once
coverage. Complete multisets cover four time zones, DATETIME(0..6), nine
Decimal precision/scale combinations, pushdown switches and mixed keys.
Independent BSON-source precision and concurrent transactional split-key
oracles passed; command/history failures, genuine history expiration,
cancellation/deadlines, no-rescan recovery and one-client contention passed.
- Repeated range queries: 30 cycles left no live source cursor or connection
growth; both BEs returned to their idle thread counts and memory baseline after
worker retirement. This is a bounded cleanup check, not a soak qualification.
- Two-BE qualification: MongoDB 7/8 replica-set/mongos exact-T oracles
passed and required actual native partitions on both BEs. Together with one-BE
runs, this covers 2240 complete multiset comparisons, 346176 independent
precision cells and 160 exact-T transactional comparisons.
- Preset regression runner: all six MongoDB suites passed with zero
failed/fatal/skipped scripts. The new split golden was generated by the
framework and verified on both MongoDB 7 and 8.
### Notes for Reviewers
The sampling distribution can be skewed and the complement adds source work;
no speedup or release-soak result is claimed. The retained external split suite
and fault/oracle script require explicitly configured disposable fixtures;
default PR external jobs do not provision them. D08 planning budgets are local
to a statement and pinned source, rather than a new cross-statement/cross-BE
source service.
### Release note
Native MongoDB scans can opt into bounded snapshot range planning for
ordinary collections. Single-cursor reads remain the default; new session
variables control range selection and planning limits.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: Yes (connector/fixture README included;
user-facing catalog documentation can incorporate these qualified options
before release)
- Pre-submission status: Ready
### AI Disclosure
- Tool: Codex / GPT-6
- Generated: implementation, unit/regression tests, fixtures, documentation
and this description
- Human review: not stated
- Prompts: Complete D08 and the remaining merged-PR comment, add sufficient
tests with extra attention to temporal and Decimal precision, preserve
D06/StopFlag exclusions, and self-review/fix findings before submitting the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]