Gabriel39 opened a new pull request, #70708:
URL: https://github.com/apache/doris/pull/70708

   ### Background
   
   The native MongoDB catalog reads logical collections through BE cursors and 
leaves SQL residual evaluation in Doris. A snapshot read pins one 
server-selected timestamp across participating scans. Existing scans use one 
cursor per namespace; the connector contract is described in 
`fe/fe-connector/fe-connector-mongodb/README.md`.
   
   ### What problem does this PR solve?
   
   Issue Number: #70251
   Related PR: #70654 (builds on common snapshots and fixes its remaining 
pool-key review comment)
   PR Type: Feature
   
   Problem Summary:
   
   D08 needs parallel ordinary-collection reads whose partitions cover every 
document exactly once at the same snapshot. Runtime session settings now opt 
into bounded `range` or conservative `auto` planning; `single` remains the 
default. Coverage requires typed boundaries, a disjoint complement, frozen 
admission and failure handling; most added lines exercise those contracts. The 
pool-key follow-up also makes unequal remaining query deadlines contend for the 
configured client limit.
   
   ```sql
   SELECT /*+ SET_VAR(mongodb_read_concern='snapshot', 
mongodb_split_strategy='range',
                     mongodb_split_count=8) */ count(*) FROM 
mongo_native.sample.records;
   ```
   
   ### Details
   
   - **Change map:** Production code +993/-36; tests and fixture documentation 
+1801; generated regression output +42; connector documentation +83/-2. Read 
the production change in three parts: FE planning/probes, settings and 
statement budgets (about 720 lines, `MongoRangePlanner`, `MongoSplitPlan`, 
`MongoDriverCatalogClient`); lazy descriptor publication and diagnostics (about 
145 lines, `MongoScanPlanProvider`, `MongoScanRange`); versioned wire 
validation, native predicate conjunction and stable pool settings (about 130 
lines, `TMongoSplit`, `MongoCursorStream`).
   - **Scope:** Ordinary collections on qualified MongoDB 7/8 replica sets and 
mongos. Scalar range families are ObjectId, signed int32/int64 and BSON Date. 
Other types, missing/null and arrays remain covered by the complement. Explicit 
view/time-series range requests reject; their existing single-cursor contracts 
remain available.
   - **Design:** Implements the D08 contract in #70251. Bounded indexed-prefix 
sampling supplies balance hints; full sorting/pagination would defeat the 
planning budget. The complete half-open boundary/complement table and 
requested/admitted/actual counts freeze before publication. Plan limits cover 
probes, counts and repeated descriptor bytes at statement/source scope; a 
published failure cannot repin or append a full rescan. SQL candidates remain 
independent conjuncts and retain Doris residuals.
   - **Usage:** The example enables snapshot range reads for one statement. Set 
`mongodb_split_field` for a top-level custom field in a statement containing 
exactly one MongoDB scan; otherwise `_id` is used. EXPLAIN/Profile report the 
selected family, requested/admitted/actual totals and stable decision reason.
   - **Default behavior:** `mongodb_split_strategy=single`, count=0, 
maximum=256, target=256 MiB, sample rows=4096 and probe timeout=2000 ms. `auto` 
retains single for unqualified or small/unknown-cost plans. Settings are 
runtime session variables and statement hints; no restart is needed.
   - **Compatibility:** Requires matching FE/BE for the new path. New optional 
split field 26 and version-2 descriptors make older native readers reject 
ranges instead of scanning the full namespace repeatedly. Version-1 single 
reads remain valid; no persisted catalog/image/storage format changes. 
Rolling-upgrade/downgrade qualification is outside this issue's agreed scope.
   - **Security and privileges:** Uses the existing configured MongoDB endpoint 
and credential contract. Range probes additionally require `listIndexes` and 
`collStats`; missing privileges fail. Real fixture queries use read-only 
identities plus the existing snapshot-identity metadata privileges. No new 
privilege or outbound HTTP path is introduced.
   - **Behavior change:** EXPLAIN and completed Profiles gain split 
diagnostics, including on single reads. Configured connect/selection caps now 
remain stable across pooled streams; socket/server/local deadlines remain per 
stream. Synchronous driver connect/selection may continue for the configured 
transport cap after local cancellation.
   - **Follow-up plan:** General D06 resource/credential extensions and the 
shared IOContext StopFlag change remain excluded. D09/D10 release 
soak/performance qualification remains tracked separately in #70251.
   
   ### Testing
   
   All runs below are local, against this candidate:
   
   - Preset FE UT runner: 120 Mongo connector tests and 30 session/core tests 
passed. Includes randomized full-domain partition checks, exact 
int64/Date/ObjectId boundaries, duplicate/complement coverage, sticky failures, 
probe cleanup/cancellation, descriptor growth and admission limits.
   - Preset ASAN BE UT runner: 70 BSON/reader/native-stream tests passed with 
leak and stack-use-after-return checks. The below-transport-cap pool regression 
failed before the fix and passed afterward; cancellation after pool setup and 
candidate/partition conjunction also have regression tests.
   - MongoDB 7.0.28/8.0.20 authenticated replica-set and mongos fixtures: 
one-BE fault/oracle runs passed. Source commands verify one T and exactly-once 
coverage. Complete multisets cover four time zones, DATETIME(0..6), nine 
Decimal precision/scale combinations, pushdown switches and mixed keys. 
Independent BSON-source precision and concurrent transactional split-key 
oracles passed; command/history failures, genuine history expiration, 
cancellation/deadlines, no-rescan recovery and one-client contention passed.
   - Repeated range queries: 30 cycles left no live source cursor or connection 
growth; both BEs returned to their idle thread counts and memory baseline after 
worker retirement. This is a bounded cleanup check, not a soak qualification.
   - Two-BE qualification: MongoDB 7/8 replica-set/mongos exact-T oracles 
passed and required actual native partitions on both BEs. Together with one-BE 
runs, this covers 2240 complete multiset comparisons, 346176 independent 
precision cells and 160 exact-T transactional comparisons.
   - Preset regression runner: all six MongoDB suites passed with zero 
failed/fatal/skipped scripts. The new split golden was generated by the 
framework and verified on both MongoDB 7 and 8.
   
   ### Notes for Reviewers
   
   The sampling distribution can be skewed and the complement adds source work; 
no speedup or release-soak result is claimed. The retained external split suite 
and fault/oracle script require explicitly configured disposable fixtures; 
default PR external jobs do not provision them. D08 planning budgets are local 
to a statement and pinned source, rather than a new cross-statement/cross-BE 
source service.
   
   ### Release note
   
   Native MongoDB scans can opt into bounded snapshot range planning for 
ordinary collections. Single-cursor reads remain the default; new session 
variables control range selection and planning limits.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes (connector/fixture README included; 
user-facing catalog documentation can incorporate these qualified options 
before release)
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Codex / GPT-6
   - Generated: implementation, unit/regression tests, fixtures, documentation 
and this description
   - Human review: not stated
   - Prompts: Complete D08 and the remaining merged-PR comment, add sufficient 
tests with extra attention to temporal and Decimal precision, preserve 
D06/StopFlag exclusions, and self-review/fix findings before submitting the PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to