Gabriel39 opened a new pull request, #70327:
URL: https://github.com/apache/doris/pull/70327
### Background
The native MongoDB dependency preparation (#70249) provides the FE Java
driver and BE C driver, but does not register a usable Catalog. Doris needs a
native metadata and scan path that preserves BSON types and reads collection,
view, and time-series namespaces directly.
### What problem does this PR solve?
Issue Number: #70251
Related PR: #70249 (client dependencies)
PR Type: Feature
Problem Summary:
Implement D01, D03, and D04 of the native MongoDB Catalog roadmap, plus the
minimal single-Split descriptor needed to connect them. Users can create a
`type=mongodb` Catalog and query ordinary collections, standard views, and
time-series measurements through a native C++ reader. FE pins explicit,
validator, or document schemas; concrete `DataTypeSerDe` implementations
convert BSON without losing int64 precision, ObjectId bytes, or binary subtype
identity.
### Details
- **Change map:** 2,958 added production/build lines, 1,808 test/fixture
lines, 239 generated regression-result lines, and 142 documentation lines (34
removed lines total). FE adds the Connector SPI provider, catalog-owned
metadata client, schema resolution, stable handles, plugin packaging, and a
single native scan range. BE adds the cursor/pool lifecycle and BSON
view/context, with decoding in existing concrete SerDes. Tests include
deterministic MongoDB fixtures, FE/BE unit tests, and a SQL regression suite
with generated expected results.
- **Scope:** read-only single-cursor scans of ordinary collections, standard
views, and time-series collections. Views retain their own pipeline/collation
and work with view-only `find` access; internal time-series buckets are hidden.
All SQL filters, LIKE/REGEXP, joins, windows, aggregates, sorting, and LIMIT
are evaluated in Doris. Predicate/projection pushdown, parallel range planning,
common snapshots, and shared-source admission remain separate work items.
- **Design:** FE sends stable namespace/configuration/schema values through
`TMongoScanDesc`. BE owns bounded client-pool leases and cursors with RAII and
closes them on EOF, early termination, cancellation, and errors. BSON
conversions validate physical tags and exact numeric/date ranges; compound
columns roll back on failure. Canonical Extended JSON and raw BSON provide
document access. Reader profiles expose read/decode/filter time, rows, and BSON
bytes.
- **Usage:** `CREATE CATALOG mongo_native PROPERTIES ("type"="mongodb",
"mongodb.uri"="mongodb://mongo.example:27017", "mongodb.databases"="sample")`.
See the connector README for explicit schemas, credentials, supported mappings,
and fixture setup. ObjectId uses `VARBINARY(12)`; use `FROM_BINARY` for
supported hex/string analysis because direct VARBINARY operators retain their
existing engine restrictions.
- **Default behavior:** `auto` chooses a namespace's explicit mapping, a
fully representable ordinary-collection validator, or document mode; it never
samples source rows. All mapped fields are nullable and are not declared keys.
Mismatches fail by default; optional top-level null substitution retains atomic
nested writes. Dates use UTC. Batch size and query timeout use session
settings; connection/read defaults belong to the Catalog.
- **Compatibility:** the new Catalog uses the dependencies introduced by
#70249. Existing connectors and their serialization paths retain their
behavior. Cross-version capability negotiation is outside this change.
### Testing
All checks below ran locally:
- Complete FE build (`build.sh --fe`) and complete ASAN BE build (`build.sh
--be`).
- FE connector unit tests: **26 passed**, including schema resolution,
metadata/driver cursor cleanup, handle serialization, session policies and
native scan descriptors. Related FE core and legacy MongoDB connector tests
also passed through `run-fe-ut.sh`.
- BE unit tests through `run-be-ut.sh`: **136 passed** across BSON
SerDe/reader/real-driver error paths and existing decoded-value/name/VARBINARY
coverage.
- Native MongoDB SQL regression against the deployed FE/BE and MongoDB
8.0.20: generated expected output, then independently compared it; final rerun
**1 suite passed, 0 failed/fatal/skipped**.
- FE Checkstyle: zero violations. Clang-format **16.0.6** on all **31**
affected C++ source/header files, build hygiene, and the repository clang-tidy
check passed. Static analysis used compilation commands from the completed
BE/UT builds, expanded from unity commands for individual files.
- Full diff/self-review and private-context audit completed. The branch
merges cleanly with the current master checked before submission.
### Notes for Reviewers
Start with `MongoSchema`, `MongoScanPlanProvider`, `mongodb_reader.cpp`, and
the BSON SerDe tests. The regression suite covers all three logical object
kinds, schema refresh, historical validator violations, source permissions,
exact/special BSON values, nested rollback, SQL analysis, batch boundaries, and
repeated early cursor closure. The current local integration environment uses
MongoDB 8.0.20; MongoDB 7, replica-set/mongos topology qualification,
TLS/authentication matrices, performance, and soak evidence remain in the
tracking issue.
### Release note
Add a native read-only MongoDB Catalog supporting
explicit/validator/document schemas, native BSON decoding, and single-cursor
reads of collections, standard views, and time-series measurements.
### Check List (For Author)
- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; connector and fixture READMEs are
included. Public user-documentation and release qualification remain tracked in
#70251.
- Pre-submission status: Ready for review; local validation listed above, CI
pending.
### AI Disclosure
- Tool: Codex (GPT-6).
- Generated: implementation, tests, documentation, and this description.
- Human review: pending maintainer review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
