Gabriel39 opened a new issue, #70251: URL: https://github.com/apache/doris/issues/70251
## Description Track the implementation of a **native, read-only MongoDB Catalog** on Doris master, from client dependencies and the first single-cursor query to safe pushdown, parallel scans, and release qualification. **Design:** [Complete design PDF — Chinese, 47 pages, 16 chapters, four diagrams](https://github.com/Gabriel39/incubator-doris/blob/2d7a6287f05c823af9b682cb961145f6964556c2/design/mongodb/native-mongodb-catalog-design.pdf). The PDF covers the architecture, FE/BE contracts, Schema management, BSON-to-Doris mapping, predicate semantics, split planning, consistency, resource ownership, SQL analytics, regression coverage, and the implementation roadmap. It is an RFC: proposed interfaces and settings are not claims of implemented functionality. The code-reading baseline is [c5bfd6819f393bde37b73147abddc2f9a44641cb](https://github.com/apache/doris/commit/c5bfd6819f393bde37b73147abddc2f9a44641cb). **Related PR:** #70249 prepares the client dependencies and packaging. It is currently open; it does not register a usable Catalog or Scanner. Keep D00 unchecked until integration and artifact validation pass. ## Use cases and initial scope Query MongoDB collections through SQL for filters, joins, aggregation, windows, JSON/array analysis, and pattern matching. Doris evaluates operators that are not safely pushed down. Repeated full scans and indexed text analytics can use an explicit import into a Doris internal table. - FE: a MongoDB connector using the existing Connector SPI and Java Sync Driver for metadata, Schema discovery, and scan planning. - BE: a native C++ Reader using libmongoc/libbson with RAII; BSON conversion belongs in `DataTypeSerDe`. - Initial qualification targets: ordinary collections on MongoDB 7/8, replica sets and mongos, with an explicitly validated compatibility matrix. - Read-only access. Writes, source DDL, CDC, and cross-cluster global snapshots are outside the initial release. ## Work items ### M0 — Minimal single-cursor path - [ ] **D00 — Client dependencies and build artifacts** (W1; PR #70249): land the dependency preparation, validate pinned drivers and transitive dependencies, licenses, complete FE/BE builds, plugin packaging, supported architectures, TLS/SASL/SRV, and compression. A dependency archive alone must not activate an unfinished connector. - [ ] **D01 — Catalog and Schema management** (W1–W4): register the FE connector, discover databases/collections, map source validators, support explicit overrides and document mode, retain source logical types, and pin Schema/configuration versions. Define refresh/evolution behavior; a validator must not be treated as proof that historical rows conform. - [ ] **D02 — Versioned query and single-Split contract** (W1–W3): serialize stable configuration/Schema versions, collection identity, field bindings, projection/filter descriptions, and read policy. Wire the scan descriptors and capability negotiation. Active cursors, driver sessions, and client objects remain local to their owners. - [ ] **D03 — Native BE Reader** (W2–W4): integrate C++ pool leases, cursor/getMore, batching, deadlines, cancellation, error translation, ScanNode execution, and Profile metrics. Release every lease/cursor on success, cancellation, decoding failure, and network errors. - [ ] **D04a — Basic BSON `DataTypeSerDe` support** (W2–W4): implement initial scalar/Nullable conversions and ObjectId as 12 raw bytes in `VARBINARY(12)`; preserve source type/subtype metadata and copy borrowed BSON bytes before their backing storage is released. **M0 gate:** Catalog creation, discovery, and end-to-end SELECT work with one Split/cursor; basic authentication, cancellation, type errors, cleanup, and complete build/package validation pass. Filters remain in Doris until their pushdown contracts are verified. ### M1 — Complete single-cursor queries - [ ] **D04b — Remaining first-release type mappings** (W5–W6): complete the design's numeric/Decimal128, UTC date, binary, Extended JSON, array, struct, and missing/null rules; validate precision/overflow, recursive offsets, row-level rollback, and memory limits. Declare unsupported mappings explicitly. - [ ] **D05 — Safe predicate/LIKE pushdown and projection** (W4–W7): classify translations as exact/superset/unsupported; preserve SQL three-valued logic, CAST, arrays, collation, and strict error semantics. Keep required residual predicates, implement conservative LIKE translation and column pruning, and gate behavior with Session variables. Verify the existing SPI/applyFilter path cannot remove predicates incorrectly. - [ ] **D06 — Read policy, security, and resource control** (W4–W7): implement Catalog defaults plus statement-pinned Session overrides for read preference/concern, sensitive configuration handling, TLS/credential versioning, pool isolation, batch/memory/time limits, and shared-source admission. Distinguish scanner-instance concurrency from query-wide and source-wide budgets. **M1 gate:** complete result-multiset and error-semantics differential tests pass with pushdown enabled/disabled; type mappings, permissions, policy/pool isolation, and failure-path cleanup pass. Any separately delivered M0/M1 preview must document its single-cursor and consistency limits. ### M2 — Parallel scans and a common snapshot - [ ] **D07a — Snapshot feasibility prototype** (**W1–W2, performed early**): demonstrate a server-selected snapshot time T, its FE-to-BE representation, and the actual libmongoc command path that reads the same `atClusterTime` on replica sets and mongos. Majority reads alone do not satisfy this gate. - [ ] **D07b — Snapshot consistency implementation** (W7–W9): pin the read contract across Splits and multiple collections in the same cluster; define history-expiration, identity/topology-change, cancellation, and retry behavior. Fail explicitly on unavailable/unsupported snapshots instead of silently downgrading consistency. - [ ] **D08 — Bounded range/auto Split planning and execution** (W8–W10): implement field/type-family selection, requested/admitted/actual Split counts, query/source budgets, bounded sampling, duplicate-boundary removal, and half-open ranges with unbounded ends. Preserve a disjoint complement for missing/null/array/other-type values when the scalar type family is not proven exhaustive. Freeze the complete plan before publishing any Split; expose decisions and fallback reasons in EXPLAIN/Profile. An already published plan must not fall back to a full single-Split rescan. **M2 gate:** every source document belongs to exactly one partition; the complete parallel result multiset matches a single-Split read at the same snapshot. Snapshot history failures, planning limits, scanner scheduling, and shared-source admission are verified before enabling parallel scans. ### Continuous validation and release qualification - [ ] **D09 — Regression and fault matrix** (W1–W12, delivered with each implementation): provide controlled MongoDB services with environment-injected credentials; cover Schema sources/evolution, all mappings, SQL analytics, exact and superset pushdown, NULL/CAST/array counterexamples, malformed/deep/wide documents, duplicate/skewed split boundaries, complement partitions, authentication/TLS, topology changes, concurrent writes, timeout/cancellation, and resource cleanup. Add deterministic fixtures and reproducible differential checks. - [ ] **D10 — Performance, compatibility, and release evidence** (W11–W12): establish repeatable single-cursor/pushdown/parallel baselines, source-side budgets, complete build/artifact and upgrade qualification, supported driver/server/authentication/architecture combinations, EXPLAIN/Profile coverage, and user/operations documentation. Target at least 1,000 query/cancel/failure cycles and a 24-hour mixed-workload soak without continuously growing memory, threads, cursors, or connections. Run FE Checkstyle, clang-format 16 on affected BE files, and the relevant UT/regression checks. **Release gate:** M1/M2 correctness and resource criteria plus all mandatory compatibility cases pass; archive reproducible CI, fault, performance, and upgrade results. Week numbers are suggested relative development windows, not release commitments. The design reserves W13–W14 for blocking fixes and affected revalidation. ## Dependency order ```mermaid flowchart TD P["D00 dependencies + D02 query contract"] --> A["D01 Catalog / Schema"] P --> B["D03 native Reader"] P --> C["D04a basic DataTypeSerDe"] A --> M0["M0: single-cursor path"] B --> M0 C --> M0 P -. "validate in W1-W2" .-> S0["D07a snapshot prototype"] M0 --> T["D04b complete type semantics"] T --> F["D05 safe pushdown / projection"] M0 --> R["D06 read policy / resource admission"] F --> M1["M1: complete single-cursor queries"] R --> M1 M1 --> S["D07b fixed snapshot implementation"] S0 --> S M1 --> Q["D08 range / auto planner"] S --> M2["M2: common snapshot + complete partitions + admission"] Q --> M2 M2 --> RC["D10 release qualification"] ``` Independent branches may be developed in parallel. Planner-only tests can start before the snapshot implementation is complete, but multi-Split reads require both snapshot and partition/admission gates. D09 runs throughout every branch and milestone. ## Follow-up capabilities, tracked separately after the initial release - Views and time-series collections: independently validate command/snapshot capabilities; time-series splitting uses the logical collection/timeField and does not assume a unique `_id`. - Opt-in, bounded Schema inference with pinned versions; samples do not prove global type homogeneity. - Whitelisted REGEXP, bounded runtime filters, LIMIT/TopN, and aggregation pushdown, each with a separate semantic contract, switch, differential suite, and benchmark. - Source-side search interfaces; CDC, VARIANT document storage, and a large Schema registry require separate designs and lifecycles. ## Tracking convention Keep a work item unchecked until its implementation PRs and acceptance evidence are linked and verified. Link follow-up implementation PRs here and maintain the capability/support matrix as scope changes. The dependency preparation PR is not evidence that native MongoDB queries already work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
