morningman opened a new pull request, #66399: URL: https://github.com/apache/doris/pull/66399
### What problem does this PR solve? Issue Number: close #xxx Problem Summary: Doris cannot read [Apache Fluss (incubating)](https://fluss.apache.org/) tables. This adds a **fluss catalog**, reading all three shapes a fluss table can take: | Table | How it is read | |---|---| | log table | the fluss change log, one scan range per bucket | | primary-key table | a kv snapshot plus the change log after it, merged by key | | tiered table (`table.datalake.enabled`) | its **paimon lake plus the log written after it**, in one scan | For a tiered table the lake half is not re-implemented: it is planned by the **paimon connector**, on the snapshot the fluss coordinator pinned, so that half gets native ORC/Parquet, deletion vectors and the file cache for free — and the fluss plugin ships **no paimon dependency at all** (it borrows the paimon plugin through `createSiblingConnector`, the way hive borrows iceberg and hudi). A tiered table also exposes `tbl$lake`, which reads the lake alone. For a tiered **primary-key** table the two halves are merged BY KEY, natively in BE: FE wraps each lake split with the key set of that bucket's log tail, the BE reader drops the lake rows that tail replaced before materialization, and the tail is replayed once as its own range. ```sql CREATE CATALOG fluss PROPERTIES ( "type" = "fluss", "fluss.bootstrap.servers" = "host:9123" ); SELECT * FROM fluss.db.tbl; -- lake + log, merged SELECT * FROM fluss.db.`tbl$lake`; -- the lake alone ``` ### Reviewing this 35 commits. **Five of them are not in the fluss module**, stand on their own, and are first in the history — they can be reviewed independently of everything else: | Commit | Lands in | Why it is separate | |---|---|---| | `[fix](be) Stop exporting the statically linked RocksDB symbols` | `be/src/service/CMakeLists.txt` | Fixes a Doris bug, not a fluss one. `doris_be` exports 4840 statically linked rocksdb symbols, so **any** JNI library carrying its own RocksDB resolves 2576 of them into doris_be's copy — across two different libstdc++ string ABIs. An object built with one layout, used by functions compiled for another ⇒ `bad_alloc` through a JNI frame ⇒ the BE process aborts. Fluss's embedded frocksdbjni is simply the first library to hit it. | | `[fix](be) Pick the table reader per scan range, not per scan node` | `be/src/exec/scan/file_scanner_v2.{h,cpp}` | `_open_impl` builds one `_table_reader` from the **first** split and every later split reuses it. A scan node holding two `table_format_type`s — the shape any union read has — hands the second kind to the first kind's reader. The symptom is non-deterministic, because which splits share a scanner is the engine's choice. | | `[fix](paimon) Claim the table handles this connector produces` | `fe/fe-connector/fe-connector-paimon` | `Connector.ownsHandle` defaults to false; iceberg and hudi override it, paimon never did. Any connector using paimon as a sibling is told "not mine" about every handle paimon itself produced. | | `[feat](paimon) Say which bucket a scan range came from` | `fe/fe-connector/fe-connector-paimon` | An FE-only scan range property, not forwarded to BE, so BE is unaffected. Fluss's own lake SPI treats `LakeSplit.bucket()` as first class; this is the same fact on the Doris side. | | `[feat](connector) Let a connector name the columns its reader must read` | `fe/fe-connector/fe-connector-api` + `fe/fe-core` | The one engine-side change. A connector whose reader merges by key needs the key columns in the scan's tuple whether or not the user selected them. **This is not a new mechanism** — Doris's own aggregate and merge-on-write tables do exactly this (`preserveExtraStorageKeySlots` + `extra_key_column_slot_ids`), for the same reason. The new branch sits beside that one, before the same `removeIf`. The SPI method defaults to an empty set, so every other connector is unaffected. | The remaining 30 commits are the connector itself, grouped so that each group compiles, runs and has its own acceptance: 1. **metadata** — module, type mapping, catalog/table/partition metadata, the e2e docker environment 2. **log tables** — thrift payload, FE planning, the BE java scanner, the BE C++ glue 3. **primary-key tables** — kv snapshot + change log 4. **the lake** — `$lake` through the paimon sibling, and lake-plus-log for log tables 5. **merging a primary-key table's halves by key** — FE planning, the tail replayer, the BE suppression reader 6. **coverage** — a partition-column-type gate, and five more e2e suites ### Things worth saying out loud - **This depends on `fluss 1.0-SNAPSHOT`**, resolved from the Apache snapshot repository `fe/pom.xml` already declares. Fluss 1.0 is not released and the APIs this uses are not in 0.9.1. The coordinates switch to the release when there is one. If CI goes red while this is green locally, compare the fluss snapshot timestamps first. - **The e2e suites do not run in CI yet, on purpose.** They need a fluss cluster, and the two docker images are built locally because fluss 1.0 is unreleased; `enableFlussTest` defaults to false and the external pipeline's conf deliberately does not set it. That switches on with the fluss release. - Two pre-existing bugs in neighbouring code were found by this work and are **not** fixed here, so as not to widen the diff. Both reproduce without fluss: an equality predicate on a microsecond `TIMESTAMP` pushed into paimon matches no row (a plain paimon catalog over the same warehouse behaves the same), and `java-common`'s VARBINARY read-back decodes what it wrote as a StringView, so it always reads zeros (the write side and BE agree; only the Java read-back is wrong, and no production path takes it). - `TIME` is mapped to `UNSUPPORTED` rather than to a string or to elapsed millis: Doris has no type meaning what fluss's TIME means, and handing back a plausible value of a different meaning is not something a later error would catch. The paimon and iceberg connectors mark their own TIME the same way, which also keeps `tbl` and `tbl$lake` agreeing on one schema. - Only `filesystem` is supported as the paimon lake catalog for now. HMS and REST lake catalogs, predicate pass-through into the lake half, and the write path are follow-ups. ### Release note Support reading Apache Fluss tables through a new `fluss` catalog, including tables tiered into a paimon lake, which are read as the lake plus the change log written after it. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test `regression-test/suites/external_table_p0/fluss` — 12 suites, 0 skipped, green on two consecutive runs against a real fluss + flink + paimon cluster (`docker/thirdparties/docker-compose/fluss`): catalog, log tables, primary-key tables, lake-only, lake+log, primary-key merge, nested complex types, partition-column types, 100k rows, compound predicates, and empty/negative/misc. Unit tests: `fe-connector-fluss` 192, `fe-connector-paimon` 511, `fluss-scanner` 36, `fe-connector-api` + `fe-core` (the must-read-columns chain) 36 — all 0 skipped. BE: `FileScannerV2*:FileScannerTest*:Fluss*` 55 and `Paimon*:*Iceberg*:*EqualityDelete*` 227, all passing. Every new behaviour was mutation-tested: the change was inverted and the suite had to go red. That includes the end-to-end ones — the merge was mutated in C++, in BE-java and in FE, rebuilt and redeployed each time, and each mutation turned a suite red. - Behavior changed: - [x] No. <!-- New catalog type; the one engine-side SPI addition defaults to an empty set, so existing connectors are unaffected. --> - Does this need documentation? - [x] Yes. <!-- A doris-website PR for the fluss catalog page will follow. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
