yunfengzhou-hub opened a new pull request, #10833: URL: https://github.com/apache/paimon/pull/10833
### Purpose Closes #10692 and closes #10693, part of #10690. Implements [PIP-39](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/398000424) for Flink, plus the engine-neutral table-format contract it depends on. A lake-stream table keeps its history in Paimon while a pluggable Lakestream Store serves its most recent writes, so a query can return data that is seconds old instead of waiting for the next snapshot. Fluss is the first store. The catalog option `lakestream.store` names the store. The table option `lakestream.enabled` marks a table as a lake-stream table and stays mutable, so `ALTER TABLE` can turn the mode off. While it is `true`, `renameTable` and `replaceTable` are rejected, because both break the store's side of the table; `JdbcCatalog` and `RESTCatalog` override those methods without calling super, so the rejection has to be repeated in each. The store is discovered through Flink's own `CatalogFactory` SPI, with the catalog options prefixed with `<identifier>.` forwarded after the prefix is stripped, so Paimon compiles against no store class. It is created on first use rather than together with the catalog, so an unreachable store still leaves the catalog able to resolve table metadata and to turn the mode off again. Reading `my_table$rt` is served by the store, reading `my_table` keeps returning Paimon snapshots, and a regular write goes to the store's sink. A query hint cannot move a write between Paimon and the store, and a write planned by a catalog that names no store is rejected rather than quietly written to Paimon, which would leave the two holding different values for the same keys. Adds `docs/docs/concepts/lake-stream.md` for the table semantics and `docs/docs/flink/lake-stream.md` for the Flink configuration, the SQL surface and a Fluss walkthrough. A Flink CDC pipeline does not plan through the SQL sink, so it still writes straight to Paimon; the Flink page documents the precaution. ### Tests - `CatalogTestBase` gains cases for the identity guards, so they run for every catalog implementation and not only the filesystem one. - `LakeStreamCatalogTest` covers store discovery and the `$rt` interception, including branch suffixes, Paimon system tables, time travel and a missing store-side table. - `LakeStreamTableFactoryTest` covers source and sink routing, query hints, the format-table rejection, and the missing-store rejection in both catalog and connector mode. - `LakeStreamRoutingITCase` runs the behaviour end to end over SQL in batch and in streaming mode, against a fake store registered through the same `CatalogFactory` SPI a real store uses. - `LakeStreamFlussITCase` runs it against a real Fluss mini cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
