JunRuiLee opened a new pull request, #8819: URL: https://github.com/apache/paimon/pull/8819
### Purpose Introduce a portable, versioned JSON description of a table — `TableDescriptor` — so a non-Java runtime (e.g. a paimon-rust / paimon-cpp reader embedded in a compute engine) can rebuild a table for scan/read **without a catalog lookup**. Today a Java engine can serialize a `Table` via native Java serialization, but that blob is not consumable by non-Java readers (it carries JVM-only state such as `FileIO` / `CatalogEnvironment`). As a result, an embedding engine has to re-resolve the table through a catalog on the reader side, which can observe a different schema/snapshot than the one the planner used, and ties the reader to a derived filesystem-catalog layout. This PR adds a slim, language-neutral contract that captures only what a reader needs: - `org.apache.paimon.table.TableDescriptor` (`paimon-api`, `@Experimental`): `version`, `path`, the full `TableSchema` (identical to an on-disk `schema/schema-N`), optional `options` / `database` / `name` / `branch`. Serialized through `JsonSerdeUtil` so the nested `TableSchema` uses Paimon's custom serde; unknown fields are ignored for forward compatibility; null fields are omitted to keep the payload slim. - `org.apache.paimon.table.TableDescriptorSerializer` (`paimon-core`, `@Experimental`): builds a descriptor from a resolved `FileStoreTable` (`location()` + `schema()`), and serializes it to JSON. A non-main branch is carried verbatim so a reader can fail fast rather than silently read `main`; database/name must be set together or not at all. The reader side (in the paimon-rust cross-language reader) consumes the same JSON and reconstructs the table directly; a byte-identical golden fixture (`table-descriptor-v1.json`) pins the wire form on both sides. ### Tests - `TableDescriptorTest` (paimon-api): JSON round-trip; nested `TableSchema` uses Paimon's custom serde; unknown fields ignored. - `TableDescriptorSerializerTest` (paimon-core): build from a `FileStoreTable`; round-trip; partial-identifier rejection; non-main branch carried through. - `TableDescriptorGoldenTest` (paimon-core): producer output equals the checked-in canonical golden, and the golden parses back — the exact wire form the cross-language reader is tested against. ### API and Format - New `@Experimental` public API: `TableDescriptor` (paimon-api) and `TableDescriptorSerializer` (paimon-core). - New JSON descriptor format, versioned (`version = 1`). No change to any existing on-disk or wire format. ### Documentation None yet (experimental API). Docs can follow once the contract stabilizes. --- Motivated by the Apache Doris Paimon-external-table integration (apache/doris#65883), where the compute node currently re-resolves the table through a catalog on the backend; this contract lets it consume the FE-planned table instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
