hbgstc123 opened a new issue, #8810: URL: https://github.com/apache/paimon/issues/8810
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation ### Background Chain tables use three branches — `main`, `snapshot`, and `delta` — each maintaining its own independent schema log (`schema/schema-N`). The current documentation requires users to keep schemas consistent across all three branches manually (e.g. when setting up fallback branches or partition expiration options). Today, a single logical `ALTER TABLE` on the base table only commits schema to **one** branch. There is no catalog-level coordination to update main, snapshot, and delta atomically. ### Problem Because each branch commits schema independently via `SchemaManager.commit()` → `FileIO.tryToWriteAtomic`, partial failures can leave the chain group in an inconsistent state: 1. **Immediate read failure**: `FallbackReadFileStoreTable.validateSchema()` / `ChainGroupReadTable.newScan()` checks row type and primary key consistency across branches. If e.g. `ADD COLUMN` succeeds on `main` but fails on `snapshot`/`delta`, the entire chain table becomes unreadable at scan planning time. 2. **Operational burden**: Users must remember to run the same DDL three times (main + snapshot + delta), which is error-prone and not enforced by the catalog. Example failure after partial `ADD COLUMN`: ``` main latest: schema-11, fields = [a, b, c] snapshot latest: schema-10, fields = [a, b] delta latest: schema-10, fields = [a, b] ``` → chain reader fails with "does not have the same row type" before any data file is read. ### Current behavior (as of master) - `AbstractCatalog.alterTable` delegates to `alterTableImpl` for a single identifier; no chain-table-aware multi-branch sync. - `FileStoreTableFactory.createChainTable` loads snapshot/delta schemas from their respective branch-local `SchemaManager`. - Docs explicitly state: *"Chain table should ensure that the schema of each branch is consistent."* - No rollback if multi-branch DDL is attempted manually and fails midway. ### Proposed direction (high level) We are exploring a design where: 1. **Main branch schema log becomes the single canonical source** for shared fields / field IDs / PK / partition keys. 2. **One atomic write** to `main/schema/schema-N` publishes the schema for the entire chain group. 3. **Branch-specific options** (e.g. different `bucket` per branch) are stored in the canonical main schema and materialized per branch at read/write time. 4. **Legacy schemas before a cutover point** continue to be resolved from branch-local schema files for backward compatibility. Key goals: - One `ALTER TABLE` on the base table atomically evolves the chain group schema. - Readers observe either the old schema or the new schema, never a mixed state. - Preserve per-branch bucket layout and branch-local options. - No change to `TableSchema` JSON top-level structure. Non-goals for an initial version: - Supporting old readers after canonical protocol activation. #### Example: canonical main schema with per-branch options Shared columns / PK / partition keys live in the top-level `TableSchema`. Per-branch effective options are stored as JSON strings in `options` (exact key names TBD): ```json { "id": 25, "fields": [ {"id": 0, "name": "id", "type": "BIGINT NOT NULL"}, {"id": 1, "name": "dt", "type": "STRING NOT NULL"} ], "partitionKeys": ["dt"], "primaryKeys": ["id", "dt"], "options": { "chain-table.enabled": "true", "scan.fallback-snapshot-branch": "snapshot", "scan.fallback-delta-branch": "delta", "bucket": "64", "bucket-key": "id", "chain-table.member-options.snapshot": "{\"bucket\":\"32\",\"bucket-key\":\"id\"}", "chain-table.member-options.delta": "{\"bucket\":\"128\",\"bucket-key\":\"id\"}" } } ``` At resolve time, each branch gets a normal `TableSchema` with the **same** `id`, fields, PK, and partition keys, but different effective options: | Branch | Effective `bucket` | Source | |----------|-------------------|--------| | `main` | 64 | top-level `options` | | `snapshot` | 32 | `member-options.snapshot` | | `delta` | 128 | `member-options.delta` | A base-table `ALTER TABLE ... SET TBLPROPERTIES ('bucket' = '96')` would update all three members in one canonical commit. An explicit `ALTER TABLE t$branch_delta SET TBLPROPERTIES ('bucket' = '256')` would only update `member-options.delta`, still via a single main schema write. ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
