MisterRaindrop opened a new pull request, #1842:
URL: https://github.com/apache/cloudberry/pull/1842
## Summary
Kernel-side DDL scaffolding for Iceberg lake tables, per the design proposal
in #1683 (this PR is the "core syntax" milestone of the roadmap posted there).
It adds the system catalogs, parse nodes, grammar, commands, and client-tool
awareness that a datalake provider extension builds on. **No table AM
implementation or provider is included** — those are later milestones (agent /
FDW phases in the roadmap).
```sql
CREATE FOREIGN CATALOG hive_cat SERVER hive_cat_srv OPTIONS (type 'hive',
uri 'thrift://...');
CREATE FOREIGN VOLUME s3_vol SERVER s3_srv OPTIONS (path
's3://bucket/prefix');
CREATE ICEBERG TABLE t (a int, b text) CATALOG hive_cat VOLUME s3_vol
OPTIONS (fileformat 'parquet');
DROP CATALOG hive_cat; DROP VOLUME s3_vol;
```
## What's included (structured for commit-by-commit review)
1. **catalog**: `pg_foreign_catalog`, `pg_foreign_volume` (metadata service
/ storage location handles, hanging off `pg_foreign_server`), `pg_lake_table`
(per-relation lake metadata keyed by relid). All three have TOAST tables; name
indexes are single-column unique (see "Design decisions").
2. **nodes**: `CreateLakeTableStmt` (embeds `CreateStmt`),
`CreateForeignCatalogStmt`, `CreateForeignVolumeStmt` + hand-maintained
node-support functions.
3. **parser**: `CREATE ICEBERG TABLE ... CATALOG ... VOLUME ... OPTIONS
(...)`, `CREATE FOREIGN CATALOG|VOLUME ... SERVER ...`, `DROP CATALOG|VOLUME`
via `drop_type_name`.
4. **commands (foreign catalog/volume)**: create/drop with IF [NOT] EXISTS,
dependencies, ownership, object addressing / COMMENT / event-trigger
integration, syscaches, QD→QE dispatch with synchronized OIDs.
5. **commands (CREATE ICEBERG TABLE)**: resolves catalog/volume (explicit
clause or `iceberg_default_catalog`/`iceberg_default_volume` GUCs), validates
the statement uses the `iceberg` AM, creates the relation + TOAST +
`pg_lake_table` row + dependencies; guards `ALTER TABLE SET ACCESS METHOD` /
`SET DISTRIBUTED BY` both directions; lake tables are forced `DISTRIBUTED
RANDOMLY`.
6. **bin**: `pg_dump` skips iceberg tables (by AM name, with a warning —
same pattern as PAX); psql tab completion.
7. **tests**: `lake_table` regression in `greenplum_schedule` (DDL
lifecycle, duplicate/USING rejection, TOAST wide rows, ownership,
dependency/CASCADE, QD/QE dispatch checks) + catalog-expected refresh across
`regress`, `singlenode_regress`, and pax copies.
## Design decisions to review
- **Provider decoupling**: the kernel resolves the AM strictly by name
(`get_table_am_oid("iceberg")`); no hardcoded AM OID and no extension-name
checks. Everything degrades gracefully (with a hint) when no provider is
installed.
- **Provider integration contract**: `CreateLakeTable()` runs after
`DefineRelation()` and ends with `CommandCounterIncrement()` +
`InvokeObjectPostCreateHook(LakeTableRelationId, ...)` — a provider performs
remote Iceberg creation from that object-access hook (on QD and QEs), where
catalog/volume/options metadata is already visible.
`relation_set_new_filelocator` remains local-storage-only.
- **Catalog/volume names are database-global** (single-column unique index),
like `pg_foreign_server` — every reference syntax (`DROP CATALOG x`, the
`CATALOG x` clause, the GUCs) identifies them by bare name, so the uniqueness
scope matches what the syntax can express. (The alternative — per-server names
like user mappings — would require server-qualified reference syntax
everywhere.)
- `CREATE ICEBERG TABLE` **rejects a `USING` clause naming any other AM**;
without it the statement implies `USING iceberg`.
## Known limitations / open questions (deliberately out of scope here)
- **pg_dump / pg_upgrade**: FOREIGN CATALOG/VOLUME objects and pg_lake_table
metadata are not dumped yet; iceberg tables are skipped with a warning (PAX
precedent). Restore/upgrade support is planned as a follow-up.
- **Permission model**: creating a catalog/volume requires `USAGE` on its
server. Referencing one from `CREATE ICEBERG TABLE` currently checks existence
only — whether that should require `USAGE` on the underlying server, or
dedicated ACLs (`GRANT USAGE ON CATALOG`), is an open question we'd like
reviewer input on.
- The GUC defaults are synced to QEs and behave like `default_tablespace`
w.r.t. objects dropped after `SET`.
## Testing
- Full CI matrix green on the fork (25 installcheck jobs incl.
`ic-good-opt-off/on`, pax, singlenode, isolation2):
https://github.com/MisterRaindrop/cloudberry/actions/runs/28768436319
- Validated live on a 3-segment demo cluster with a heap-backed stand-in AM
(`CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler`): DDL
lifecycle, duplicate-name rejection, USING rejection, wide-row TOAST inserts,
QD/QE toast-OID consistency (`gp_dist_random('pg_class')`), drop cleanup,
CASCADE.
Discussion: #1683
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
