VISHAL B created FLINK-40985:
--------------------------------
Summary: [Iceberg] Support per-table location override in Iceberg
pipeline sink
Key: FLINK-40985
URL: https://issues.apache.org/jira/browse/FLINK-40985
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Affects Versions: cdc-3.6.0
Reporter: VISHAL B
Fix For: cdc-3.8.0
{{When the Iceberg pipeline sink auto-creates a table (via
include.schema.changes:
create.table), the table's storage location is always computed by the configured
Iceberg catalog's default location logic — there's no way to set an explicit
per-table location through sink configuration.
This is a real limitation with catalogs like AWS Glue, which — for
Hive-Metastore
compatibility — computes a default database location as
<warehouse-root>/<database>.db/<table>/ whenever the catalog's
database/namespace
has no explicit LocationUri set. For a shared Glue database (used by multiple
systems, not just this pipeline), changing the database's own LocationUri to fix
this isn't viable, since it would retroactively change the default location for
every other table anyone creates there. Sink-side, per-table control is needed
instead.
Checked IcebergMetadataApplier.applyCreateTable() in
flink-cdc-pipeline-connector-iceberg (3.6.0): it calls the catalog's standard
createTable(identifier, schema, partitionSpec, properties), which always defers
to the catalog's default-location computation. There's no withLocation() call
anywhere in the create-table path, so there's currently no supported way to plug
an explicit location in, even via table.properties.*.
Proposal: support an explicit per-table location override in the sink config
(e.g. a table.properties.location-style key, or a location-template option
applied per source table), used via Catalog.buildTable(id, schema)
.withLocation(resolvedLocation).create() instead of the current
catalog.createTable(...) call, when a location is configured.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
