JTaky opened a new issue, #9226:
URL: https://github.com/apache/paimon/issues/9226

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   When Paimon syncs a table to an Iceberg REST catalog 
(metadata.iceberg.storage = rest-catalog), the only properties ever written to 
the Iceberg table are the two internal retention knobs 
(write.metadata.previous-versions-max, 
write.metadata.delete-after-commit.enabled) set in 
IcebergRestMetadataCommitter. Any other table option a user sets is silently 
dropped and never reaches the Iceberg table's properties.
   
   This matters for us because we have downstream tooling and conventions 
(table categorization tags, ownership/color-coding metadata, etc.) that rely on 
custom Iceberg table properties, and today there's no way to set them on a 
Paimon-produced table without hand-editing the table after the fact outside 
Paimon.
   
   Paimon already supports this pattern for its own native catalogs — 
HiveCatalog forwards table options into Hive/JDBC metastore table parameters by 
default (CatalogOptions.SYNC_ALL_PROPERTIES, defaulting to true). The Iceberg 
REST path just never got the equivalent treatment.
   
   ### Solution
   
   Add an explicit opt-in prefix, e.g. _metadata.iceberg.table-properties.<key> 
→ <key>_, mirroring the existing metadata.iceberg.rest.* 
(IcebergOptions.REST_CONFIG_PREFIX) pattern already used to configure the REST 
catalog client. On commit, strip the prefix and merge the resulting keys into 
the Iceberg table's properties, alongside the two existing hardcoded ones.
   
   Two things worth getting right:
   - Filter out Iceberg's reserved property keys 
(TableProperties.RESERVED_PROPERTIES) before merging, since Iceberg throws if 
they're present.
   - Apply the properties on both the create path and the drop-and-recreate 
path (the REST committer recreates the table on metadata-base mismatch), not 
just the update-in-place path — otherwise custom properties would silently 
disappear whenever that happens.
   
   An explicit opt-in prefix (rather than forwarding all Paimon table options) 
avoids leaking Paimon-internal config (bucket, merge-engine, storage 
credentials, etc.).
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to