yihua opened a new issue, #20169:
URL: https://github.com/apache/hudi/issues/20169
Users who already write Iceberg tables from Spark, Flink or Kafka Connect
cannot adopt the pluggable table format without moving their pipelines to
Hudi's entry points (`format("hudi")`, `HoodieCatalog`, `USING hudi`), which is
a rewrite rather than a switch. That keeps Hudi's write path, indexing and
table services out of reach for the largest population of Iceberg tables.
Proposal: an Iceberg-side plugin that occupies Iceberg's own pluggable
seams, so a user adds a jar and
`spark.plugins=org.apache.spark.HudiIcebergPlugin` and keeps everything else:
the catalog wrapper returns a table whose writes, row-level commands,
maintenance procedures and streaming commits go through Hudi, and the pluggable
table format publishes each Hudi instant as an Iceberg snapshot in the same
table. Hudi-side prerequisites include persisting table format settings from
the first commit, tolerating foreign data files in the file-system view,
threading the catalog configuration into the format plugin, and metadata table
coexistence. The design is in RFC-115; a proof of concept exists in Apache
XTable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
