vbabenkoru commented on PR #9245:
URL: https://github.com/apache/paimon/pull/9245#issuecomment-5324755892
@JingsongLi Thank you for the review! Fixed 4 issues (with some AI assist),
and two left for discussion:
* Compaction replaces row ids: this is the most complicated thing to fix
actually. This requires _real_, fully Iceberg-compatible row lineage
implemented in the data layer. More details on this are in the PR #9244, but
the main point is that 1) Paimon lineage fields are incompatible with Iceberg
(different reserved column IDs); 2) row lineage/row tracking isn't implemented
for PK tables at all. I think we have three options:
* Adopt this set of changes as-is as default: we create a synthetic,
metadata-only implementation of row lineage that is intentionally incomplete
without the data-layer changes, but is somewhat better than writing V3 tables
without row lineage whatsoever.
* Do the same, but make it opt-in behind a table setting, so that this
synthetic implementation is not enabled by default.
* Reject synthetic metadata-only approach: revert #9244 and fully
implement row tracking/lineage for PK tables + make make backwards-incompatible
changes to the Paimon data format for row lineage compatibility with Iceberg,
likely through a table setting that can only be set on creation. Then add the
Iceberg metadata implementation on top.
* Fencing older writers running #9244: The prior PR was merged yesterday and
isn't part of an actual release yet, and it looks like implementing fencing
will be fairly involved; do you think it's still necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
