junmuz commented on PR #9822:
URL: https://github.com/apache/paimon/pull/9822#issuecomment-6297340698
Thanks for the suggestions. I have tried addressing the feedback in
5a998464e5 (also rebased on master)
1. **[P2] Nested fallback shape:** Both the raw and merge readers now read
the source field with the union of the nested children that it and its metadata
field need, then trim each fallback value to the requested metadata shape. This
covers the case where the physical column and the metadata request different
children. `ChangelogEventMetadataNestedReadTest` covers Parquet/Avro/ORC on the
raw path and on the merge path via `forceKeepDelete()`.
2. **[P2] Temporal lookup join:** I went with rejecting the plan. A lookup
only sees the latest table state, so it can't return per-event metadata.
`DataTableSource.getLookupRuntimeProvider` now fails with a clear error when
metadata columns are requested, and lookups of physical columns still work.
Both cases are covered in `LookupChangelogEventMetadataITCase`.
3. **[P2] Duplicate Avro dependency:** Removed.
While testing, I also fixed `incremental-between-scan-mode=diff` reads,
which failed with `Cannot find field index by FieldId` when metadata fields
were requested.
I have also raised [a different
PR](https://github.com/apache/paimon/pull/10702) that preserves the field with
additional configuration option. That's much more simpler, but it comes at the
expense of breaking CDC contract (which should be ok IMO as long as we clearly
document it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
