rdblue opened a new pull request, #18248:
URL: https://github.com/apache/iceberg/pull/18248
This implements MDV filtering in V4ManifestReader.
The MDV filter is run after inheritance and first row ID assignment because
first row ID assignment must be done for all files. The filter is also run
before more expensive filtering, like content stats and partition filtering. If
no DV is present, only status filtering is enabled to avoid checking whether
the MDV is null on every record.
This also updates the `includeAll` mode, which returns records that have
`DELETED` or `REPLACED` status in the file. When `includeAll` encounters a file
that is live in the manifest and deleted by its MDV, it sets the status to
`DELETED` and the snapshot ID to null because the actual snapshot ID when the
file was deleted is not known. This is strange because there are no other
situations in which a manifest entry contains a null snapshot ID.
I think the right fix for the null snapshot ID issue is not to have an
`includeAll` mode, but since that may be controversial I think we should fix
this when we are implementing reads to detect changes. When reading a manifest
to find changes, the reader needs the manifest's replaced and deleted positions
from `Tracking` that have a known snapshot ID (`dv_snapshot_id`, if it matches
the snapshot for which changes are being scanned).
Test plan:
- TestTrackingStruct:
- Test `convertToDeleted` and `convertToReplaced` methods modify only
status and snapshot ID
- TestV4ManfiestReader:
- Validate that MDV filtering removes live files and is a no-op for
non-live files
- Validate that `includeAll` returns MDV-deleted files as `DELETED` with
a null `snapshot_id` (see above)
- Validate that `includeAll` does not modify already deleted records
- Update inheritance, first row ID, and MDV filtering tests to use full
validation of `Tracking`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
