kbendick commented on PR #5009:
URL: https://github.com/apache/iceberg/pull/5009#issuecomment-1155440960
This changes some of the existing behavior:
- `changed-partition-count` does not include "unpartitioned" - so the number
of changed partitions could potentially go down by 1 in some scenarios (or not
be present at all).
- We can decouple the function that writes the `partition.` stats and the
count of changed partitions to keep the old behavior.
- Stops writing partition summaries for unpartitioned tables (the goal of
the PR). So there won't be a `partitions.` summary in the file for the
unpartitioned section of the table.
- Void partition transforms (dropped partitions) in v1 tables will still
be included in partition summary stats as they transform to `alwaysNull` and
where the change was made, it's not possible to differentiate between an actual
partition field with a null value vs the void transform.
- This does not affect v2 tables which don't use the void transform
The big one is the `changed-partition-count`. We can decouple the logic so
that `changed-partition-count` is still 1 for the unpartitioned parts of the
table when they are touched (technically at present, if a V2 table has multiple
unpartitioned specs, there will only be 1 value in total added to
`changed-partition-count`, instead of the value being incremented for each
unpartitioned section of the table).
If anybody has a real-world use case where the original behavior is needed,
I'm very open to hearing it. My goal is just to remove `partition.` partition
summary fields caused by the PartitionSpec for `PartitionSpec.unpartitioned()`
being defined as an empty list of fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
