wombatu-kun opened a new pull request, #19336:
URL: https://github.com/apache/hudi/pull/19336
### Describe the issue this Pull Request addresses
Two edge-case defects in the partition path of `hudi-hive-sync`:
1. `PartitionFilterGenerator.ValueComparator` sorts `int`/`bigint` partition
values with subtraction (`i1 - i2`, `Long.signum(l1 - l2)`) to derive the
min/max bounds of the pushdown filter (used when the number of written
partitions exceeds `hoodie.datasource.hive_sync.filter_pushdown_max_size`).
Subtraction overflows when the difference exceeds the type range (for example
`Integer.MAX_VALUE` and a negative value), producing a wrong ordering and
therefore wrong `>= min AND <= max` bounds that can exclude valid partitions
(or trip the `Comparator` contract check).
2. `MultiPartKeysValueExtractor` and `HiveStylePartitionValueExtractor`
split a hive-style `key=value` segment with `split("=")` (no limit). A value
that itself contains `=` (for example base64 padding `col=YWJjZA==`, or an
embedded `=`) is either rejected because `length != 2` and aborts the whole
sync, or silently truncated because trailing `=` is dropped, causing the stored
value to diverge from the metastore and produce spurious ADD/DROP churn on
every sync.
### Summary and Changelog
Makes partition value parsing and pushdown comparison robust to edge values.
- `PartitionFilterGenerator.ValueComparator` now uses
`Integer.compare`/`Long.compare` instead of overflow-prone subtraction.
- `MultiPartKeysValueExtractor` and `HiveStylePartitionValueExtractor` now
split on the first `=` only via `split("=", 2)`, preserving values that contain
`=` (including base64 padding and empty values).
- Extends `TestPartitionFilterGenerator` (min/max bounds for extreme
int/bigint values), `TestMultiPartKeysValueExtractor` and
`TestPartitionValueExtractor` (values containing `=`).
### Impact
Tables whose numeric partition columns span extreme values now get correct
pushdown min/max bounds, and hive-style partitions whose values contain `=` are
synced correctly instead of failing or drifting. No public API or configuration
change.
### Risk Level
low
Small, localized changes with unit tests covering the previously broken
inputs.
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
