slachiewicz opened a new pull request, #724: URL: https://github.com/apache/hudi-rs/pull/724
## Description Hive-style partition pruning has never worked for `TimestampBasedKeyGenerator`, and two separate defects made it fail in opposite directions. Every timestamp-keygen fixture in `crates/test/data` is `hive_style_partitioning=false`, so the hive branch shipped with no table-backed coverage and both survived #286. **The path shape was invented.** `format_partition_path` split `output.dateformat` on `/` and derived a column name per token (`yyyy` to `year`, `MM` to `month`), so field `ts` with format `yyyy/MM/dd` produced `year=2024/month=03/day=01`. With no `/` in the format it used the format string itself as the name, giving `yyyy-MM-dd=2024-03-01`. Hudi does neither. The final line of `TimestampBasedAvroKeyGenerator.getPartitionPath` is: ```java return hiveStylePartitioning ? getPartitionPathFields().get(0) + "=" + partitionPath : partitionPath; ``` The whole formatted value gets one prefix, taken from `hoodie.table.partition.fields`. Any `/` inside the format stays in the value and becomes nested directories. So the real paths are `ts=2024/03/01` and `ts=2024-03-01`, and the reader's predicate matched neither. Where the writer configs were present this excluded every partition; where they were absent the filter failed to bind and the table was scanned in full. **Negation dropped whole partitions.** The transform is lossy: a partition holds every instant of its period, so `ts != 2024-03-01T14:30:00Z` is satisfied by almost every row of `ts=2024-03-01`. Mapping the negation onto the path discarded that partition and lost all of them, which is a wrong-results bug rather than a slow one. `!=` and `NOT IN` now emit no partition predicate and are left to row-level evaluation. Four fixtures come with this: three written by Hudi 0.15.0 (table version 6, the last release that writes it) and one by Hudi 1.2.0 on Spark 4.1 (version 9), all COW. They cover day granularity, an hour format whose slashes nest under a single hive prefix, and a custom key generator pairing an identity field with a time-derived one. The two custom-keygen tables are deliberately a pair, and they document something a reader has to live with: version 6 records `hoodie.table.partition.fields=region,ts_str` and persists no per-field key types at all, while version 9 records `region:SIMPLE,ts_str:TIMESTAMP`. One expectation shifted while writing the integration tests, and it is worth stating rather than leaving for a reviewer to rediscover. Partition pruning widens a half-open upper bound to the boundary partition, since the transform has already discarded the time of day. File-level column statistics then drop that file when its real range misses the predicate. The two stages compose and the observable result is the tighter of the two. `CustomKeyGenerator` support for #549 is not in this PR. The version 6 fixture showed that the obvious fallback for an untyped field is unsafe, and that needs its own change. Verified: `make format check test` passes, and `cargo clippy --all-targets --all-features --workspace -- -D warnings` is clean. ## How are the changes test-covered - [ ] N/A - [x] Automated tests (unit and/or integration tests) - [ ] Manual tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
