slachiewicz opened a new issue, #798: URL: https://github.com/apache/hudi-rs/issues/798
### Is there an existing issue for this? - [x] I have searched the existing issues — #286 (closed, added the transform via #462), #580 (multi-field ordering), #549 (CustomKeyGenerator) and #779 (storage listing) are adjacent; none covers the hive-style path shape. ### Description of the bug On a hive-style table with `hoodie.table.partition.fields=ts` and `hoodie.keygen.timebased.output.dateformat=yyyy-MM-dd`, an equality predicate on `ts` returns zero file slices and zero rows although a partition holds a matching row. No warning is logged. Without the `hoodie.keygen.timebased.*` keys the same predicate returns the right row, because the pruner logs `Failed to create TimestampBasedKeyGenerator ... Filters will not be transformed` and column-stats pruning takes over. Supplying the generator's parameters turns a correct read into a wrong one. Cause, at `release-0.5.0`: [`parse_partition_fields`](https://github.com/apache/hudi-rs/blob/release-0.5.0/crates/core/src/keygen/timestamp_based.rs#L281-L302) derives one hive field name per `/`-separated token of `output.dateformat` (`yyyy` → `year`, …), and an unrecognised token becomes its own name, so a format with no `/` yields the field name `yyyy-MM-dd`. [`format_partition_path`](https://github.com/apache/hudi-rs/blob/release-0.5.0/crates/core/src/keygen/timestamp_based.rs#L425-L444) then rewrites the predicate to `_hoodie_partition_path = "yyyy-MM-dd=2024-03-02"`. The directory is `ts=2024-03-02`. Hudi's `TimestampBasedAvroKeyGenerator.getPartitionPath` prefixes the whole formatted value with `partition.fields[0] + "="` exactly once, so no hive-style timestamp-keygen table can match the rewritten path. The shape does not depend on `timestamp.type`. #724 corrects the path shape and carries Hudi 0.15 and 1.2 fixtures for it; this issue is the symptom-level entry for that PR. ### Steps To Reproduce Table written by polytable (an independent Go Hudi writer), COW, table version 6, three day partitions: ``` ts=2024-03-01/<one base file, 1 row> ts=2024-03-02/<one base file, 2 rows, one with ts = 2024-03-02 12:00:00> ts=2024-03-03/<one base file, 3 rows> ``` `hoodie.properties` — polytable writes the first three lines; the four `hoodie.keygen.timebased.*` lines were added by hand so the generator could be built (they are the keys Hudi 1.x persists on its own; the shipped `v9_timebasedkeygen_*` fixtures carry `timestamp.type` and `output.dateformat`): ``` hoodie.table.partition.fields=ts hoodie.datasource.write.hive_style_partitioning=true hoodie.table.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator hoodie.keygen.timebased.timestamp.type=DATE_STRING hoodie.keygen.timebased.input.dateformat=yyyy-MM-dd HH:mm:ss hoodie.keygen.timebased.output.dateformat=yyyy-MM-dd hoodie.keygen.timebased.timezone=UTC ``` ```rust let opts = ReadOptions::new().with_filters([("ts", "=", "2024-03-02 12:00:00")])?; table.get_file_slices(&opts).await? // 0 slices table.read(&opts).await? // 0 rows ``` Remove the four `hoodie.keygen.timebased.*` lines and the same call returns 1 slice and 1 row. ### Expected behavior 1 file slice (`ts=2024-03-02`) and 1 row, the same as the read without the keygen keys. ### Screenshots / Logs With the keys present, `RUST_LOG=warn` prints nothing. Without them: ``` WARN hudi_core::table::partition] Failed to create TimestampBasedKeyGenerator: Config error: Config 'Missing required configuration: hoodie.keygen.timebased.timestamp.type' not found. Filters will not be transformed. ``` ### Software information - Operating system: macOS 27.0, arm64 - Rust version: rustc 1.98.1 - Project version: hudi 0.5.0 (crates.io, `default-features = false`) ### Additional context A date-only literal on the same table, `("ts", "=", "2024-03-02")`, fails separately with `Failed to parse date string '2024-03-02' with format '%Y-%m-%d %H:%M:%S': premature end of input`, because the literal is parsed with the column's `input.dateformat`. That is #746's case, not this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
