slachiewicz opened a new pull request, #746:
URL: https://github.com/apache/hudi-rs/pull/746
## Description
> **Stacked on #725, which is stacked on #724.** The diff shown here
includes both PRs'
> commits and will shrink to the four below once they merge. Review the last
four commits
> only: `test(core): add a timestamp-column fixture for the timestamp key
generator`,
> `fix(core): parse timestamp keygen filter literals by their own spelling`,
> `fix(core): keep reading when a partition filter cannot be rewritten` and
> `fix(datafusion): forward timestamp keygen source-column predicates to
listing`.
`TimestampBasedKeyGenerator` parsed an incoming filter literal with
`hoodie.keygen.timebased.timestamp.type`, which describes how the *source
column* stores
its value. A literal is spelled by whoever wrote the query, and the two need
not agree. A
table whose partition column is a real `TIMESTAMP` can only declare an epoch
type honestly,
since that is what Spark hands the key generator, and then no datetime a
caller writes
survives `parse::<i64>()`. The failure was not a degraded scan: the error
came out of
`PartitionPruner::new` and the read failed outright.
The reachable paths turned out to be narrower than the description above
suggests, and the
new fixture pins which is which. A DataFusion timestamp literal never hit
the parser at
all, because `ScalarValue`'s `Display` prints the raw epoch integer in the
column's unit,
which is exactly the declared encoding;
`test_convert_timestamp_literal_is_epoch_integer_in_its_unit`
pins that spelling since the key generator now depends on it. The string
filter API is
the path that failed: `ReadOptions::with_filters(("ts", ">=",
"2024-03-01T00:00:00Z"))`,
and with it the Python and C++ bindings, returned
`Failed to parse epoch microseconds '2024-03-01T00:00:00Z'` on the fixture.
**Literals are parsed by their own spelling.** The configured encoding is
still tried
first, so every input that works today behaves exactly as before, and the
fallback takes
RFC 3339 with an offset, a naive datetime with `T` or space, and a bare
date, all as UTC.
The two families are disjoint: no epoch integer parses as a datetime and no
datetime
parses as an integer, so the fallback cannot reinterpret an accepted input,
and
`test_parse_timestamp_literal_families_are_disjoint` holds that.
`input.dateformat` keeps
describing the source column and is not consulted for the fallback. One side
effect worth
naming: a `DATE_STRING` table with a time-bearing input format now accepts a
bare date
literal, which previously errored because the format demanded a time.
**An unparseable literal no longer fails the read.** Pruning is an
optimization, so a
filter the generator cannot rewrite is kept untransformed, the way #724
already handles
`!=` and `NOT IN`. It still names the source column, which the partition
schema does not
contain, so it fails to bind and lands in #725's `unapplied_filters` while
the read goes
ahead and enforces it per row.
`unparseable_literal_is_reported_as_unapplied_instead_of_failing_the_read`
checks that
composition.
**DataFusion never forwarded the predicate.** Once the fixture existed, the
DataFusion
query returned the right rows but scanned all four partitions. The provider
classified
partition predicates by the partition schema, which for a timestamp key
generator holds
only `_hoodie_partition_path`, so a predicate on `ts` was never passed to
file listing.
`HudiDataSource` now also lists the key generator's source column for
listing purposes.
Exactness is unchanged: a predicate on a derived column is never `Exact`,
because a day
partition holds every instant of that day, and the test asserts the equality
query still
returns one row rather than the whole day.
The fixture is a table version 9 COW written by Hudi 1.2.0 on Spark 3.5.7
through the
DataFrame writer. Two things ruled out the Spark SQL recipe the other
fixtures use: Hudi
0.15.0 has no `EPOCHMICROSECONDS` and formats the micros as millis (year
56134), and Hudi
1.2.0's SQL layer re-parses a timestamp partition path as a long in
`SqlKeyGenerator.convertPartitionPathToSqlType` and throws on `2024-03-01`.
Both are
recorded in the fixture's recipe.
Not in this PR: `CustomKeyGenerator` (#549), and the typed-value refactor in
#609. That
refactor is the proper end state for this class of bug, since a typed
`Filter` would carry
the DataFusion literal without a string round trip at all. This change is
the narrow
increment ahead of it because the string API is what the bindings expose
today and will
keep exposing after #609, whose plan keeps the string tuple form as the
cross-language
entry point, so a parser that reads a literal by its own spelling is needed
either way.
Verified: `cargo clippy --all-targets --all-features --workspace -- -D
warnings` is clean,
the no-spill `hudi-core` leg passes, and the keygen, partition,
`table_read_tests` and
`hudi-datafusion` suites pass locally. `make format check test` did not
complete locally
(the machine ran out of disk during the Python build), so CI is the check
for that.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
*This change was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
