Indian-boult opened a new pull request, #20744:
URL: https://github.com/apache/hudi/pull/20744
### Describe the issue this Pull Request addresses
Closes #20743
With column stats index version 2 (used from table version 9) and
`spark.sql.datetime.java8API.enabled=false` (the Spark default), data skipping
drops files and partitions that hold matching rows when a TIMESTAMP column has
values before 1582-10-15. The query returns too few rows, even for a plain
filter such as `ts = timestamp'0001-01-01 00:00:00'`.
The stats hold each bound as an `Instant`.
`SparkValueMetadataUtils#convertJavaTypeToSparkType` turned it into a
`java.sql.Timestamp` with `Timestamp.from(instant)`, and Spark reads that back
with `DateTimeUtils.fromJavaTimestamp`, which takes a `java.sql.Timestamp` to
be in the hybrid Julian / Gregorian calendar. Before 1582-10-15 the two
calendars differ by days, so each bound moves:
| stored bound (UTC) | bound data skipping compares, JVM zone UTC |
| --- | --- |
| 0001-01-01 00:00:00 | 0001-01-03 00:00:00 (2 days later) |
| 1500-03-05 00:00:00 | 1500-02-25 00:00:00 (9 days earlier) |
| 1582-10-04 12:00:00 | 1582-09-24 12:00:00 (10 days earlier) |
| 1582-10-15 12:00:00 and later | unchanged |
In a JVM zone with a local mean time offset the shift is not whole days, and
it reaches a little past 1582-10-15 (for ex, 1408 s at 1582-10-15 12:00 UTC in
Asia/Kolkata). Column stats and partition stats both load through this code.
The index version 1 decoder (`ColumnStatsIndexSupport#deserialize`) already
uses `DateTimeUtils.toJavaTimestamp`, and with the Java 8 API on the `Instant`
is passed through, so neither is affected. `0001-01-01 00:00:00` is a common
placeholder for an unknown time (it is `DateTime.MinValue` in .NET).
Repro on master, Spark 3.5, JVM zone UTC, COW table with column stats on `ts
timestamp`, one file per partition (`par1`: 0001-01-01 and 0001-01-05, `par2`:
1500-03-01 and 1500-03-05, `par3`: 2024-01-01 and 2024-01-05), written and read
with the parquet and Avro `datetimeRebaseMode` confs set to CORRECTED (Spark 3
refuses such values otherwise):
| filter | data skipping off | on, master | on, this PR |
| --- | --- | --- | --- |
| `ts = timestamp'0001-01-01 00:00:00'` | id 1 | no rows, 0 files | id 1, 1
file |
| `ts <= timestamp'0001-01-02 00:00:00'` | id 1 | no rows, 0 files | id 1, 1
file |
| `ts = timestamp'1500-03-05 00:00:00'` | id 4 | no rows, 0 files | id 4, 1
file |
| `ts > timestamp'1500-03-02 00:00:00' and ts < timestamp'2000-01-01
00:00:00'` | id 4 | no rows, 0 files | id 4, 1 file |
| `ts in (timestamp'0001-01-01 00:00:00', timestamp'2024-01-05 00:00:00')` |
ids 1, 6 | id 6, 1 file | ids 1, 6, 2 files |
| `ts = timestamp'2024-01-05 00:00:00'` | id 6 | id 6, 1 file | same as
master |
Partition stats on or off give the same result.
### Summary and Changelog
Data skipping on a TIMESTAMP column now reads stats before 1582-10-15 as
they were written, so queries over such values return every matching row and
still prune.
- `SparkValueMetadataUtils#convertJavaTypeToSparkType`: with the Java 8 API
off, an `Instant` becomes a `java.sql.Timestamp` through
`DateTimeUtils.toJavaTimestamp(DateTimeUtils.instantToMicros(instant))`, the
inverse of the `fromJavaTimestamp` that Spark reads it with, instead of
`Timestamp.from`. Dates are unchanged: `Date.valueOf(LocalDate)` keeps the
year, month and day, which Spark reads back as the same day.
- Tests:
- `TestSparkValueMetadataUtils`: an `Instant` before, at and after the
cutover, BC and with microseconds reads back through `fromJavaTimestamp` as the
same microseconds. On master the 6 instants up to 1582-10-15 fail (this
module's tests run in the machine's zone, here Asia/Kolkata). A `LocalDate`
before and after the cutover reads back through `fromJavaDate` as the same day,
before and after this change.
- `TestDataSkippingQuery`: the repro table above, with partition stats off
and on, checks the rows of each filter with data skipping off and on. On master
it fails at `ts = timestamp'0001-01-01 00:00:00'`.
- `TestMercifulJsonToRowConverterBase` (hudi-utilities): the timestamp
cases now also check the microseconds Spark reads from the converted `Row`. The
year-0 case expected `Timestamp.from`, and on master it reads back 171392 s
late (`expected: <-62167219200000000> but was: <-62167047808000000>`, JVM zone
Asia/Kolkata). Its expected legacy value is now Spark's own conversion.
### Impact
Queries with data skipping on that filter a TIMESTAMP column with values
before 1582-10-15 now return every matching row. Later values decode as before.
The same helper builds the `Row`s of `MercifulJsonToRowConverter`, which Hudi
Streamer passes to `createDataFrame` for JSON sources when the Java 8 API is
off, so a JSON timestamp before 1582-10-15 is now written as given instead of
days off. No public API or config change.
### Risk Level
low. For instants after the cutover (2022 in UTC, America/Los_Angeles and
Asia/Kolkata JVM zones) both conversions give the same `java.sql.Timestamp`,
and the existing tests that compare with `Timestamp.from` still pass. Before
the cutover, the bounds move back to the stored values, so files that hold
matching rows are kept and files outside the filter still prune. For JSON
sources, timestamps before the cutover are now written differently from earlier
versions (as given, instead of shifted); rows written before keep the shifted
value.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]