Indian-boult opened a new pull request, #20744:
URL: https://github.com/apache/hudi/pull/20744

   
   ### Describe the issue this Pull Request addresses
   
   Closes #20743
   
   With column stats index version 2 (used from table version 9) and 
`spark.sql.datetime.java8API.enabled=false` (the Spark default), data skipping 
drops files and partitions that hold matching rows when a TIMESTAMP column has 
values before 1582-10-15. The query returns too few rows, even for a plain 
filter such as `ts = timestamp'0001-01-01 00:00:00'`.
   
   The stats hold each bound as an `Instant`. 
`SparkValueMetadataUtils#convertJavaTypeToSparkType` turned it into a 
`java.sql.Timestamp` with `Timestamp.from(instant)`, and Spark reads that back 
with `DateTimeUtils.fromJavaTimestamp`, which takes a `java.sql.Timestamp` to 
be in the hybrid Julian / Gregorian calendar. Before 1582-10-15 the two 
calendars differ by days, so each bound moves:
   
   | stored bound (UTC) | bound data skipping compares, JVM zone UTC |
   | --- | --- |
   | 0001-01-01 00:00:00 | 0001-01-03 00:00:00 (2 days later) |
   | 1500-03-05 00:00:00 | 1500-02-25 00:00:00 (9 days earlier) |
   | 1582-10-04 12:00:00 | 1582-09-24 12:00:00 (10 days earlier) |
   | 1582-10-15 12:00:00 and later | unchanged |
   
   In a JVM zone with a local mean time offset the shift is not whole days, and 
it reaches a little past 1582-10-15 (for ex, 1408 s at 1582-10-15 12:00 UTC in 
Asia/Kolkata). Column stats and partition stats both load through this code. 
The index version 1 decoder (`ColumnStatsIndexSupport#deserialize`) already 
uses `DateTimeUtils.toJavaTimestamp`, and with the Java 8 API on the `Instant` 
is passed through, so neither is affected. `0001-01-01 00:00:00` is a common 
placeholder for an unknown time (it is `DateTime.MinValue` in .NET).
   
   Repro on master, Spark 3.5, JVM zone UTC, COW table with column stats on `ts 
timestamp`, one file per partition (`par1`: 0001-01-01 and 0001-01-05, `par2`: 
1500-03-01 and 1500-03-05, `par3`: 2024-01-01 and 2024-01-05), written and read 
with the parquet and Avro `datetimeRebaseMode` confs set to CORRECTED (Spark 3 
refuses such values otherwise):
   
   | filter | data skipping off | on, master | on, this PR |
   | --- | --- | --- | --- |
   | `ts = timestamp'0001-01-01 00:00:00'` | id 1 | no rows, 0 files | id 1, 1 
file |
   | `ts <= timestamp'0001-01-02 00:00:00'` | id 1 | no rows, 0 files | id 1, 1 
file |
   | `ts = timestamp'1500-03-05 00:00:00'` | id 4 | no rows, 0 files | id 4, 1 
file |
   | `ts > timestamp'1500-03-02 00:00:00' and ts < timestamp'2000-01-01 
00:00:00'` | id 4 | no rows, 0 files | id 4, 1 file |
   | `ts in (timestamp'0001-01-01 00:00:00', timestamp'2024-01-05 00:00:00')` | 
ids 1, 6 | id 6, 1 file | ids 1, 6, 2 files |
   | `ts = timestamp'2024-01-05 00:00:00'` | id 6 | id 6, 1 file | same as 
master |
   
   Partition stats on or off give the same result.
   
   ### Summary and Changelog
   
   Data skipping on a TIMESTAMP column now reads stats before 1582-10-15 as 
they were written, so queries over such values return every matching row and 
still prune.
   
   - `SparkValueMetadataUtils#convertJavaTypeToSparkType`: with the Java 8 API 
off, an `Instant` becomes a `java.sql.Timestamp` through 
`DateTimeUtils.toJavaTimestamp(DateTimeUtils.instantToMicros(instant))`, the 
inverse of the `fromJavaTimestamp` that Spark reads it with, instead of 
`Timestamp.from`. Dates are unchanged: `Date.valueOf(LocalDate)` keeps the 
year, month and day, which Spark reads back as the same day.
   - Tests:
     - `TestSparkValueMetadataUtils`: an `Instant` before, at and after the 
cutover, BC and with microseconds reads back through `fromJavaTimestamp` as the 
same microseconds. On master the 6 instants up to 1582-10-15 fail (this 
module's tests run in the machine's zone, here Asia/Kolkata). A `LocalDate` 
before and after the cutover reads back through `fromJavaDate` as the same day, 
before and after this change.
     - `TestDataSkippingQuery`: the repro table above, with partition stats off 
and on, checks the rows of each filter with data skipping off and on. On master 
it fails at `ts = timestamp'0001-01-01 00:00:00'`.
     - `TestMercifulJsonToRowConverterBase` (hudi-utilities): the timestamp 
cases now also check the microseconds Spark reads from the converted `Row`. The 
year-0 case expected `Timestamp.from`, and on master it reads back 171392 s 
late (`expected: <-62167219200000000> but was: <-62167047808000000>`, JVM zone 
Asia/Kolkata). Its expected legacy value is now Spark's own conversion.
   
   ### Impact
   
   Queries with data skipping on that filter a TIMESTAMP column with values 
before 1582-10-15 now return every matching row. Later values decode as before. 
The same helper builds the `Row`s of `MercifulJsonToRowConverter`, which Hudi 
Streamer passes to `createDataFrame` for JSON sources when the Java 8 API is 
off, so a JSON timestamp before 1582-10-15 is now written as given instead of 
days off. No public API or config change.
   
   ### Risk Level
   
   low. For instants after the cutover (2022 in UTC, America/Los_Angeles and 
Asia/Kolkata JVM zones) both conversions give the same `java.sql.Timestamp`, 
and the existing tests that compare with `Timestamp.from` still pass. Before 
the cutover, the bounds move back to the stored values, so files that hold 
matching rows are kept and files outside the filter still prune. For JSON 
sources, timestamps before the cutover are now written differently from earlier 
versions (as given, instead of shifted); rows written before keep the shifted 
value.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): read timestamp... Indian-boult (via GitHub) via commits

Reply via email to