Indian-boult opened a new issue, #20743:
URL: https://github.com/apache/hudi/issues/20743
### Bug Description
**What happened:**
Data skipping drops files and partitions that hold matching rows when a
TIMESTAMP column has values before 1582-10-15. The query returns too few rows,
even for a plain filter on the column.
Column stats index version 2 (used from table version 9) stores each
timestamp bound as an `Instant`. When the stats are loaded for data skipping,
`SparkValueMetadataUtils#convertJavaTypeToSparkType` (hudi-spark-client) turns
the `Instant` into a `java.sql.Timestamp` with `Timestamp.from(instant)`
whenever `spark.sql.datetime.java8API.enabled` is false, which is the Spark
default. Spark then reads the `java.sql.Timestamp` with
`DateTimeUtils.fromJavaTimestamp`, which takes it to be in the hybrid Julian /
Gregorian calendar. Before 1582-10-15 the two calendars differ by days, so the
bounds that data skipping compares move away from the stored ones:
| stored bound (UTC) | bound data skipping compares, JVM zone UTC |
| --- | --- |
| 0001-01-01 00:00:00 | 0001-01-03 00:00:00 |
| 1500-03-05 00:00:00 | 1500-02-25 00:00:00 |
| 1582-10-04 12:00:00 | 1582-09-24 12:00:00 |
| 1582-10-15 12:00:00 and later | unchanged |
In a JVM zone with a local mean time offset the shift is not whole days and
reaches a little past 1582-10-15 (for ex, 1408 s at 1582-10-15 12:00 UTC in
Asia/Kolkata, 422 s in America/Los_Angeles). Column stats and partition stats
both load through this code. The index version 1 decoder
(`ColumnStatsIndexSupport#deserialize`) already uses
`DateTimeUtils.toJavaTimestamp`, and with the Java 8 API on the `Instant` is
passed through, so neither is affected. DATE stats are not affected, except the
10 days 1582-10-05 to 1582-10-14, which a `java.sql.Date` cannot hold. The same
helper builds the rows of Hudi Streamer JSON sources
(`MercifulJsonToRowConverter`), so a JSON timestamp before 1582-10-15 is also
written days off (171392 s for 0000-01-01T00:00Z in an Asia/Kolkata JVM).
**What you expected:**
Every file and partition that holds a matching row stays a candidate,
whatever the year of the timestamp.
**Steps to reproduce:**
```sql
set spark.sql.session.timeZone = UTC;
-- Spark 3 refuses to write or read datetimes before 1582-10-15 without these
set spark.sql.parquet.datetimeRebaseModeInWrite = CORRECTED;
set spark.sql.parquet.datetimeRebaseModeInRead = CORRECTED;
set spark.sql.avro.datetimeRebaseModeInWrite = CORRECTED;
set spark.sql.avro.datetimeRebaseModeInRead = CORRECTED;
set hoodie.metadata.index.column.stats.enable = true;
set hoodie.metadata.index.column.stats.column.list = ts;
create table t (id int, ts timestamp, ord long, dt string) using hudi
tblproperties (primaryKey = 'id', orderingFields = 'ord', type = 'cow')
partitioned by (dt);
insert into t values (1, timestamp'0001-01-01 00:00:00', 1, 'par1'), (2,
timestamp'0001-01-05 00:00:00', 1, 'par1');
insert into t values (3, timestamp'1500-03-01 00:00:00', 1, 'par2'), (4,
timestamp'1500-03-05 00:00:00', 1, 'par2');
insert into t values (5, timestamp'2024-01-01 00:00:00', 1, 'par3'), (6,
timestamp'2024-01-05 00:00:00', 1, 'par3');
set hoodie.enable.data.skipping = true;
select id from t where ts = timestamp'0001-01-01 00:00:00'; -- expected 1,
returns nothing
select id from t where ts = timestamp'1500-03-05 00:00:00'; -- expected 4,
returns nothing
select id from t where ts in (timestamp'0001-01-01 00:00:00',
timestamp'2024-01-05 00:00:00'); -- expected 1 and 6, returns 6
```
With `hoodie.enable.data.skipping = false`, or with
`spark.sql.datetime.java8API.enabled = true`, each query returns its rows. The
result is the same with partition stats on or off.
This is one of the extreme-value cases in the gaps listed in #20449.
### Environment
- Hudi version: master (1.3.0-SNAPSHOT), table version 9, column stats index
version 2
- Query engine: Spark 3.5
- Relevant configs: `hoodie.enable.data.skipping=true`, column stats on the
TIMESTAMP column, `spark.sql.datetime.java8API.enabled=false` (default)
### Logs and Stack Trace
No error. The files are pruned and the rows are missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]