morningman opened a new pull request, #67449:
URL: https://github.com/apache/doris/pull/67449
## Problem
`DATE` values in year zero are shipped one day late by every format that
encodes a day ordinal. `CAST('0000-01-01' AS DATE)` arrives as `0000-01-02`
over Arrow Flight SQL while the MySQL protocol reports `0000-01-01` for the
same value.
Doris follows MySQL's calendar, where year 0 is **not** a leap year
(`is_leap()` in `be/src/util/time_lut.h`): `0000-02-29` does not exist, so
`calc_daynr()` numbers `0000-01-01 .. 0000-02-28` one day ahead of the
proleptic Gregorian calendar. Arrow `date32`, Parquet `DATE`, ORC `DATE` and
Iceberg all define their day ordinal in the proleptic Gregorian calendar, where
year 0 **is** a leap year. The two numberings agree from `0000-03-01` onwards,
so exactly 59 dates are affected.
Every boundary added or subtracted the epoch day number directly instead of
converting:
| | Doris `daynr` | shipped `date32` | proleptic |
|---|---:|---:|---:|
| `0000-01-01` | 1 | -719527 | **-719528** |
| `0000-02-28` | 59 | -719469 | **-719470** |
| `0000-03-01` | 60 | -719468 | -719468 |
| `2024-01-01` | 739251 | 19723 | 19723 |
The MySQL protocol is unaffected because it carries `year`/`month`/`day`
verbatim (`mysql_row_buffer.cpp`) and never needs a calendar at all.
Doris was already inconsistent with itself: `DATETIMEV2` over Arrow goes
through `unix_timestamp()` (cctz, proleptic) and was correct, while `DATEV2`
was not; ORC predicate pushdown built its literals with `cctz::civil_day`
(proleptic) while the ORC writer emitted Doris day numbers, so a pushed-down
`WHERE d = '0000-01-01'` could prune the very row it was looking for; and
`bucket(timestamp)` used `unix_timestamp()` while `bucket(date)` used `daynr()`.
### A second, independent defect in the Iceberg transforms
The Iceberg partition transforms were derived from `datetime_diff()`, which
rounds **towards zero**, while Iceberg **floors**
(`DateTimeUtil.convertDays`/`convertMicros` evaluate one unit later for a
negative input and then subtract one). This affects **every pre-1970 value**
that is not exactly on a unit boundary, not just year zero:
| | Doris (before) | Iceberg |
|---|---:|---:|
| `year('1969-06-15')` | 0 | **-1** |
| `month('1969-06-15')` | -6 | **-7** |
| `day('1969-12-31 23:59:59')` | 0 | **-1** |
| `hour('1969-12-31 23:59:59')` | 0 | **-1** |
`regression-test/data/.../test_iceberg_write_partition_types_null.out` had
recorded this as expected: `1969-12-31 23:59:59` and `1970-01-01 00:00:00` were
both stored with `p_ts_day_day = 1970-01-01` and `p_ts_hour_hour = 0`, i.e.
**two distinct timestamps collapsed into one partition**.
## Solution
Add `daynr_to_epoch_days()` / `epoch_days_to_daynr()` in `vdatetime_value.h`
and route every format boundary through them. `calc_daynr()`, `is_leap()` and
all SQL-level date semantics (`TO_DAYS`, `DATEDIFF`, partition pruning) are
untouched.
- **Arrow** `date32`/`date64` read and write. Parquet export shares the
Arrow writer, so it is covered by the same change.
- **Parquet** `DATE` reader (`decode_parquet_date`), which also backs
min/max and dictionary predicates.
- **ORC** `DATE` reader and writer. The reader now passes the file's day
value straight through instead of laundering it through `date_day_offset_dict`;
that lookup also had a silent fallback which decoded any out-of-dictionary
value as `1900-01-01`, so a Spark-written `0000-01-01` used to read back as
`1900-01-01`.
- **Iceberg** `year`, `month`, `day`, `hour` and `bucket` for both `DATE`
and `TIMESTAMP`, plus `human_hour()` so a negative hour ordinal renders as
`1969-12-31-23` instead of `1970-01-01--1`.
- **FE** `ExpressionEstimation`'s `to_days()`/`from_days()` min/max model,
which had the same off-by-one.
Two behaviours change deliberately: `0000-02-29` (proleptic-only) is now
**rejected** rather than silently colliding with `0000-03-01`, and the Arrow
`date32` reader now checks the result of `get_date_from_daynr()` instead of
discarding it, matching what the `date64` path already did.
Both helpers compile to branchless code (`cmp`/`cset`, `cinc`/`ccmp`/`csel`
on arm64). Measured cost: **+0.19 ns per value on the write path**, read side a
wash. Three paths got *faster* because they no longer go through
`datetime_diff()` or a dictionary lookup: Iceberg `day(timestamp)` −38.7%,
`hour(timestamp)` −37.1%, ORC read −4.2%.
## Compatibility
Files previously written by Doris containing year-zero dates encoded
`-719527` for `0000-01-01`; they will now read back as `0000-01-02`. Those
files never conformed to the Parquet/ORC/Iceberg specs — Spark and Trino have
always read them as `0000-01-02`. Iceberg partition values for pre-1970 data
change to the spec-conformant values.
## Test
**BE unit tests — 38/38 pass**
- `epoch_days_conversion_matches_cctz_over_full_range` walks **all 3,652,424
representable Doris dates** and compares against `cctz::civil_day` (proleptic
Gregorian), plus a full round trip. Zero mismatches, 32 ms.
- Boundary and rejection tests for the conversion helpers.
- `DataTypeDateV2SerDeCalendarTest` (new): Arrow `date32`/`date64` read and
write, null map, lossless round trip, rejection of `0000-02-29` and
out-of-range values, and `ARRAY<DATE>` through the nested SerDe.
- Seven new `PartitionTransformersTest` cases whose expected values were
produced by **iceberg-api 1.10.1**, after first reproducing the spec's own
Appendix B test vector (`2017-11-16` → `-653330422`) to prove the harness is
faithful. The nine pre-existing transform cases (all using `2017-11-16`) still
pass unchanged, which shows post-1970 behaviour is untouched.
**Regression tests**
- `arrow_flight_sql_p0/test_date_year_zero` (new): compares the MySQL and
Arrow Flight protocols for scalar, nullable and `ARRAY<DATE>` columns. Reads
through `getString()` rather than `getObject()`, because `java.sql.Date` runs
year-zero values through the Julian/Gregorian hybrid calendar and would mask
the difference.
-
`external_table_p0/iceberg/write/test_iceberg_write_partition_epoch_boundary`
(new): writes identical boundary rows from **Doris and from Spark** into two
identically partitioned Iceberg tables and compares the resulting `$partitions`
metadata. All 7 rows × 8 partition fields match byte for byte. This suite has
real discriminating power — run against a build without the `year`/`month` fix
it correctly reported those two columns as divergent.
- `export_p0/outfile/test_outfile_date_year_zero` (new): Parquet/ORC export
and read-back through the S3 TVF.
- `test_iceberg_write_partition_types_null.out` updated: the `1969-12-31`
row moves from `0 / 0 / 1970-01-01 / 0` to `-1 / -1 / 1969-12-31 / -1`. The
other rows are unchanged, confirming the change only touches pre-epoch values.
- `test_iceberg_write_transform_partitions` and
`test_iceberg_write_partition_path` pass unchanged.
**Absolute verification of the on-disk encoding**: a raw byte scan of a
Doris-exported Parquet file finds `-719528` and `-719470` in place, with the
pre-fix `-719527`/`-719469` absent — so this is pinned against the file format,
not merely against a symmetric round trip.
Not run locally: `test_iceberg_write_partitions` needs the hive2 stack
(containers were down), and the ADBC suites need `libadbc_driver_flightsql`,
which is not installed in this thirdparty; their date literals start at 1900,
well inside the range where both calendars agree.
## Related findings, filed separately
Found while building and testing this, all unrelated to the change: #67445
(BE unit tests do not compile on aarch64), #67446 (Iceberg writes on an FE
start failing on a terminated worker pool), #67447 (year-zero `DATETIME`
survives the Parquet write but is rejected on read), #67448
(`kuromoji_build_dict` fails to link on arm64).
Fixes #67366
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
