Doris-Breakwater commented on issue #67366: URL: https://github.com/apache/doris/issues/67366#issuecomment-5490602797
Initial assessment: **confirmed server-side BE serialization bug** at commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`. The raw value reported by the client is exactly what the Doris code produces, so additional logs or a Doris profile are not required to establish the initial root cause. ### Verified facts * Doris maps `DATEV2` to Arrow `date32` in [`convert_to_arrow_type`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/format/arrow/arrow_row_batch.cpp#L97-L99), and the Flight result converter delegates the value encoding to the type SerDe ([call site](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/format/arrow/arrow_block_convertor.cpp#L309-L317)). * [`DataTypeDateV2SerDe::write_column_to_arrow`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_datev2_serde.cpp#L182-L198) sends `DateV2Value::daynr() - 719528` directly to `Date32Builder`. * `daynr()` uses Doris's MySQL-compatible [`calc_daynr`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/value/vdatetime_value.h#L1734-L1764). That internal numbering is not identical to Arrow's proleptic-Gregorian day ordinal around the start of year zero. Doris also deliberately treats year zero as non-leap in [`is_leap`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/util/time_lut.h#L36-L38), while proleptic Gregorian year zero is divisible by 400. | Doris value | Doris `daynr()` | Current `date32` | Proleptic-Gregorian `date32` | |---|---:|---:|---:| | `0000-01-01` | 1 | -719527 | **-719528** | | `0000-02-28` | 59 | -719469 | **-719470** | | `0000-03-01` | 60 | -719468 | -719468 | | `2024-01-01` | 739251 | 19723 | 19723 | This confirms the reported values and narrows the affected range to valid Doris dates from `0000-01-01` through `0000-02-28`; this is not a blanket shift for every year-zero date. The issue's calendar-mismatch hypothesis is directionally correct, but the precise problem is reusing the Doris/MySQL internal ordinal without an explicit calendar-boundary conversion. `ARRAY<DATE>` is affected through the same code path: [`DataTypeArraySerDe`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_array_serde.cpp#L303-L320) delegates each element to the nested DATE SerDe. There is no separate array root cause. ### Important reverse-path consideration The fix should cover Arrow-to-Doris conversion as well as Doris-to-Arrow conversion. The current DATE32 reader adds `719528` and calls `get_date_from_daynr` ([code](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_datev2_serde.cpp#L240-L255)). A correct Arrow value of `-719528` for `0000-01-01` becomes internal day number `0`, which that API rejects; `-719470` for `0000-02-28` becomes `58` and decodes one day early. Fixing only the Flight output expression would therefore leave asymmetric round trips. ### Missing information Nothing is blocking initial confirmation. For a complete compatibility record, the exact `adbc_driver_flightsql` and PyArrow versions would still be useful. It would also be helpful to record raw values for `0000-01-02`, `0000-03-01`, and `0001-01-01`, but the checked source already predicts that only the pre-March year-zero boundary needs adjustment. ### Recommended next steps 1. Add a dedicated conversion helper between Doris DATE values and Arrow/Parquet proleptic-Gregorian day ordinals. Avoid changing shared `calc_daynr`, changing the epoch constant globally, or applying an unconditional `-1`: those approaches would break `0000-03-01` and/or the many SQL date operations that rely on Doris's existing numbering. 2. Add focused BE unit tests for both `write_column_to_arrow` and `read_column_from_arrow`, covering `0000-01-01`, `0000-02-28`, `0000-03-01`, `0001-01-01`, `1969-12-31`, `1970-01-01`, and a modern date. Include scalar, nullable, and nested array cases. 3. Add an end-to-end Flight SQL regression comparing native `date32` results with the MySQL/JDBC control path. 4. Audit the other users of the same ordinal assumption. In particular, the ORC writer also emits `daynr() - 719528`, and Parquet/Arrow input paths add the same threshold before calling `get_date_from_daynr`. This is code evidence of potential shared boundary behavior, but those formats have not been reproduced end to end here. 5. Land the primary fix and then evaluate a `branch-4.1` backport because the supplied reproducer is from 4.1.3-rc02. The current `CAST(... AS STRING)` workaround is valid until a native-type fix is available. The issue currently has no labels; `kind/bug` and `area/rpc` would be appropriate if those labels remain the repository convention. Breakwater-GitHub-Analysis-Slot: slot_eb5d685887c8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
