ODriscollBoys opened a new pull request, #26174: URL: https://github.com/apache/datafusion/pull/26174
## Which issue does this PR close? - Closes #24907. ## Rationale for this change `COPY ... PARTITIONED BY` a `Date32` / `Date64` column panics the writer task when a value cannot be represented as a `chrono` date, e.g. `arrow_cast(2147483647, 'Date32')` hits `attempt to add with overflow` in `demux.rs` and `arrow_cast(-2147483648, 'Date32')` / `arrow_cast(9223372036854775807, 'Date64')` hit an `unwrap` on `None`. These should be ordinary execution errors. As noted in the issue, the `Date64` branch also divides by `86_400_000` with truncation toward zero, so a value just before the epoch (e.g. `-1` ms, which is `1969-12-31T23:59:59.999`) was written to the `1970-01-01` partition. #24908 proposed a fix for this and was closed together with a batch of other PRs rather than on its merits; this PR is a fresh take on current `main`. ## What changes are included in this PR? In `compute_partition_keys_by_row`, `Date32` and `Date64` values are converted with arrow's `date32_to_datetime` / `date64_to_datetime` instead of hand-rolled day arithmetic. These are the same conversions arrow uses when displaying or casting these types to strings, and they return `None` for out-of-range values, which is now reported as ``` Execution error: Date32 value 2147483647 of partition column 'column1' is out of range for a date ``` `date64_to_datetime` uses floor division, so non-whole-day `Date64` values now go to the partition of the day they fall on (`-1` ms -> `1969-12-31`), consistent with `arrow_cast(-1, 'Date64')` displaying as `1969-12-31T23:59:59.999`. The now unused `EPOCH_DAYS_FROM_CE` constant and `NaiveDate` import are removed. ## What is the testing strategy for this PR? New cases in `copy.slt`: - the three out-of-range values from the issue now fail with the execution error above (they panic on `main`); - `Date64` values `-1`, `-86400001` and `86399999` ms are written and read back through a partitioned external table as `1969-12-31`, `1969-12-30` and `1970-01-01`. The existing partitioned `COPY` tests for `Date32` / `Date64` (`partitioned_table5`) still pass, as does the rest of the sqllogictest suite. ## Are there any user-facing changes? - Out-of-range date partition values return an error instead of panicking. - `Date64` partition values that are not a whole number of days and fall before the epoch are now written to the correct (earlier) day's directory. Values that are whole days, which is what the Arrow spec expects for `Date64`, are unaffected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
