Subodh-17 opened a new pull request, #119:
URL: https://github.com/apache/datafusion-spark/pull/119

   ## Summary
   
   Fixes premature integer overflow in Spark's `make_dt_interval` expression 
(#109).
   
   ### Context & Defect
   `make_dt_interval` returns an Arrow `Duration(Microsecond)` backed by a 
64-bit signed integer (`i64`), supporting durations up to $\approx \pm 106.75$ 
million days. However, intermediate conversions previously accumulated hours 
and minutes using 32-bit signed integers (`i32`). Consequently, any interval 
with `days >= 1,491,309` (or `days <= -1,491,309`) overflowed `i32` during 
minute calculation ($1,491,309 \times 24 \times 60 = 2,147,484,960 > 
\text{i32::MAX}$), prematurely returning `NULL` for valid day-time intervals 
that Apache Spark evaluates successfully.
   
   ### Key Changes
   1. **Widening Intermediate Accumulation to `i64`:** Promoted intermediate 
conversion constants `HOURS_PER_DAY` and `MINS_PER_HOUR` to `i64` and widened 
`day`, `hour`, and `min` to `i64` before multiplication and addition.
   2. **Defensive Arithmetic:** All accumulation steps use checked arithmetic 
(`checked_mul`, `checked_add`, `checked_sub`), safely returning `None` (`NULL`) 
on genuine overflow.
   3. **Float Seconds Range Guard:** Added float guard `const MAX_SECS_ABS: f64 
= (1i64 << 53) as f64;` rejecting non-finite floats and $|sec| \ge 2^{53}$. 
This prevents float-to-int cast saturation in `sec.trunc() as i64` without 
rejecting any representable interval (guard margin is $>44\times$ beyond the 
theoretical maximum seconds that could ever produce a representable duration 
under extreme opposite-sign cancellation).
   4. **Elimination of `.abs()`:** Replaced `frac_us.abs()` with safe signed 
comparisons (`frac_us >= MICROS_PER_SEC` and `frac_us <= -MICROS_PER_SEC`) and 
checked carry operations (`checked_add(1)?` / `checked_sub(1)?`), eliminating 
debug panic risks on `i64::MIN`.
   5. **Preserved True Overflow Contract (#20):** Genuine overflows beyond 
64-bit duration limits continue to return `NULL`, preserving the distinct 
architectural boundary tracked under #20 (error-raising on true overflow).
   
   ### Spark Parity & Compatibility
   - Verified against live Apache Spark 4.3.0 execution (with Py4J 0.10.9.9 and 
ANSI enabled). Spark returns valid day-time intervals up to 106,751,991 days, 
matching this implementation.
   - Note: The original issue description references PySpark 4.2.0; behavior is 
identical across Spark 4.x releases.
   
   ### Verification & Tests
   - **Focused Unit Tests:** 23/23 passed in 
`crates/spark/src/function/datetime/make_dt_interval.rs` (testing 
positive/negative boundaries, exact `i64::MAX`/`i64::MIN`, $\pm 1\ \mu\text{s}$ 
overflows, fractional carry rounding, and extreme floats).
   - **Focused SQLLogicTests:** 1/1 passed in 
`crates/sqllogictest/test_files/datetime/make_dt_interval.slt` (updating 
1,500,000 days from NULL to valid interval and adding positive/negative 
boundary assertions).
   - **Full Crate Tests:** 345/345 passed (`cargo test -p datafusion-spark`).
   - **Full SQLLogicTest Suite:** 245/245 test files passed (`cargo test -p 
datafusion-spark-sqllogictest`).
   - **Linters:** `cargo fmt --all -- --check` clean; `cargo clippy -p 
datafusion-spark --all-targets -- -D warnings` clean (0 warnings); `git diff 
--check` clean.
   
   Fixes #109.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to