Ritesh17rb commented on PR #392:
URL: https://github.com/apache/datafusion-spark/pull/392#issuecomment-7047341679

   Detailed validation for commit `18a4a00`.
   
   The main description is intentionally limited to the summary sections. 
Captured output, source audit notes, and platform-specific reproduction details 
are below.
   
   <details>
   <summary>Captured test, lint, and generation results</summary>
   
   The new SQL file was run against the unfixed base before adding the 
implementation:
   
   ```text
   FAIL [simplify] datetime\timestamp_millis.slt
   Error during planning: Invalid function 'timestamp_millis'.
   FAIL [evaluate] datetime\timestamp_millis.slt
   Error during planning: Invalid function 'timestamp_millis'.
   0 of 2 test runs passed
   ```
   
   After implementing the function:
   
   ```text
   test 
function::datetime::timestamp_millis::tests::return_field_tracks_session_timezone_and_input_nullability
 ... ok
   test function::datetime::timestamp_millis::tests::null_slots_do_not_overflow 
... ok
   
   test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 592 filtered out
   
   PASS [simplify] datetime\timestamp_millis.slt
   PASS [evaluate] datetime\timestamp_millis.slt
   2 of 2 test runs passed
   ```
   
   Full library and documentation tests:
   
   ```text
   test result: ok. 594 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
   
   test result: ok. 3 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out
   ```
   
   The two ignored documentation examples already exist in `implicit_cast.rs` 
and `lib.rs`.
   
   Full SQL suite:
   
   ```text
   516 of 516 test runs passed
   ```
   
   Spark comparison of the changed file:
   
   ```text
   Spark 4.2.0: 42 records in 1 file(s)
         38  pass
          4  skipped
                  3  arrow_cast to an unsigned integer, which Spark does not 
have
                  1  Spark has no function `arrow_typeof`
   ```
   
   All four Arrow-specific records pass locally in both SQL modes. Additional 
live Spark BIGINT probes confirm the numeric results used for the 
unsigned-value cases. A separate Spark 4.2.0 ground-truth harness confirmed 
exact `long overflow` errors in both ANSI modes, including scalar and column 
inputs.
   
   Formatting exited successfully with no differences. Workspace Clippy with 
all targets/features and `-D warnings` exited with code 0. The documentation 
generator completed with:
   
   ```text
   Wrote docs/functions
   DOCS_EXIT_CODE=0
   ```
   
   Regeneration also includes `dayofweek`'s existing `#[user_doc]` content, 
whose generated entry was missing from base `1181abf`.
   
   </details>
   
   <details>
   <summary>Validation commands and native Windows reproduction</summary>
   
   From the repository root:
   
   ```shell
   cargo fmt --all -- --check
   cargo test --locked -p datafusion-spark --lib timestamp_millis
   cargo test --locked -p datafusion-spark-sqllogictest -- timestamp_millis
   cargo test --locked -p datafusion-spark
   cargo test --locked -p datafusion-spark-sqllogictest
   cargo clippy --locked --all-targets --all-features --workspace -- -D warnings
   dev/update_function_docs.sh
   ```
   
   Using the basename as the SQL test filter works with both Windows and Unix 
path separators. No expected results were generated with `--complete`.
   
   The repository's Spark comparator can be run with its pinned dependency:
   
   ```shell
   python -m pip install -r crates/sqllogictest/scripts/requirements.txt
   python crates/sqllogictest/scripts/pyspark_slt_runner.py 
crates/sqllogictest/test_files/datetime/timestamp_millis.slt --verbose
   ```
   
   Local environment: native Windows 11 x64, Rust stable 1.100.0/MSVC, Python 
3.12, PySpark 4.2.0, and JDK 20.0.2. Builds used one job, 
`CARGO_PROFILE_DEV_DEBUG=0`, and `CARGO_INCREMENTAL=0` to limit memory use. The 
DataFusion dependency resolved to `branch-56` commit `3f197fe0`.
   
   On Windows, the comparator used the same UTC epoch-arithmetic result 
converter documented in [the previous PR's validation 
comment](https://github.com/apache/datafusion-spark/pull/381#issuecomment-7040926226),
 because Windows Python rejects negative epoch seconds in 
`datetime.fromtimestamp`. Spark evaluated the original queries. The separate 
ground-truth harness collected timestamp strings and integer counts to avoid 
that Python conversion limitation.
   
   The documentation script was run from Git Bash with the configured MSVC 
binary directory first in PATH. The local wrapper initializes the Visual Studio 
environment once. Cargo 1.100 prints pre-existing workspace-manifest warnings 
for `readme` and `rust-version`; these also appear in the baseline run. Rust 
Clippy itself passed with warnings denied.
   
   </details>
   
   <details>
   <summary>Spark source audit and coverage</summary>
   
   **Function summary:** integral milliseconds since the UTC epoch become a 
timestamp with a signed 64-bit microsecond count. NULL remains NULL. The 
input's nullability is preserved, and the session time zone affects display.
   
   **Version comparison:** read the registries, `IntegralToTimestampBase`, 
`MillisToTimestamp`, relevant unit tests, and timestamp golden outputs for 
Spark 3.5.8, 4.0.4, 4.1.3, and 4.2.0. No relevant behavior difference was 
found. The conversion uses `Math.multiplyExact(value, 1000)` in both 
interpreted and generated execution. Changing ANSI's default in Spark 4.x does 
not affect this function.
   
   Reference: [Spark 4.2.0 conversion 
implementation](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L562)
 and [its unit 
tests](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala#L1690).
   
   **Implementation review:** `SparkInputTypes::check_only` preserves the 
strict integral signature and shared argument errors from #381. Accepted widths 
widen to Int64 before the checked kernel. Scalar and array paths share 
`checked_mul(1000)`. The result carries microsecond precision and 
session-time-zone metadata. Comet's physical planner was also checked: it calls 
`coerce_types()` and inserts the returned widening casts before invoking the 
physical expression.
   
   | Dimension | Coverage |
   | --- | --- |
   | Scalar and column execution | SQL, in both execution modes |
   | Zero, negatives, pre-epoch timestamps | SQL and live Spark |
   | Untyped/typed NULL, mixed and all-null columns | SQL and live Spark |
   | Overflowing storage beneath an Arrow null mask | Direct Rust invocation 
test |
   | Both inclusive range endpoints | Integer round trips, avoiding 
calendar-format limits |
   | First out-of-range values and Int64 extrema | Scalar/column errors, ANSI 
on and off |
   | Signed integer widths | SQL and live Spark |
   | FLOAT/DOUBLE/DECIMAL/STRING/BOOLEAN rejection | Both ANSI modes |
   | Dictionaries and Arrow unsigned types | Local SQL in both modes |
   | Time zones and DST | Los Angeles display plus unchanged epoch counts |
   | Return precision, time zone, and nullability | Arrow SQL assertion and 
Rust metadata matrix |
   | Floating-point special values, collation, containers | Not applicable: 
non-integral inputs are rejected |
   
   Live Spark additionally confirmed that setting 
`spark.sql.timestampType=TIMESTAMP_NTZ` leaves this function's result as 
`timestamp`.
   
   **Findings:** the function was absent and the SQL file was a commented-out 
stub. This PR implements it and activates the regression coverage. No deferred 
divergence or additional coverage gap was identified within this function's 
scope. Comet and Sail were consulted as peer references; Spark source and live 
Spark results set the expectations.
   
   </details>
   
   Surfaced by the audit-datafusion-spark-expression skill.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to