Ritesh17rb commented on PR #392:
URL: https://github.com/apache/datafusion-spark/pull/392#issuecomment-7047341679
Detailed validation for commit `18a4a00`.
The main description is intentionally limited to the summary sections.
Captured output, source audit notes, and platform-specific reproduction details
are below.
<details>
<summary>Captured test, lint, and generation results</summary>
The new SQL file was run against the unfixed base before adding the
implementation:
```text
FAIL [simplify] datetime\timestamp_millis.slt
Error during planning: Invalid function 'timestamp_millis'.
FAIL [evaluate] datetime\timestamp_millis.slt
Error during planning: Invalid function 'timestamp_millis'.
0 of 2 test runs passed
```
After implementing the function:
```text
test
function::datetime::timestamp_millis::tests::return_field_tracks_session_timezone_and_input_nullability
... ok
test function::datetime::timestamp_millis::tests::null_slots_do_not_overflow
... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 592 filtered out
PASS [simplify] datetime\timestamp_millis.slt
PASS [evaluate] datetime\timestamp_millis.slt
2 of 2 test runs passed
```
Full library and documentation tests:
```text
test result: ok. 594 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out
```
The two ignored documentation examples already exist in `implicit_cast.rs`
and `lib.rs`.
Full SQL suite:
```text
516 of 516 test runs passed
```
Spark comparison of the changed file:
```text
Spark 4.2.0: 42 records in 1 file(s)
38 pass
4 skipped
3 arrow_cast to an unsigned integer, which Spark does not
have
1 Spark has no function `arrow_typeof`
```
All four Arrow-specific records pass locally in both SQL modes. Additional
live Spark BIGINT probes confirm the numeric results used for the
unsigned-value cases. A separate Spark 4.2.0 ground-truth harness confirmed
exact `long overflow` errors in both ANSI modes, including scalar and column
inputs.
Formatting exited successfully with no differences. Workspace Clippy with
all targets/features and `-D warnings` exited with code 0. The documentation
generator completed with:
```text
Wrote docs/functions
DOCS_EXIT_CODE=0
```
Regeneration also includes `dayofweek`'s existing `#[user_doc]` content,
whose generated entry was missing from base `1181abf`.
</details>
<details>
<summary>Validation commands and native Windows reproduction</summary>
From the repository root:
```shell
cargo fmt --all -- --check
cargo test --locked -p datafusion-spark --lib timestamp_millis
cargo test --locked -p datafusion-spark-sqllogictest -- timestamp_millis
cargo test --locked -p datafusion-spark
cargo test --locked -p datafusion-spark-sqllogictest
cargo clippy --locked --all-targets --all-features --workspace -- -D warnings
dev/update_function_docs.sh
```
Using the basename as the SQL test filter works with both Windows and Unix
path separators. No expected results were generated with `--complete`.
The repository's Spark comparator can be run with its pinned dependency:
```shell
python -m pip install -r crates/sqllogictest/scripts/requirements.txt
python crates/sqllogictest/scripts/pyspark_slt_runner.py
crates/sqllogictest/test_files/datetime/timestamp_millis.slt --verbose
```
Local environment: native Windows 11 x64, Rust stable 1.100.0/MSVC, Python
3.12, PySpark 4.2.0, and JDK 20.0.2. Builds used one job,
`CARGO_PROFILE_DEV_DEBUG=0`, and `CARGO_INCREMENTAL=0` to limit memory use. The
DataFusion dependency resolved to `branch-56` commit `3f197fe0`.
On Windows, the comparator used the same UTC epoch-arithmetic result
converter documented in [the previous PR's validation
comment](https://github.com/apache/datafusion-spark/pull/381#issuecomment-7040926226),
because Windows Python rejects negative epoch seconds in
`datetime.fromtimestamp`. Spark evaluated the original queries. The separate
ground-truth harness collected timestamp strings and integer counts to avoid
that Python conversion limitation.
The documentation script was run from Git Bash with the configured MSVC
binary directory first in PATH. The local wrapper initializes the Visual Studio
environment once. Cargo 1.100 prints pre-existing workspace-manifest warnings
for `readme` and `rust-version`; these also appear in the baseline run. Rust
Clippy itself passed with warnings denied.
</details>
<details>
<summary>Spark source audit and coverage</summary>
**Function summary:** integral milliseconds since the UTC epoch become a
timestamp with a signed 64-bit microsecond count. NULL remains NULL. The
input's nullability is preserved, and the session time zone affects display.
**Version comparison:** read the registries, `IntegralToTimestampBase`,
`MillisToTimestamp`, relevant unit tests, and timestamp golden outputs for
Spark 3.5.8, 4.0.4, 4.1.3, and 4.2.0. No relevant behavior difference was
found. The conversion uses `Math.multiplyExact(value, 1000)` in both
interpreted and generated execution. Changing ANSI's default in Spark 4.x does
not affect this function.
Reference: [Spark 4.2.0 conversion
implementation](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L562)
and [its unit
tests](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala#L1690).
**Implementation review:** `SparkInputTypes::check_only` preserves the
strict integral signature and shared argument errors from #381. Accepted widths
widen to Int64 before the checked kernel. Scalar and array paths share
`checked_mul(1000)`. The result carries microsecond precision and
session-time-zone metadata. Comet's physical planner was also checked: it calls
`coerce_types()` and inserts the returned widening casts before invoking the
physical expression.
| Dimension | Coverage |
| --- | --- |
| Scalar and column execution | SQL, in both execution modes |
| Zero, negatives, pre-epoch timestamps | SQL and live Spark |
| Untyped/typed NULL, mixed and all-null columns | SQL and live Spark |
| Overflowing storage beneath an Arrow null mask | Direct Rust invocation
test |
| Both inclusive range endpoints | Integer round trips, avoiding
calendar-format limits |
| First out-of-range values and Int64 extrema | Scalar/column errors, ANSI
on and off |
| Signed integer widths | SQL and live Spark |
| FLOAT/DOUBLE/DECIMAL/STRING/BOOLEAN rejection | Both ANSI modes |
| Dictionaries and Arrow unsigned types | Local SQL in both modes |
| Time zones and DST | Los Angeles display plus unchanged epoch counts |
| Return precision, time zone, and nullability | Arrow SQL assertion and
Rust metadata matrix |
| Floating-point special values, collation, containers | Not applicable:
non-integral inputs are rejected |
Live Spark additionally confirmed that setting
`spark.sql.timestampType=TIMESTAMP_NTZ` leaves this function's result as
`timestamp`.
**Findings:** the function was absent and the SQL file was a commented-out
stub. This PR implements it and activates the regression coverage. No deferred
divergence or additional coverage gap was identified within this function's
scope. Comet and Sail were consulted as peer references; Spark source and live
Spark results set the expectations.
</details>
Surfaced by the audit-datafusion-spark-expression skill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]