Ritesh17rb commented on PR #381:
URL: https://github.com/apache/datafusion-spark/pull/381#issuecomment-7040926226
Moving the detailed validation output and Windows reproduction instructions
out of the PR description, as requested. These logs describe the original
commit `6ebf1e3`; validation of the review fixes will be summarized separately.
<details>
<summary>Captured Rust test and lint output</summary>
```text
test
function::datetime::timestamp_micros::tests::return_field_tracks_session_timezone_and_input_nullability
... ok
test result: ok. 584 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 1.78s
test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out;
finished in 0.10s
PASS [simplify] datetime\timestamp_micros.slt
PASS [evaluate] datetime\timestamp_micros.slt
516 of 516 test runs passed
Finished `dev` profile [unoptimized] target(s) in 15.74s
dev/update_function_docs.sh: Wrote docs/functions
```
Formatting produced no output and exited successfully. The two ignored
documentation examples are existing examples in `implicit_cast.rs` and
`lib.rs`. Local builds used one build job, disabled debug symbols, and disabled
incremental compilation to reduce memory use.
</details>
<details>
<summary>Captured Spark 4.2.0 comparison output</summary>
```text
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:14: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:21: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:27: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:33: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:42: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:50: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:57: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:66: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:69: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:72: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:75: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:78: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:82: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:85: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:88: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:94: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:98: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:101: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:107: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:114: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:118: skipped:
Spark has no function `arrow_typeof`
Spark 4.2.0: 21 records in 1 file(s)
20 pass
1 skipped
1 Spark has no function `arrow_typeof`
```
The skipped record calls `arrow_typeof`, which Spark does not provide. The
Arrow return type is asserted by the local SQL test and unit test above.
</details>
<details>
<summary>Spark comparison reproduction and Windows compatibility
details</summary>
Install the repository's pinned PySpark dependency, then run its comparator
against the changed file:
```shell
python -m pip install -r crates/sqllogictest/scripts/requirements.txt
python crates/sqllogictest/scripts/pyspark_slt_runner.py
crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose
```
For the local Windows verification, install `tzdata` for IANA time zones and
save the following temporary wrapper as `windows_spark_check.py` in the
repository root. Windows Python's `datetime.fromtimestamp` rejects negative
epoch seconds; the wrapper decodes PySpark's returned microsecond count with
UTC epoch arithmetic, as the repository comparator expects. This affects Python
result conversion; Spark evaluates the original queries. The wrapper is
included here for reproducibility and is not part of the library change.
```python
"""Run the upstream Spark comparator with Windows timestamp conversion
support."""
import datetime
import runpy
import sys
from pathlib import Path
from pyspark.sql.types import TimestampType
def utc_from_internal(self, timestamp):
# The upstream comparator expects UTC datetimes. Windows' fromtimestamp
# rejects negative epoch seconds, so use epoch arithmetic in this
process.
if timestamp is None:
return None
return datetime.datetime(1970, 1, 1) +
datetime.timedelta(microseconds=timestamp)
TimestampType.fromInternal = utc_from_internal
runner = Path.cwd() / "crates/sqllogictest/scripts/pyspark_slt_runner.py"
sys.argv[0] = str(runner)
runpy.run_path(str(runner), run_name="__main__")
```
The command used locally was equivalent to:
```shell
python -m pip install tzdata
python windows_spark_check.py
crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose
```
On Windows, the documentation generation script was run from Git Bash with
the MSVC binary directory placed first in PATH so that Rust selects Microsoft's
`link.exe`.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]