Ritesh17rb commented on PR #381:
URL: https://github.com/apache/datafusion-spark/pull/381#issuecomment-7040926226

   Moving the detailed validation output and Windows reproduction instructions 
out of the PR description, as requested. These logs describe the original 
commit `6ebf1e3`; validation of the review fixes will be summarized separately.
   
   <details>
   <summary>Captured Rust test and lint output</summary>
   
   ```text
   test 
function::datetime::timestamp_micros::tests::return_field_tracks_session_timezone_and_input_nullability
 ... ok
   test result: ok. 584 passed; 0 failed; 0 ignored; 0 measured; 0 filtered 
out; finished in 1.78s
   test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; 
finished in 0.10s
   
   PASS [simplify] datetime\timestamp_micros.slt
   PASS [evaluate] datetime\timestamp_micros.slt
   516 of 516 test runs passed
   
       Finished `dev` profile [unoptimized] target(s) in 15.74s
   
   dev/update_function_docs.sh: Wrote docs/functions
   ```
   
   Formatting produced no output and exited successfully. The two ignored 
documentation examples are existing examples in `implicit_cast.rs` and 
`lib.rs`. Local builds used one build job, disabled debug symbols, and disabled 
incremental compilation to reduce memory use.
   
   </details>
   
   <details>
   <summary>Captured Spark 4.2.0 comparison output</summary>
   
   ```text
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:14: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:21: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:27: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:33: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:42: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:50: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:57: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:66: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:69: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:72: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:75: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:78: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:82: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:85: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:88: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:94: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:98: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:101: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:107: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:114: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:118: skipped: 
Spark has no function `arrow_typeof`
   Spark 4.2.0: 21 records in 1 file(s)
         20  pass
          1  skipped
                  1  Spark has no function `arrow_typeof`
   ```
   
   The skipped record calls `arrow_typeof`, which Spark does not provide. The 
Arrow return type is asserted by the local SQL test and unit test above.
   
   </details>
   
   <details>
   <summary>Spark comparison reproduction and Windows compatibility 
details</summary>
   
   Install the repository's pinned PySpark dependency, then run its comparator 
against the changed file:
   
   ```shell
   python -m pip install -r crates/sqllogictest/scripts/requirements.txt
   python crates/sqllogictest/scripts/pyspark_slt_runner.py 
crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose
   ```
   
   For the local Windows verification, install `tzdata` for IANA time zones and 
save the following temporary wrapper as `windows_spark_check.py` in the 
repository root. Windows Python's `datetime.fromtimestamp` rejects negative 
epoch seconds; the wrapper decodes PySpark's returned microsecond count with 
UTC epoch arithmetic, as the repository comparator expects. This affects Python 
result conversion; Spark evaluates the original queries. The wrapper is 
included here for reproducibility and is not part of the library change.
   
   ```python
   """Run the upstream Spark comparator with Windows timestamp conversion 
support."""
   import datetime
   import runpy
   import sys
   from pathlib import Path
   
   from pyspark.sql.types import TimestampType
   
   
   def utc_from_internal(self, timestamp):
       # The upstream comparator expects UTC datetimes. Windows' fromtimestamp
       # rejects negative epoch seconds, so use epoch arithmetic in this 
process.
       if timestamp is None:
           return None
       return datetime.datetime(1970, 1, 1) + 
datetime.timedelta(microseconds=timestamp)
   
   
   TimestampType.fromInternal = utc_from_internal
   runner = Path.cwd() / "crates/sqllogictest/scripts/pyspark_slt_runner.py"
   sys.argv[0] = str(runner)
   runpy.run_path(str(runner), run_name="__main__")
   ```
   
   The command used locally was equivalent to:
   
   ```shell
   python -m pip install tzdata
   python windows_spark_check.py 
crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose
   ```
   
   On Windows, the documentation generation script was run from Git Bash with 
the MSVC binary directory placed first in PATH so that Rust selects Microsoft's 
`link.exe`.
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to