Ritesh17rb opened a new pull request, #381:
URL: https://github.com/apache/datafusion-spark/pull/381

   ## Which issue does this PR close?
   
   Closes #236.
   
   ## Rationale for this change
   
   Spark's `timestamp_micros` accepts signed integers and interprets them 
directly as microseconds since the UTC epoch. DataFusion's 
`to_timestamp_micros` also accepts floating-point, decimal, and string values, 
so its signature does not provide Spark-compatible validation.
   
   ## What changes are included in this PR?
   
   - Add and register `timestamp_micros`, including its Rust expression helper.
   - Accept TINYINT, SMALLINT, INT, BIGINT, and NULL, widening valid inputs to 
Int64; reject other input types with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`.
   - Return microsecond timestamps tagged with the session time zone, using UTC 
by default. Preserve the original epoch count, including `i64::MIN` and 
`i64::MAX`, and propagate input nullability.
   - Enable and extend the SQL logic tests; add a unit test for return-field 
type, session configuration, and nullability.
   - Add `#[user_doc]` documentation and regenerate the function reference.
   
   ## Are these changes tested?
   
   Yes. Local validation of commit `6ebf1e387925244063a1d2a34b343f6fdd96ffe4` 
passed on Windows x86_64 with Rust 1.100.0, Cargo 1.100.0, MSVC Build Tools 
2026, Python 3.12.7, Java 20.0.2, and PySpark 4.2.0.
   
   | Check | Command | Result |
   | --- | --- | --- |
   | Formatting | `cargo fmt --all -- --check` | Exit 0 |
   | Clippy | `cargo clippy --all-targets --all-features --workspace -- -D 
warnings` | Exit 0 |
   | Library and documentation tests | `cargo test -p datafusion-spark 
--all-features` | 584 unit tests passed; 5 documentation tests passed, 2 
ignored |
   | Full SQL logic-test suite | `cargo test -p datafusion-spark-sqllogictest` 
| 516/516 runs passed |
   | Generated function reference | `dev/update_function_docs.sh` | Exit 0; 
regenerated reference committed |
   | Spark 4.2.0 comparison | Repository PySpark runner, through the Windows 
compatibility wrapper below | 20 records passed; 1 DataFusion-specific record 
skipped; exit 0 |
   
   The [SQL 
tests](https://github.com/Ritesh17rb/datafusion-spark/blob/6ebf1e387925244063a1d2a34b343f6fdd96ffe4/crates/sqllogictest/test_files/datetime/timestamp_micros.slt)
 cover all four signed integer widths, scalar and column inputs, 
dictionary-encoded input, typed/untyped NULLs, negative epoch values, full 
Int64 boundaries, rejected argument types, ANSI-mode behavior, `unix_micros` 
round trips, session time zones, and a daylight-saving transition. They pass in 
both `simplify` and `evaluate` modes. The [unit 
test](https://github.com/Ritesh17rb/datafusion-spark/blob/6ebf1e387925244063a1d2a34b343f6fdd96ffe4/crates/spark/src/function/datetime/timestamp_micros.rs#L132)
 checks the return type and nullable/non-nullable input metadata with default 
and configured time zones.
   
   <details>
   <summary>Captured Rust test and lint output</summary>
   
   ```text
   test 
function::datetime::timestamp_micros::tests::return_field_tracks_session_timezone_and_input_nullability
 ... ok
   test result: ok. 584 passed; 0 failed; 0 ignored; 0 measured; 0 filtered 
out; finished in 1.78s
   test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; 
finished in 0.10s
   
   PASS [simplify] datetime\timestamp_micros.slt
   PASS [evaluate] datetime\timestamp_micros.slt
   516 of 516 test runs passed
   
       Finished `dev` profile [unoptimized] target(s) in 15.74s
   
   dev/update_function_docs.sh: Wrote docs/functions
   ```
   
   Formatting produced no output and exited successfully. The two ignored 
documentation examples are existing examples in `implicit_cast.rs` and 
`lib.rs`. Local builds used one build job, disabled debug symbols, and disabled 
incremental compilation to reduce memory use.
   
   </details>
   
   <details>
   <summary>Captured Spark 4.2.0 comparison output</summary>
   
   ```text
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:14: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:21: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:27: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:33: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:42: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:50: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:57: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:66: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:69: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:72: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:75: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:78: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:82: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:85: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:88: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:94: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:98: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:101: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:107: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:114: pass
   crates\sqllogictest\test_files\datetime\timestamp_micros.slt:118: skipped: 
Spark has no function `arrow_typeof`
   Spark 4.2.0: 21 records in 1 file(s)
         20  pass
          1  skipped
                  1  Spark has no function `arrow_typeof`
   ```
   
   The skipped record calls `arrow_typeof`, which Spark does not provide. The 
Arrow return type is asserted by the local SQL test and unit test above.
   
   </details>
   
   <details>
   <summary>Spark comparison reproduction and Windows compatibility 
details</summary>
   
   Install the repository's pinned PySpark dependency, then run its comparator 
against the changed file:
   
   ```shell
   python -m pip install -r crates/sqllogictest/scripts/requirements.txt
   python crates/sqllogictest/scripts/pyspark_slt_runner.py 
crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose
   ```
   
   For the local Windows verification, install `tzdata` for IANA time zones and 
save the following temporary wrapper as `windows_spark_check.py` in the 
repository root. Windows Python's `datetime.fromtimestamp` rejects negative 
epoch seconds; the wrapper decodes PySpark's returned microsecond count with 
UTC epoch arithmetic, as the repository comparator expects. This affects Python 
result conversion; Spark evaluates the original queries. The wrapper is 
included here for reproducibility and is not part of the library change.
   
   ```python
   """Run the upstream Spark comparator with Windows timestamp conversion 
support."""
   import datetime
   import runpy
   import sys
   from pathlib import Path
   
   from pyspark.sql.types import TimestampType
   
   
   def utc_from_internal(self, timestamp):
       # The upstream comparator expects UTC datetimes. Windows' fromtimestamp
       # rejects negative epoch seconds, so use epoch arithmetic in this 
process.
       if timestamp is None:
           return None
       return datetime.datetime(1970, 1, 1) + 
datetime.timedelta(microseconds=timestamp)
   
   
   TimestampType.fromInternal = utc_from_internal
   runner = Path.cwd() / "crates/sqllogictest/scripts/pyspark_slt_runner.py"
   sys.argv[0] = str(runner)
   runpy.run_path(str(runner), run_name="__main__")
   ```
   
   The command used locally was equivalent to:
   
   ```shell
   python -m pip install tzdata
   python windows_spark_check.py 
crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose
   ```
   
   On Windows, the documentation generation script was run from Git Bash with 
the MSVC binary directory placed first in PATH so that Rust selects Microsoft's 
`link.exe`.
   
   </details>
   
   ## Are there any user-facing changes?
   
   Yes. Spark mode now provides `timestamp_micros(microseconds)`, for example 
`timestamp_micros(0::BIGINT)` returns the UTC epoch. NULL inputs return NULL, 
and floating-point, decimal, string, and boolean arguments are rejected during 
planning.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to