Ritesh17rb opened a new pull request, #381: URL: https://github.com/apache/datafusion-spark/pull/381
## Which issue does this PR close? Closes #236. ## Rationale for this change Spark's `timestamp_micros` accepts signed integers and interprets them directly as microseconds since the UTC epoch. DataFusion's `to_timestamp_micros` also accepts floating-point, decimal, and string values, so its signature does not provide Spark-compatible validation. ## What changes are included in this PR? - Add and register `timestamp_micros`, including its Rust expression helper. - Accept TINYINT, SMALLINT, INT, BIGINT, and NULL, widening valid inputs to Int64; reject other input types with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`. - Return microsecond timestamps tagged with the session time zone, using UTC by default. Preserve the original epoch count, including `i64::MIN` and `i64::MAX`, and propagate input nullability. - Enable and extend the SQL logic tests; add a unit test for return-field type, session configuration, and nullability. - Add `#[user_doc]` documentation and regenerate the function reference. ## Are these changes tested? Yes. Local validation of commit `6ebf1e387925244063a1d2a34b343f6fdd96ffe4` passed on Windows x86_64 with Rust 1.100.0, Cargo 1.100.0, MSVC Build Tools 2026, Python 3.12.7, Java 20.0.2, and PySpark 4.2.0. | Check | Command | Result | | --- | --- | --- | | Formatting | `cargo fmt --all -- --check` | Exit 0 | | Clippy | `cargo clippy --all-targets --all-features --workspace -- -D warnings` | Exit 0 | | Library and documentation tests | `cargo test -p datafusion-spark --all-features` | 584 unit tests passed; 5 documentation tests passed, 2 ignored | | Full SQL logic-test suite | `cargo test -p datafusion-spark-sqllogictest` | 516/516 runs passed | | Generated function reference | `dev/update_function_docs.sh` | Exit 0; regenerated reference committed | | Spark 4.2.0 comparison | Repository PySpark runner, through the Windows compatibility wrapper below | 20 records passed; 1 DataFusion-specific record skipped; exit 0 | The [SQL tests](https://github.com/Ritesh17rb/datafusion-spark/blob/6ebf1e387925244063a1d2a34b343f6fdd96ffe4/crates/sqllogictest/test_files/datetime/timestamp_micros.slt) cover all four signed integer widths, scalar and column inputs, dictionary-encoded input, typed/untyped NULLs, negative epoch values, full Int64 boundaries, rejected argument types, ANSI-mode behavior, `unix_micros` round trips, session time zones, and a daylight-saving transition. They pass in both `simplify` and `evaluate` modes. The [unit test](https://github.com/Ritesh17rb/datafusion-spark/blob/6ebf1e387925244063a1d2a34b343f6fdd96ffe4/crates/spark/src/function/datetime/timestamp_micros.rs#L132) checks the return type and nullable/non-nullable input metadata with default and configured time zones. <details> <summary>Captured Rust test and lint output</summary> ```text test function::datetime::timestamp_micros::tests::return_field_tracks_session_timezone_and_input_nullability ... ok test result: ok. 584 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.78s test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.10s PASS [simplify] datetime\timestamp_micros.slt PASS [evaluate] datetime\timestamp_micros.slt 516 of 516 test runs passed Finished `dev` profile [unoptimized] target(s) in 15.74s dev/update_function_docs.sh: Wrote docs/functions ``` Formatting produced no output and exited successfully. The two ignored documentation examples are existing examples in `implicit_cast.rs` and `lib.rs`. Local builds used one build job, disabled debug symbols, and disabled incremental compilation to reduce memory use. </details> <details> <summary>Captured Spark 4.2.0 comparison output</summary> ```text crates\sqllogictest\test_files\datetime\timestamp_micros.slt:14: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:21: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:27: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:33: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:42: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:50: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:57: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:66: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:69: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:72: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:75: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:78: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:82: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:85: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:88: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:94: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:98: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:101: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:107: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:114: pass crates\sqllogictest\test_files\datetime\timestamp_micros.slt:118: skipped: Spark has no function `arrow_typeof` Spark 4.2.0: 21 records in 1 file(s) 20 pass 1 skipped 1 Spark has no function `arrow_typeof` ``` The skipped record calls `arrow_typeof`, which Spark does not provide. The Arrow return type is asserted by the local SQL test and unit test above. </details> <details> <summary>Spark comparison reproduction and Windows compatibility details</summary> Install the repository's pinned PySpark dependency, then run its comparator against the changed file: ```shell python -m pip install -r crates/sqllogictest/scripts/requirements.txt python crates/sqllogictest/scripts/pyspark_slt_runner.py crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose ``` For the local Windows verification, install `tzdata` for IANA time zones and save the following temporary wrapper as `windows_spark_check.py` in the repository root. Windows Python's `datetime.fromtimestamp` rejects negative epoch seconds; the wrapper decodes PySpark's returned microsecond count with UTC epoch arithmetic, as the repository comparator expects. This affects Python result conversion; Spark evaluates the original queries. The wrapper is included here for reproducibility and is not part of the library change. ```python """Run the upstream Spark comparator with Windows timestamp conversion support.""" import datetime import runpy import sys from pathlib import Path from pyspark.sql.types import TimestampType def utc_from_internal(self, timestamp): # The upstream comparator expects UTC datetimes. Windows' fromtimestamp # rejects negative epoch seconds, so use epoch arithmetic in this process. if timestamp is None: return None return datetime.datetime(1970, 1, 1) + datetime.timedelta(microseconds=timestamp) TimestampType.fromInternal = utc_from_internal runner = Path.cwd() / "crates/sqllogictest/scripts/pyspark_slt_runner.py" sys.argv[0] = str(runner) runpy.run_path(str(runner), run_name="__main__") ``` The command used locally was equivalent to: ```shell python -m pip install tzdata python windows_spark_check.py crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose ``` On Windows, the documentation generation script was run from Git Bash with the MSVC binary directory placed first in PATH so that Rust selects Microsoft's `link.exe`. </details> ## Are there any user-facing changes? Yes. Spark mode now provides `timestamp_micros(microseconds)`, for example `timestamp_micros(0::BIGINT)` returns the UTC epoch. NULL inputs return NULL, and floating-point, decimal, string, and boolean arguments are rejected during planning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
