HyukjinKwon opened a new pull request, #259:
URL: https://github.com/apache/spark-connect-rust/pull/259
### What changes were proposed in this pull request?
- The Arrow decoder keeps TIMESTAMP_NTZ values apart, as
`Value::TimestampNtz`, from an Arrow timestamp without a time zone, as Spark's
`ArrowUtils` declares NTZ (`crates/spark-connect/src/dataframe.rs`, `row.rs`;
the encoder, literals and inference take the new variant too). The Python
conversion makes an NTZ value the wall clock, `datetime(1970, 1, 1) +
timedelta(microseconds=...)`, in any client time zone, nested in arrays, maps
and structs too (`crates/pyspark-rs/src/row.rs`).
- TIMESTAMP values convert with exact integer arithmetic to the client's
local time, as PySpark's conversion does, instead of `datetime.fromtimestamp`
of a float, which lost a microsecond for some values.
- `collect`, `first`, `head`, `take` and `toLocalIterator` raise
`PySparkNotImplementedError` `NOT_IMPLEMENTED` ("Collecting a year-month
interval value in Spark Connect") for a schema with a year-month interval,
nested or not, empty results included, as PySpark does
(`crates/pyspark-rs/src/dataframe.rs`). The Rust core still renders such values
as strings.
### Why are the changes needed?
[SPARK-61069](https://issues.apache.org/jira/browse/SPARK-61069): against a
local Spark 4.3.0 server (session time zone UTC):
| Code (client `TZ=America/Los_Angeles` for the first two) | Current master
| This change and PySpark 4.3.0 |
| --- | --- | --- |
| `spark.sql("select timestamp_ntz'2020-07-01 12:00:00' t").collect()` |
`datetime(2020, 7, 1, 5, 0)` | `datetime(2020, 7, 1, 12, 0)` |
| an NTZ `0001-01-01` | `ValueError: year 0 is out of range` | `datetime(1,
1, 1, 0, 0)` |
| `spark.sql("select timestamp_micros(9223372036854775) t").collect()` (UTC)
| `...16.854776` | `...16.854775` |
| `spark.sql("SELECT INTERVAL '1-2' YEAR TO MONTH ym").collect()` |
`[Row(ym='1-2')]` | `PySparkNotImplementedError [NOT_IMPLEMENTED]` |
1,812 random and extreme microsecond values, in three time zones, as
TIMESTAMP and as TIMESTAMP_NTZ, now collect identically to PySpark 4.3.0.
### Does this PR introduce _any_ user-facing change?
Yes, these values collect as in PySpark.
### How was this patch tested?
- New offline tests in `python/tests/test_dropin_offline.py` for NTZ values
in several time zones and years, nested; exact TIMESTAMP values; and year-month
intervals through every collecting API; live tests in
`python/tests/test_dropin_e2e.py` with `TZ=America/Los_Angeles` and for
year-month intervals; and decoder unit tests for NTZ.
- The lib tests of `apache-spark-connect` (458 passed), the offline suite
(949 passed, 8 skipped), and the live suite.
Unchanged: `toArrow()` of a year-month interval returns a table where
PySpark raises `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
