Vamsi-klu opened a new pull request, #58039:
URL: https://github.com/apache/spark/pull/58039
### What changes were proposed in this pull request?
Carry `TimeType(p)` precision across the PySpark Arrow / pandas type
mapping, matching the JVM fix in SPARK-57661 (#56778).
Arrow `time64` has no fractional-precision field, so `TIME(0)`, `TIME(3)`,
and `TIME(9)` all came back as `TIME(6)` after `toPandas` / `createDataFrame` /
`mapInArrow`.
This PR writes and reads `SPARK::time::precision` on the Arrow field (not
the Arrow type), same key as `ArrowUtils`:
- Tag every `pa.field` path in `to_arrow_schema` and the struct / array /
map branches of `to_arrow_type`.
- Recover precision from field metadata in `from_arrow_schema` and the
struct-fields loop in `from_arrow_type`. Accept `[0, 9]`. Missing, non-numeric,
or out-of-range values (`-1`, `10`) fall back to `TimeType()` (precision 6).
- Bare `from_arrow_type(time64)` stays `TimeType()` for foreign Arrow /
pandas inference.
- The key does not leak into reconstructed column metadata.
- No value conversion change. Precision is a type label only. Connect proto
mapping is unchanged.
### Why are the changes needed?
`python/pyspark/sql/pandas/types.py` mapped every `TimeType` to
`pa.time64("ns")` and read it back as `TimeType()`, which defaults to precision
6. The JVM path already keeps `p`. PySpark did not. Dongjoon called this out on
#56778, and Max filed this sibling ticket under SPARK-57550.
### Does this PR introduce _any_ user-facing change?
No. `spark.sql.timeType.enabled` stays internal and off in production.
Values are already nanoseconds and are unchanged. With the flag on, a `TIME(p)`
column keeps its declared precision instead of always reading back as `TIME(6)`.
### How was this patch tested?
Added `test_time_precision_arrow_round_trip` in
`python/pyspark/sql/tests/arrow/test_arrow.py`:
- Round-trip `TIME(p)` for `p` in `{0, 3, 6, 9}`. Arrow type stays
`time64[ns]`.
- Untagged `time64` and invalid keys (`-1`, `10`, `"x"`) fall back to
`TIME(6)`.
- Precision key does not leak into `StructField.metadata`.
- Nested struct / array / map fields keep precision.
Schema-level mapping checks for those cases passed. I did not run the full
`python/run-tests --testnames pyspark.sql.tests.arrow.test_arrow` suite locally
(needs `build/sbt -Phive package`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
