Max Gekk created SPARK-57696:
--------------------------------
Summary: Preserve TIME precision in the PySpark Arrow/pandas type
mapping
Key: SPARK-57696
URL: https://issues.apache.org/jira/browse/SPARK-57696
Project: Spark
Issue Type: Sub-task
Components: Pandas API on Spark
Affects Versions: 4.3.0
Reporter: Max Gekk
h2. What
Carry the {{TimeType(p)}} fractional-second precision {{p}} (in [0, 9]) across
the
PySpark Arrow/pandas type mapping in {{python/pyspark/sql/pandas/types.py}} so
that a
{{TIME(p)}} column round-trips back to the same {{TIME(p)}}, instead of
collapsing to the
default {{TimeType()}} (precision 6).
This is the PySpark counterpart of SPARK-57661, which fixes the same gap in the
JVM
{{ArrowUtils}} (Spark <-> Arrow) mapping.
h2. Why
{{to_arrow_type}} maps {{TimeType}} to {{pa.time64("ns")}}, which carries only
a time unit
and no fractional-second precision field. {{from_arrow_type}} maps any
{{is_time64(at)}}
back to {{TimeType()}}, whose constructor defaults to {{precision = 6}}. As a
result the
declared precision is lost on any Arrow/pandas round-trip ({{TIME(0)}},
{{TIME(3)}},
{{TIME(9)}}, ... all read back as {{TIME(6)}}), so PySpark Arrow-based transfer
(createDataFrame from Arrow/pandas, {{mapInArrow}}, {{toPandas}} schema, etc.)
silently
widens or narrows the type label. The stored values are unaffected; this is
purely a
type-fidelity gap.
PyArrow's {{time64}} logical type only encodes the unit (us/ns) and cannot
represent
precision [0, 9], so the precision must be carried in the Arrow field metadata,
mirroring
the {{SPARK::time::precision}} key the JVM {{ArrowUtils}} uses in SPARK-57661.
h2. Scope
* {{python/pyspark/sql/pandas/types.py}}: when building the Arrow field for a
{{TimeType(p)}},
tag it with the precision metadata key {{SPARK::time::precision}}; when reading
an Arrow
{{time64}} field back, read that key to reconstruct {{TimeType(p)}}.
* Reuse the same metadata key and fallback semantics as the JVM side
(SPARK-57661) for
consistency.
h2. Behavior on read-back
* Metadata present and in [0, 9]: reconstruct the exact {{TimeType(p)}}.
* Metadata absent (foreign Arrow data) or out of [0, 9]: fall back to the
default
{{TimeType()}} (precision 6), preserving today's behavior for non-Spark
producers.
h2. Out of scope
* Value semantics / rounding: values are carried verbatim; no change to how
{{TIME(p)}}
values are truncated.
* The JVM {{ArrowUtils}} mapping (handled in SPARK-57661) and the Spark Connect
proto/converters (separate sub-tasks).
h2. How tested
PySpark Arrow conversion tests: round-trip {{TIME(p)}} for {{p}} in {0, 3, 6,
9} preserves
{{p}}; a {{time64}} field with no precision metadata falls back to {{TIME(6)}};
the
precision key does not leak into reconstructed field metadata.
h2. Does this introduce any user-facing change
No. The TIME data type is gated behind the internal flag
{{spark.sql.timeType.enabled}},
which is off by default in production. With the flag enabled, a {{TIME(p)}}
column
transferred over Arrow/pandas in PySpark retains its declared precision instead
of always
reading back as {{TIME(6)}}. No change to stored values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
