Hyukjin Kwon created SPARK-62058:
------------------------------------
Summary: Log the events of a pipeline run at the exact times of
their timestamps, as PySpark's Timestamp.ToDatetime() gives them
Key: SPARK-62058
URL: https://issues.apache.org/jira/browse/SPARK-62058
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark 4.3.0's handle_pipeline_events logs each event of a pipeline run at
event.timestamp.ToDatetime().replace(tzinfo=timezone.utc)
(python/pyspark/pipelines/spark_connect_pipeline.py:49-65, line 64). protobuf's
Timestamp.ToDatetime is exact: the epoch plus timedelta(seconds=seconds,
microseconds=nanos // 1000)
(google/protobuf/internal/well_known_types.py:239-266), and
log_with_provided_timestamp prints the time to the second
(pyspark/pipelines/logging_utils.py:33-42).
The drop-in's run stream yields the microseconds of the event's Timestamp, and
its
handle_pipeline_events (python/pyspark/pipelines/spark_connect_pipeline.py)
converted them with
datetime.fromtimestamp(timestamp_micros / 1_000_000, tz=timezone.utc): a float
of the seconds,
whose spacing exceeds a microsecond for a time far from 1970. So an event whose
time is far from
1970 can be logged a second late, and an event at the last microsecond of 9999
fails the run's
log. Events of the years near 1970, before it too, convert the same.
Repro (master 80046f2f; the stream's values as pipeline_start_run gives them)
from pyspark.pipelines.spark_connect_pipeline import handle_pipeline_events
handle_pipeline_events(iter([
("first", -62135596800 * 1_000_000 + 999_999), # 0001-01-01
00:00:00.999999
("last", 253402300799 * 1_000_000 + 999_999), # 9999-12-31
23:59:59.999999
]))
Actual
1-01-01 00:00:01: first
ValueError: year 10000 is out of range
Expected (as PySpark's ToDatetime of the same Timestamps)
1-01-01 00:00:00: first
9999-12-31 23:59:59: last
Also investigated, and the same in both clients (pyspark-client 4.3.0 against
the drop-in, against
a Spark 4.3.0 server, in the session time zones UTC, America/Los_Angeles and
Asia/Kolkata and the
host time zones UTC, America/Los_Angeles and America/New_York): times before
1970 with
nanoseconds through createDataFrame (datetimes, pandas Timestamps, NumPy
datetime64[ns], pandas
datetime64[ns] and object columns, pyarrow timestamp[ns] Tables), TimestampType
and
TimestampNTZType, through collect, toLocalIterator, toPandas and toArrow,
literals, and the
Timestamp of the Rust protobuf runtime. PySpark rounds them in two directions,
which the drop-in
follows: local rows (conversion.py:784, PyArrow reading the fields of the
datetime), literals,
object columns and pyarrow Tables (pandas/types.py:554, pc.floor_temporal)
round down to the
microsecond, and a datetime64[ns] column of a pandas DataFrame rounds toward
1970 when the safe
cast is off (pandas/conversion.py:117, PyArrow's cast).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]