[
https://issues.apache.org/jira/browse/SPARK-62058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62058:
------------------------------------
Assignee: Hyukjin Kwon
> Log the events of a pipeline run at the exact times of their timestamps, as
> PySpark's Timestamp.ToDatetime() gives them
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62058
> URL: https://issues.apache.org/jira/browse/SPARK-62058
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> PySpark 4.3.0's handle_pipeline_events logs each event of a pipeline run at
> event.timestamp.ToDatetime().replace(tzinfo=timezone.utc)
> (python/pyspark/pipelines/spark_connect_pipeline.py:49-65, line 64).
> protobuf's
> Timestamp.ToDatetime is exact: the epoch plus timedelta(seconds=seconds,
> microseconds=nanos // 1000)
> (google/protobuf/internal/well_known_types.py:239-266), and
> log_with_provided_timestamp prints the time to the second
> (pyspark/pipelines/logging_utils.py:33-42).
> The drop-in's run stream yields the microseconds of the event's Timestamp,
> and its
> handle_pipeline_events (python/pyspark/pipelines/spark_connect_pipeline.py)
> converted them with
> datetime.fromtimestamp(timestamp_micros / 1_000_000, tz=timezone.utc): a
> float of the seconds,
> whose spacing exceeds a microsecond for a time far from 1970. So an event
> whose time is far from
> 1970 can be logged a second late, and an event at the last microsecond of
> 9999 fails the run's
> log. Events of the years near 1970, before it too, convert the same.
> Repro (master 80046f2f; the stream's values as pipeline_start_run gives them)
> from pyspark.pipelines.spark_connect_pipeline import handle_pipeline_events
> handle_pipeline_events(iter([
> ("first", -62135596800 * 1_000_000 + 999_999), # 0001-01-01
> 00:00:00.999999
> ("last", 253402300799 * 1_000_000 + 999_999), # 9999-12-31
> 23:59:59.999999
> ]))
> Actual
> 1-01-01 00:00:01: first
> ValueError: year 10000 is out of range
> Expected (as PySpark's ToDatetime of the same Timestamps)
> 1-01-01 00:00:00: first
> 9999-12-31 23:59:59: last
> Also investigated, and the same in both clients (pyspark-client 4.3.0 against
> the drop-in, against
> a Spark 4.3.0 server, in the session time zones UTC, America/Los_Angeles and
> Asia/Kolkata and the
> host time zones UTC, America/Los_Angeles and America/New_York): times before
> 1970 with
> nanoseconds through createDataFrame (datetimes, pandas Timestamps, NumPy
> datetime64[ns], pandas
> datetime64[ns] and object columns, pyarrow timestamp[ns] Tables),
> TimestampType and
> TimestampNTZType, through collect, toLocalIterator, toPandas and toArrow,
> literals, and the
> Timestamp of the Rust protobuf runtime. PySpark rounds them in two
> directions, which the drop-in
> follows: local rows (conversion.py:784, PyArrow reading the fields of the
> datetime), literals,
> object columns and pyarrow Tables (pandas/types.py:554, pc.floor_temporal)
> round down to the
> microsecond, and a datetime64[ns] column of a pandas DataFrame rounds toward
> 1970 when the safe
> cast is off (pandas/conversion.py:117, PyArrow's cast).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]