[ 
https://issues.apache.org/jira/browse/SPARK-62058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62058:
------------------------------------

    Assignee: Hyukjin Kwon

> Log the events of a pipeline run at the exact times of their timestamps, as 
> PySpark's Timestamp.ToDatetime() gives them
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62058
>                 URL: https://issues.apache.org/jira/browse/SPARK-62058
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> PySpark 4.3.0's handle_pipeline_events logs each event of a pipeline run at
> event.timestamp.ToDatetime().replace(tzinfo=timezone.utc)
> (python/pyspark/pipelines/spark_connect_pipeline.py:49-65, line 64). 
> protobuf's
> Timestamp.ToDatetime is exact: the epoch plus timedelta(seconds=seconds,
> microseconds=nanos // 1000) 
> (google/protobuf/internal/well_known_types.py:239-266), and
> log_with_provided_timestamp prints the time to the second 
> (pyspark/pipelines/logging_utils.py:33-42).
> The drop-in's run stream yields the microseconds of the event's Timestamp, 
> and its
> handle_pipeline_events (python/pyspark/pipelines/spark_connect_pipeline.py) 
> converted them with
> datetime.fromtimestamp(timestamp_micros / 1_000_000, tz=timezone.utc): a 
> float of the seconds,
> whose spacing exceeds a microsecond for a time far from 1970. So an event 
> whose time is far from
> 1970 can be logged a second late, and an event at the last microsecond of 
> 9999 fails the run's
> log. Events of the years near 1970, before it too, convert the same.
> Repro (master 80046f2f; the stream's values as pipeline_start_run gives them)
>   from pyspark.pipelines.spark_connect_pipeline import handle_pipeline_events
>   handle_pipeline_events(iter([
>       ("first", -62135596800 * 1_000_000 + 999_999),   # 0001-01-01 
> 00:00:00.999999
>       ("last", 253402300799 * 1_000_000 + 999_999),    # 9999-12-31 
> 23:59:59.999999
>   ]))
> Actual
>   1-01-01 00:00:01: first
>   ValueError: year 10000 is out of range
> Expected (as PySpark's ToDatetime of the same Timestamps)
>   1-01-01 00:00:00: first
>   9999-12-31 23:59:59: last
> Also investigated, and the same in both clients (pyspark-client 4.3.0 against 
> the drop-in, against
> a Spark 4.3.0 server, in the session time zones UTC, America/Los_Angeles and 
> Asia/Kolkata and the
> host time zones UTC, America/Los_Angeles and America/New_York): times before 
> 1970 with
> nanoseconds through createDataFrame (datetimes, pandas Timestamps, NumPy 
> datetime64[ns], pandas
> datetime64[ns] and object columns, pyarrow timestamp[ns] Tables), 
> TimestampType and
> TimestampNTZType, through collect, toLocalIterator, toPandas and toArrow, 
> literals, and the
> Timestamp of the Rust protobuf runtime. PySpark rounds them in two 
> directions, which the drop-in
> follows: local rows (conversion.py:784, PyArrow reading the fields of the 
> datetime), literals,
> object columns and pyarrow Tables (pandas/types.py:554, pc.floor_temporal) 
> round down to the
> microsecond, and a datetime64[ns] column of a pandas DataFrame rounds toward 
> 1970 when the safe
> cast is off (pandas/conversion.py:117, PyArrow's cast).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to