Hyukjin Kwon created SPARK-62058:
------------------------------------

             Summary: Log the events of a pipeline run at the exact times of 
their timestamps, as PySpark's Timestamp.ToDatetime() gives them
                 Key: SPARK-62058
                 URL: https://issues.apache.org/jira/browse/SPARK-62058
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
PySpark 4.3.0's handle_pipeline_events logs each event of a pipeline run at
event.timestamp.ToDatetime().replace(tzinfo=timezone.utc)
(python/pyspark/pipelines/spark_connect_pipeline.py:49-65, line 64). protobuf's
Timestamp.ToDatetime is exact: the epoch plus timedelta(seconds=seconds,
microseconds=nanos // 1000) 
(google/protobuf/internal/well_known_types.py:239-266), and
log_with_provided_timestamp prints the time to the second 
(pyspark/pipelines/logging_utils.py:33-42).

The drop-in's run stream yields the microseconds of the event's Timestamp, and 
its
handle_pipeline_events (python/pyspark/pipelines/spark_connect_pipeline.py) 
converted them with
datetime.fromtimestamp(timestamp_micros / 1_000_000, tz=timezone.utc): a float 
of the seconds,
whose spacing exceeds a microsecond for a time far from 1970. So an event whose 
time is far from
1970 can be logged a second late, and an event at the last microsecond of 9999 
fails the run's
log. Events of the years near 1970, before it too, convert the same.

Repro (master 80046f2f; the stream's values as pipeline_start_run gives them)
  from pyspark.pipelines.spark_connect_pipeline import handle_pipeline_events
  handle_pipeline_events(iter([
      ("first", -62135596800 * 1_000_000 + 999_999),   # 0001-01-01 
00:00:00.999999
      ("last", 253402300799 * 1_000_000 + 999_999),    # 9999-12-31 
23:59:59.999999
  ]))

Actual
  1-01-01 00:00:01: first
  ValueError: year 10000 is out of range

Expected (as PySpark's ToDatetime of the same Timestamps)
  1-01-01 00:00:00: first
  9999-12-31 23:59:59: last

Also investigated, and the same in both clients (pyspark-client 4.3.0 against 
the drop-in, against
a Spark 4.3.0 server, in the session time zones UTC, America/Los_Angeles and 
Asia/Kolkata and the
host time zones UTC, America/Los_Angeles and America/New_York): times before 
1970 with
nanoseconds through createDataFrame (datetimes, pandas Timestamps, NumPy 
datetime64[ns], pandas
datetime64[ns] and object columns, pyarrow timestamp[ns] Tables), TimestampType 
and
TimestampNTZType, through collect, toLocalIterator, toPandas and toArrow, 
literals, and the
Timestamp of the Rust protobuf runtime. PySpark rounds them in two directions, 
which the drop-in
follows: local rows (conversion.py:784, PyArrow reading the fields of the 
datetime), literals,
object columns and pyarrow Tables (pandas/types.py:554, pc.floor_temporal) 
round down to the
microsecond, and a datetime64[ns] column of a pandas DataFrame rounds toward 
1970 when the safe
cast is off (pandas/conversion.py:117, PyArrow's cast).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to