Jubin Soni created SPARK-57742:
----------------------------------
Summary: unix_micros docstring is missing the truncation
disclaimer present in unix_millis and unix_seconds
Key: SPARK-57742
URL: https://issues.apache.org/jira/browse/SPARK-57742
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0, 5.0.0
Reporter: Jubin Soni
*What is the issue?*
Within the {{unix_*}} epoch-unit function family, {{unix_millis}} and
{{unix_seconds}} both document that they truncate sub-unit precision.
{{unix_micros}} does not, despite having the same behaviour when given a
nanosecond-precision input.
Current one-line descriptions:
{{unix_seconds → "Returns the number of seconds since 1970-01-01 00:00:00 UTC.
Truncates higher levels of precision."
unix_millis → "Returns the number of milliseconds since 1970-01-01 00:00:00
UTC. Truncates higher levels of precision."
unix_micros → "Returns the number of microseconds since 1970-01-01 00:00:00
UTC." ← missing disclaimer}}
This became a practical problem once {{unix_nanos}} (SPARK-57579) made
nanosecond-precision timestamps usable: a user holding a {{TIMESTAMP_LTZ(9)}}
or {{TIMESTAMP_NTZ(9)}} column who reads the {{unix_micros}} docstring has no
indication that sub-microsecond digits will be silently dropped.
----
*How to reproduce*
{{spark.conf.set("spark.sql.timestampNanosTypes.enabled", "true")
df = spark.sql("SELECT TIMESTAMP_NTZ '2020-01-01 00:00:00.000000999' AS ts")
from pyspark.sql import functions as sf
# unix_micros silently drops the 999 nanoseconds — nothing in the docstring
warns of this
df.select(sf.unix_micros("ts")).show()}}
----
*Actual behavior*
The docstring for {{unix_micros}} contains no truncation warning, yet:
{{+--------------------+
|unix_micros(ts) |
+--------------------+
|1577836800000000 | ← 999 nanoseconds silently dropped
+--------------------+}}
----
*Expected behavior*
The docstring reads:
{{"Returns the number of microseconds since 1970-01-01 00:00:00 UTC. Truncates
higher levels of precision."}}
matching {{unix_millis}} and {{{}unix_seconds{}}}, so users are warned that
sub-microsecond digits are dropped.
----
*Proposed fix*
Add {{"Truncates higher levels of precision."}} to the one-line description of
{{unix_micros}} in {{{}python/pyspark/sql/functions/builtin.py{}}}. One line
change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
