This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 47cfc1c8c00c [SPARK-57740][PYTHON] Fix unix_micros truncation
disclaimer and add unix_nanos cross-references
47cfc1c8c00c is described below
commit 47cfc1c8c00c902882b262be9bfcf4344e5a0a6d
Author: Jubin Soni <[email protected]>
AuthorDate: Mon Jun 29 12:08:26 2026 +0200
[SPARK-57740][PYTHON] Fix unix_micros truncation disclaimer and add
unix_nanos cross-references
### What is the purpose of the change
Fixes SPARK-57740 — two related doc-correctness issues in the `unix_*`
epoch-unit function family in `python/pyspark/sql/functions/builtin.py`:
1. **Missing truncation disclaimer in `unix_micros`.** `unix_millis` and
`unix_seconds` both include `"Truncates higher levels of precision."` in their
one-line description. `unix_micros` did not, despite silently dropping
sub-microsecond digits when given a nanosecond-precision input
(`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in `[7, 9]`). This became a
practical concern once `unix_nanos` (SPARK-57579) made nanosecond-precision
timestamps usable in PySpark.
2. **Missing `unix_nanos` cross-references.** `unix_nanos` (SPARK-57579)
correctly links to all three siblings in its `See Also` section. None of the
three sibling functions were updated to link back to `unix_nanos`, breaking
discoverability of the function family from the existing functions.
### Brief change log
- `python/pyspark/sql/functions/builtin.py`:
- `unix_micros`: added `"Truncates higher levels of precision."` to the
one-line description
- `unix_micros`, `unix_millis`, `unix_seconds`: added
`:meth:\`pyspark.sql.functions.unix_nanos\`` to each `See Also` section
### Verifying this change
These are documentation-only changes with no logic impact. Verified by
inspection that:
- All four `unix_*` functions now consistently include a truncation
disclaimer in their description
- All four `unix_*` functions now cross-reference each other in their `See
Also` sections symmetrically
### Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`Public`/`Evolving`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
### Documentation
Does this pull request introduce a new feature? No — documentation fix only.
### Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code was used as a pair-programming assistant. All code
was written, understood, and verified by the author.
Generated-by: Claude Opus 4.8
Closes #56851 from jubins/j-SPARK-57740-added-cross-reference-to-unix-nanos.
Authored-by: Jubin Soni <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
python/pyspark/sql/functions/builtin.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index bca4704962f9..de6683eabef5 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -11710,6 +11710,7 @@ def unix_date(col: "ColumnOrName") -> Column:
@_try_remote_functions
def unix_micros(col: "ColumnOrName") -> Column:
"""Returns the number of microseconds since 1970-01-01 00:00:00 UTC.
+ Truncates higher levels of precision.
.. versionadded:: 3.5.0
@@ -11728,6 +11729,7 @@ def unix_micros(col: "ColumnOrName") -> Column:
:meth:`pyspark.sql.functions.unix_date`
:meth:`pyspark.sql.functions.unix_seconds`
:meth:`pyspark.sql.functions.unix_millis`
+ :meth:`pyspark.sql.functions.unix_nanos`
:meth:`pyspark.sql.functions.timestamp_micros`
Examples
@@ -11771,6 +11773,7 @@ def unix_millis(col: "ColumnOrName") -> Column:
:meth:`pyspark.sql.functions.unix_date`
:meth:`pyspark.sql.functions.unix_seconds`
:meth:`pyspark.sql.functions.unix_micros`
+ :meth:`pyspark.sql.functions.unix_nanos`
:meth:`pyspark.sql.functions.timestamp_millis`
Examples
@@ -11865,6 +11868,7 @@ def unix_seconds(col: "ColumnOrName") -> Column:
:meth:`pyspark.sql.functions.unix_date`
:meth:`pyspark.sql.functions.unix_millis`
:meth:`pyspark.sql.functions.unix_micros`
+ :meth:`pyspark.sql.functions.unix_nanos`
:meth:`pyspark.sql.functions.from_unixtime`
:meth:`pyspark.sql.functions.timestamp_seconds`
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
