This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4087c770bdfd [SPARK-57741][PYTHON] Add timestamp_nanos to PySpark
public API
4087c770bdfd is described below
commit 4087c770bdfd56cd0ed784a9184d4feb14377cc2
Author: Jubin Soni <[email protected]>
AuthorDate: Wed Jul 1 11:44:54 2026 +0200
[SPARK-57741][PYTHON] Add timestamp_nanos to PySpark public API
### What is the purpose of the change
Fixes SPARK-57741 (follow-up to SPARK-57526) — adds `timestamp_nanos` to the
PySpark API (`pyspark.sql.functions` and PySpark Connect), completing the
nanosecond round-trip pair in Python.
`timestamp_nanos(e)` converts a nanoseconds-since-epoch integer to a
`TIMESTAMP_LTZ(9)` value. It is the inverse of `unix_nanos` (SPARK-57579).
The SQL function and Scala API were added in SPARK-57526, but Python support
was explicitly deferred and tracked as a follow-up via
`expected_missing_in_py`:
```python
expected_missing_in_py = {
"timestamp_nanos"
} # SPARK-57526: PySpark support tracked as a follow-up
```
### The round-trip pair is now complete
| Function | Before this PR | After this PR |
|------------------|----------------|---------------|
| `unix_nanos` | present | present |
| `timestamp_nanos` | missing | added |
---
### Brief change log
- `python/pyspark/sql/functions/builtin.py`
Added `timestamp_nanos(col)` after `timestamp_micros`, decorated with
`_try_remote_functions`, with full docstring:
- `versionadded:: 4.3.0`
- parameters + return type
- See Also links
- two doctests (valid nanosecond value + NULL input)
- `python/pyspark/sql/connect/functions/builtin.py`
Added Connect-side wrapper for `timestamp_nanos`, inheriting docstring
from main module and following the same pattern as `timestamp_micros`
- `python/pyspark/sql/functions/__init__.py`
Exported `timestamp_nanos` in alphabetical order between
`timestamp_millis` and `timestamp_seconds`
- `python/docs/source/reference/pyspark.sql/functions.rst`
Added `timestamp_nanos` entry between `timestamp_millis` and
`timestamp_seconds`
- `python/pyspark/sql/tests/test_functions.py`
Removed `"timestamp_nanos"` from `expected_missing_in_py` (set is now
empty)
---
### Verifying this change
Covered by the existing parity test in `FunctionsTestsMixin`:
- `test_function_parity` previously allowlisted `timestamp_nanos` as an
expected gap
- Removing it from `expected_missing_in_py` ensures the test will now fail
if
`timestamp_nanos` is ever missing from the Python API again
The two doctests in the `timestamp_nanos` docstring verify:
- A nanosecond integer input returns the correct `TIMESTAMP_LTZ(9)` value
- A NULL input returns NULL
---
### Does this pull request potentially affect one of the following parts
- **Dependencies (adds or upgrades dependency):** No
- **Public API (`Public` / `Evolving`):** Yes — new public PySpark function
- **Serializers:** No
- **Runtime per-record code paths (performance sensitive):** No — Python
wrapper only; JVM expression unchanged
- **Deployment or recovery:** No
- **S3 file system connector:** No
---
### Documentation
- Introduces a new feature: **Yes**
- New API: `pyspark.sql.functions.timestamp_nanos`
- Documented via:
- Inline docstring (parameters, return type, See Also links)
- Doctests in `builtin.py`
---
### Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code was used as a pair-programming assistant.
All code was written, understood, and verified by the author.
Generated-by: Claude Opus 4.8
Closes #56852 from jubins/j-SPARK-57741-add-timestamp-nanos.
Authored-by: Jubin Soni <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 3a808e5274a1c82356fa752f8fecd794db4fb27b)
Signed-off-by: Max Gekk <[email protected]>
---
.../source/reference/pyspark.sql/functions.rst | 1 +
python/pyspark/sql/connect/functions/builtin.py | 7 +++
python/pyspark/sql/functions/__init__.py | 1 +
python/pyspark/sql/functions/builtin.py | 51 ++++++++++++++++++++++
python/pyspark/sql/tests/test_functions.py | 4 +-
5 files changed, 61 insertions(+), 3 deletions(-)
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst
b/python/docs/source/reference/pyspark.sql/functions.rst
index 3ad3ae9cdf12..45e0eaff0296 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -299,6 +299,7 @@ Date and Timestamp Functions
timestamp_diff
timestamp_micros
timestamp_millis
+ timestamp_nanos
timestamp_seconds
time_bucket
time_diff
diff --git a/python/pyspark/sql/connect/functions/builtin.py
b/python/pyspark/sql/connect/functions/builtin.py
index 71433899074f..7a4d2b0e1b6c 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -3788,6 +3788,13 @@ def timestamp_micros(col: "ColumnOrName") -> Column:
timestamp_micros.__doc__ = pysparkfuncs.timestamp_micros.__doc__
+def timestamp_nanos(col: "ColumnOrName") -> Column:
+ return _invoke_function_over_columns("timestamp_nanos", col)
+
+
+timestamp_nanos.__doc__ = pysparkfuncs.timestamp_nanos.__doc__
+
+
def timestamp_diff(unit: str, start: "ColumnOrName", end: "ColumnOrName") ->
Column:
return _invoke_function_over_columns("timestampdiff", lit(unit), start,
end)
diff --git a/python/pyspark/sql/functions/__init__.py
b/python/pyspark/sql/functions/__init__.py
index 914b9c7fbcb7..5083d5c0db16 100644
--- a/python/pyspark/sql/functions/__init__.py
+++ b/python/pyspark/sql/functions/__init__.py
@@ -248,6 +248,7 @@ __all__ = [ # noqa: F405
"timestamp_diff",
"timestamp_micros",
"timestamp_millis",
+ "timestamp_nanos",
"timestamp_seconds",
"time_bucket",
"time_diff",
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index e1ee8cf0b9ef..d10fed1efa3f 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -13183,6 +13183,57 @@ def timestamp_micros(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("timestamp_micros", col)
+@_try_remote_functions
+def timestamp_nanos(col: "ColumnOrName") -> Column:
+ """
+ Creates a nanosecond-precision timestamp (``TIMESTAMP_LTZ(9)``) from the
number of
+ nanoseconds since the UTC epoch.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or column name
+ a column of ``BIGINT`` or ``DECIMAL`` nanosecond values since the UTC
epoch.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a ``TIMESTAMP_LTZ(9)`` column representing the corresponding point in
time.
+
+ See Also
+ --------
+ :meth:`pyspark.sql.functions.timestamp_seconds`
+ :meth:`pyspark.sql.functions.timestamp_millis`
+ :meth:`pyspark.sql.functions.timestamp_micros`
+ :meth:`pyspark.sql.functions.unix_nanos`
+
+ Examples
+ --------
+ >>> import pyspark.sql.functions as sf
+ >>> spark.conf.set("spark.sql.session.timeZone", "UTC")
+ >>> spark.conf.set("spark.sql.timestampNanosTypes.enabled", "true")
+ >>> df = spark.createDataFrame([(1577885075123456789,)], ['nanos'])
+ >>> df.select(sf.timestamp_nanos('nanos')).show(truncate=False)
+ +-----------------------------+
+ |timestamp_nanos(nanos) |
+ +-----------------------------+
+ |2020-01-01 13:24:35.123456789|
+ +-----------------------------+
+
+ >>> df.select(sf.timestamp_nanos(sf.lit(None).cast('bigint'))).show()
+ +-------------------------------------+
+ |timestamp_nanos(CAST(NULL AS BIGINT))|
+ +-------------------------------------+
+ | NULL|
+ +-------------------------------------+
+
+ >>> spark.conf.unset("spark.sql.timestampNanosTypes.enabled")
+ >>> spark.conf.unset("spark.sql.session.timeZone")
+ """
+ return _invoke_function_over_columns("timestamp_nanos", col)
+
+
@_try_remote_functions
def timestamp_diff(unit: str, start: "ColumnOrName", end: "ColumnOrName") ->
Column:
"""
diff --git a/python/pyspark/sql/tests/test_functions.py
b/python/pyspark/sql/tests/test_functions.py
index 16928193db6d..8599d0dd46e1 100644
--- a/python/pyspark/sql/tests/test_functions.py
+++ b/python/pyspark/sql/tests/test_functions.py
@@ -82,9 +82,7 @@ class FunctionsTestsMixin:
missing_in_py = jvm_fn_set.difference(py_fn_set)
# Functions that we expect to be missing in python until they are
added to pyspark
- expected_missing_in_py = {
- "timestamp_nanos"
- } # SPARK-57526: PySpark support tracked as a follow-up
+ expected_missing_in_py = set()
self.assertEqual(
expected_missing_in_py, missing_in_py, "Missing functions in
pyspark not as expected"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
