This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 36fbc71d6e8b [SPARK-55297][PYTHON][PS] Restore timedelta dtype based
on the original dtype
36fbc71d6e8b is described below
commit 36fbc71d6e8bf3aefc4560f5e35e95517334020f
Author: Tian Gao <[email protected]>
AuthorDate: Mon Feb 2 13:57:26 2026 +0800
[SPARK-55297][PYTHON][PS] Restore timedelta dtype based on the original
dtype
### What changes were proposed in this pull request?
Add a restore method for timedelta. This is a similar change as
https://github.com/apache/spark/pull/54017
### Why are the changes needed?
Same as datetime - pandas 3 now has smart unit for timedelta and we should
try to restore it.
### Does this PR introduce _any_ user-facing change?
Just more like pandas 3 when pandas 3 is used.
### How was this patch tested?
Locally a test passed with the fix.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54079 from gaogaotiantian/pandas3-timedelta-restore.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/data_type_ops/timedelta_ops.py | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/python/pyspark/pandas/data_type_ops/timedelta_ops.py
b/python/pyspark/pandas/data_type_ops/timedelta_ops.py
index 7a9da8511e62..aa3e8f64801f 100644
--- a/python/pyspark/pandas/data_type_ops/timedelta_ops.py
+++ b/python/pyspark/pandas/data_type_ops/timedelta_ops.py
@@ -21,6 +21,7 @@ from typing import Any, Union
import pandas as pd
from pandas.api.types import CategoricalDtype
+from pyspark.loose_version import LooseVersion
from pyspark.sql.types import (
BooleanType,
DayTimeIntervalType,
@@ -64,6 +65,13 @@ class TimedeltaOps(DataTypeOps):
"""Prepare column when from_pandas."""
return col
+ def restore(self, col: pd.Series) -> pd.Series:
+ """Restore column when to_pandas."""
+ if LooseVersion(pd.__version__) < "3.0.0":
+ return col
+ else:
+ return col.astype(self.dtype)
+
def sub(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
