This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8e02a6493ef5 [SPARK-48695][PYTHON] `TimestampNTZType.fromInternal` not
use the deprecated methods
8e02a6493ef5 is described below
commit 8e02a6493ef5dc5949e161179a7c081c5ca58ff2
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Jun 24 20:15:26 2024 +0800
[SPARK-48695][PYTHON] `TimestampNTZType.fromInternal` not use the
deprecated methods
### What changes were proposed in this pull request?
`TimestampNTZType.fromInternal` not use the deprecated methods
### Why are the changes needed?
```
In [2]: ts = 111111111111
In [3]: datetime.datetime.utcfromtimestamp(ts // 1000000).replace(
...: microsecond=ts % 1000000
...: )
<ipython-input-3-166450e5ec26>:1: DeprecationWarning:
datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in
a future version. Use timezone-aware objects to represent datetimes in UTC:
datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
datetime.datetime.utcfromtimestamp(ts // 1000000).replace(
Out[3]: datetime.datetime(1970, 1, 2, 6, 51, 51, 111111)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
new tests
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #47068 from zhengruifeng/fix_ntz_conversion.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/sql/tests/test_serde.py | 8 ++++++++
python/pyspark/sql/types.py | 4 ++--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/sql/tests/test_serde.py
b/python/pyspark/sql/tests/test_serde.py
index ef8bbd2c370f..01cf3c51d7de 100644
--- a/python/pyspark/sql/tests/test_serde.py
+++ b/python/pyspark/sql/tests/test_serde.py
@@ -95,6 +95,14 @@ class SerdeTestsMixin:
self.assertEqual(now, now1)
self.assertEqual(now, utcnow1)
+ def test_ntz_from_internal(self):
+ for ts in [1, 22, 333, 44444444, 5555555555]:
+ t1 = datetime.datetime.utcfromtimestamp(ts //
1000000).replace(microsecond=ts % 1000000)
+ t2 = datetime.datetime.fromtimestamp(ts // 1000000,
datetime.timezone.utc).replace(
+ microsecond=ts % 1000000, tzinfo=None
+ )
+ self.assertEqual(t1, t2)
+
# regression test for SPARK-19561
def test_datetime_at_epoch(self):
epoch = datetime.datetime.fromtimestamp(0)
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 69074a17ca6c..d2adc53a3618 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -434,8 +434,8 @@ class TimestampNTZType(AtomicType,
metaclass=DataTypeSingleton):
def fromInternal(self, ts: int) -> datetime.datetime:
if ts is not None:
# using int to avoid precision loss in float
- return datetime.datetime.utcfromtimestamp(ts // 1000000).replace(
- microsecond=ts % 1000000
+ return datetime.datetime.fromtimestamp(ts // 1000000,
datetime.timezone.utc).replace(
+ microsecond=ts % 1000000, tzinfo=None
)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
