zhengruifeng commented on code in PR #48159:
URL: https://github.com/apache/spark/pull/48159#discussion_r1766143089
##########
python/pyspark/sql/tests/test_column.py:
##########
@@ -289,6 +290,26 @@ def test_lit_time_representation(self):
ts = datetime.datetime(2021, 3, 4, 12, 34, 56, 1234)
self.assertEqual(str(sf.lit(ts)), "Column<'2021-03-04
12:34:56.001234'>")
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_lit_delta_representation(self):
+ for delta in [
+ datetime.timedelta(days=1),
+ datetime.timedelta(hours=2),
+ datetime.timedelta(minutes=3),
+ datetime.timedelta(seconds=4),
+ datetime.timedelta(microseconds=5),
+ datetime.timedelta(days=2, hours=21, microseconds=908),
+ datetime.timedelta(days=1, minutes=-3, microseconds=-1001),
+ datetime.timedelta(days=1, hours=2, minutes=3, seconds=4,
microseconds=5),
+ ]:
+ import pandas as pd
+
+ # Column<'PT69H0.000908S'> or Column<'P2DT21H0M0.000908S'>
+ s = str(sf.lit(delta))
+
+ # Parse the ISO string representation and compare
+ self.assertTrue(pd.Timedelta(s[8:-2]).to_pytimedelta() == delta)
Review Comment:
it also works for pyspark classic.
Classic also use a ISO-8601 string, but JVM side and Pandas apply different
units.
A string representation from the JVM side can also be parsed by Pandas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
