zhengruifeng commented on code in PR #52980:
URL: https://github.com/apache/spark/pull/52980#discussion_r2522169488
##########
python/pyspark/sql/types.py:
##########
@@ -441,6 +441,12 @@ def __repr__(self) -> str:
class TimestampType(DatetimeType, metaclass=DataTypeSingleton):
"""Timestamp (datetime.datetime) data type."""
+ # We need to cache the timezone info for datetime.datetime.fromtimestamp
+ # otherwise the forked process will be extremely slow to convert the
timestamp.
+ # This is probably a glibc issue - the forked process will have a bad
cache/lock
+ # status for the timezone info.
+ tz_info = None
Review Comment:
is it possible to make timezone an optional field in `TimestampType`?
if it is not set, then respect the config "spark.sql.session.timeZone"
@cloud-fan @HyukjinKwon
e.g. in pyarrow,
```py
In [29]: pa.timestamp('us', tz='America/Los_Angeles')
Out[29]: TimestampType(timestamp[us, tz=America/Los_Angeles])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
