Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19674
@ueshin, now I am getting what you meant by DST. I think you roughly knew
this problem but let me describe it more given my debugging.
Looks the problem is in `mktime` and sounds `mktime` is platform-dependent
(roughly assuming from codes and docs).
I made a minimised reproducer:
```python
import time
import os
from datetime import datetime
os.environ["TZ"] = "America/Los_Angeles"
time.tzset()
time.mktime(datetime(2100, 4, 4, 4, 4, 4).timetuple())
```
My local, it prints:
```
4110523444.0
```
On Unbuntu 14.04:
```
4110519844.0
```
Jenkins, it prints:
```
4110519844.0
```
I am not sure if this is easily fixable within Spark as it looks dependent
on Python implementation and/or the underlying C library, up to my knowledge
and from my reading some docs.
Could you maybe avoid this time within DST for now in your PR? I currently
don't have a good idea to fix this with a simple and surgical fix
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
