HyukjinKwon edited a comment on issue #18139: [SPARK-20787][PYTHON] PySpark can't handle datetimes before 1900 URL: https://github.com/apache/spark/pull/18139#issuecomment-454245664 Yea, just double checked `mktime`. datetimes issue in Python 2 looks mostly resolved in Python 3 but this particular API call looks problematic and platform-dependent. re: https://github.com/apache/spark/pull/18139#issuecomment-454141515 and https://github.com/apache/spark/pull/18139#issuecomment-454155032 I agree with using other libraries can solve this problem but adding a Python library dependency in PySpark is possible but somewhat troublesome (due the the nature of distributed computing - like it should be guaranteed to ship that library across all different deployment mode). Was wondering if we can avoid to add an extra dependency. That's why we're trying to take a look here and there to refer the codes and port minimum changes into PySpark side.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
