HyukjinKwon commented on issue #18139: [SPARK-20787][PYTHON] PySpark can't handle datetimes before 1900 URL: https://github.com/apache/spark/pull/18139#issuecomment-454245664 Yea, just double checked `mktime`. datetimes issue in Python 2 looks mostly resolved but this particular API call looks problematic and platform-dependent. re: https://github.com/apache/spark/pull/18139#issuecomment-454141515 and https://github.com/apache/spark/pull/18139#issuecomment-454155032 I agree with using other libraries can solve this problem but adding a Python library dependency in PySpark is possible but somewhat troublesome. Was wondering if we can avoid to add an extra dependency. That's why we're trying to take a look here and there to refer the codes and port minimum changes into PySpark side.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
