Yikun commented on code in PR #36095: URL: https://github.com/apache/spark/pull/36095#discussion_r844770142
########## python/docs/source/index.rst: ########## @@ -47,6 +47,8 @@ With this package, you can: * Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets). * Switch to pandas API and PySpark API contexts easily without any overhead. +Note that pandas has different behavior per its version, and pandas API on Spark tries to match the behavior of pandas 1.4. Review Comment: https://github.com/Yikun/spark/pull/86 FYI, there are some test failed with pandas 1.4.x, I haven't got enough time to find all errors reason yet, some of thems are some panda bugs in 1.4.x (such as https://github.com/pandas-dev/pandas/issues/46589). we only tested 1.3.x with github action ci. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
