HyukjinKwon edited a comment on pull request #30098: URL: https://github.com/apache/spark/pull/30098#issuecomment-712627234
There are few things to note: - This is a temporary fix. Once PySpark supports PyArrow 2.0.0+ (SPARK-33189), we can remove this change. - We should port this back into other branches in case PyArrow 2.0.0+ support is not ported back, and in order to make the builds pass. - Python 3.8 build will pass because the packages are pre-installed in the docker image (see SPARK-33162), and PyPy3 does not have PyArrow. It fails with Python 3.6 because it newly installs the latest PyArrow 2.0.0. - I didn't update documentation and `setup.py` yet. This PR currently aims to make the build pass first. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
