BryanCutler commented on pull request #6316: URL: https://github.com/apache/arrow/pull/6316#issuecomment-649103303
I mean the current process for integration tests with the master branch is to build Spark with Arrow Java master, then run Java and Python tests. That process is good for Spark master and branch-3.0, but with Spark branch-2.4, it doesn't make too much sense to rebuild Spark with Arrow Java master. This is because the dependency on Arrow Java is pretty much frozen for branch-2.4 (unless there is a serious enough bug), but different versions of pyarrow can be used. It would be best to take the latest Spark 2.4.6 release out of the box and run the pyspark tests from the script with pyarrow from master. We are not really concerned about Java compatibility since it can't be upgraded anyway, but that still allows us to flush out any Python problems. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
