shaneknapp commented on issue #24379: [SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x URL: https://github.com/apache/spark/pull/24379#issuecomment-483857311 > I think the 2.3/2.4 branches will have to stay with pyarrow 0.8.0 and pandas 0.19.2 for the tests to pass. There were a few fixes that were done in master to make later versions of pyarrow work. These could be applied to the branches, but would have to be done manually. The risk is that we could end up breaking support for pyarrow 0.8.0, which would be pretty bad. > > @shaneknapp do you think you can try pyspark.sql.tests + py36 + pandas 0.19.2 + pyarrow 0.8.0 for the 2.4 branch? I think that should work. Sorry, I should have explained that better before.. also, this means we'll need 3 python envs to test everything. py27: master, 2.3, 2.4 py36 + pandas 0.19.2 + pyarrow 0.8.0: 2.3, 2.4 py36 + pandas 0.24.2 (or 0.19.2) + pyarrow 0.12.1: master i'm not completely opposed, as this will most likely be easier than backporting and fixing the bugs in 2.3 and 2.4...
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
