shaneknapp commented on issue #24379: [SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x URL: https://github.com/apache/spark/pull/24379#issuecomment-483854051 > I think the 2.3/2.4 branches will have to stay with pyarrow 0.8.0 and pandas 0.19.2 for the tests to pass. There were a few fixes that were done in master to make later versions of pyarrow work. These could be applied to the branches, but would have to be done manually. The risk is that we could end up breaking support for pyarrow 0.8.0, which would be pretty bad. > mostly because pyarrow > 0.8.0 requires python3.5+ (i think), this will make for 3 python envs that we test against... which i'm not opposed to. master: py36, pyarrow 0.12.1, pandas 0.24.2 2.3/2.2: py34, pyarrow 0.8.0, pandas 0.24.2 > @shaneknapp do you think you can try pyspark.sql.tests + py36 + pandas 0.19.2 + pyarrow 0.8.0 for the 2.4 branch? I think that should work. Sorry, I should have explained that better before.. if i can get py36 + pyarrow 0.8.0 to happily work together, sure. give me a couple of hours to make the env/run tests.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
