shaneknapp commented on issue #24379: [SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x URL: https://github.com/apache/spark/pull/24379#issuecomment-483764704 > Is it possible to keep pandas and pyarrow versions the same as before (0.19.2 and 0.8.0) for envs of branches 2.3/2.4 or do they need to share the same env as master? > they need to share the same env as master (or we change all of the testing framework for all branches to create temporary python envs for each branch.... which isn't actually a horrible idea but a much bigger project). regarding pandas 0.19.2, it seems that pandas 0.24.2 is the minimum according to conda forge? (output below trimmed for readability) ``` $ conda install -c conda-forge pyarrow=0.12.1 <snip> added / updated specs: - pyarrow=0.12.1 <snip> The following NEW packages will be INSTALLED: arrow-cpp conda-forge/linux-64::arrow-cpp-0.12.1-py36h0e61e49_0 mkl_fft conda-forge/linux-64::mkl_fft-1.0.11-py36h14c3975_0 mkl_random conda-forge/linux-64::mkl_random-1.0.2-py36h637b7d7_2 parquet-cpp conda-forge/noarch::parquet-cpp-1.5.1-4 pyarrow conda-forge/linux-64::pyarrow-0.12.1-py36hbbcf98d_0 The following packages will be UPDATED: numpy 1.11.3-py36h7e9f1db_12 --> 1.16.2-py36h7e9f1db_0 numpy-base 1.11.3-py36hde5b4d6_12 --> 1.16.2-py36hde5b4d6_0 pandas 0.19.2-np111py36_1 --> 0.24.2-py36hf484d3e_0 <----- NOOOOO! ``` i'll try and see if i can get pandas to 0.19.2, but it's looking to be kinda difficult. > The failures here have been fixed in master from various PRs, but not backported. It's possible to apply them, but it would take some time and could be a bit risky.. i'll take time/risk vs not having any python test coverage for 2.3 and 2.4... but we'll need a commitment from dev@ to help make this stuff work.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
