[
https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shane knapp updated SPARK-25079:
--------------------------------
Description:
for the impending arrow upgrade
(https://issues.apache.org/jira/browse/SPARK-23874) we need to bump python 3.4
-> 3.6.
i have been testing this here:
[https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/|https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69]
my methodology:
1) upgrade python + arrow to 3.5 and 0.10.0
2) run python tests
3) when i'm happy that Things Won't Explode Spectacularly, pause jenkins and
upgrade centos workers to python3.5
4) simultaneously do the following:
- create a symlink in /home/anaconda/envs/py3k/bin for python3.4 that points
to python3.5 (this is currently being tested here:
[https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69)]
- push a change to python/run-tests.py replacing 3.4 with 3.5
5) once the python3.5 change to run-tests.py is merged, we will need to
back-port this to all existing branches
6) then and only then can i remove the python3.4 -> python3.5 symlink
was:
for the impending arrow upgrade
(https://issues.apache.org/jira/browse/SPARK-23874) we need to bump python 3.4
-> 3.5.
i have been testing this here:
[https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/|https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69]
my methodology:
1) upgrade python + arrow to 3.5 and 0.10.0
2) run python tests
3) when i'm happy that Things Won't Explode Spectacularly, pause jenkins and
upgrade centos workers to python3.5
4) simultaneously do the following:
- create a symlink in /home/anaconda/envs/py3k/bin for python3.4 that points
to python3.5 (this is currently being tested here:
[https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69)]
- push a change to python/run-tests.py replacing 3.4 with 3.5
5) once the python3.5 change to run-tests.py is merged, we will need to
back-port this to all existing branches
6) then and only then can i remove the python3.4 -> python3.5 symlink
> [PYTHON] upgrade python 3.4 -> 3.6
> ----------------------------------
>
> Key: SPARK-25079
> URL: https://issues.apache.org/jira/browse/SPARK-25079
> Project: Spark
> Issue Type: Improvement
> Components: Build, PySpark
> Affects Versions: 2.3.1
> Reporter: shane knapp
> Assignee: shane knapp
> Priority: Major
> Fix For: 3.0.0
>
>
> for the impending arrow upgrade
> (https://issues.apache.org/jira/browse/SPARK-23874) we need to bump python
> 3.4 -> 3.6.
> i have been testing this here:
> [https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/|https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69]
> my methodology:
> 1) upgrade python + arrow to 3.5 and 0.10.0
> 2) run python tests
> 3) when i'm happy that Things Won't Explode Spectacularly, pause jenkins and
> upgrade centos workers to python3.5
> 4) simultaneously do the following:
> - create a symlink in /home/anaconda/envs/py3k/bin for python3.4 that
> points to python3.5 (this is currently being tested here:
> [https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69)]
> - push a change to python/run-tests.py replacing 3.4 with 3.5
> 5) once the python3.5 change to run-tests.py is merged, we will need to
> back-port this to all existing branches
> 6) then and only then can i remove the python3.4 -> python3.5 symlink
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
