2.4 PR: https://github.com/apache/spark/pull/24379 2.3 PR: https://github.com/apache/spark/pull/24380
both of these branches failed pretty spectacularly during my non-PR testing in the pyspark sql tests, but let's see how they fare when things are run automagically by jenkins. shane On Mon, Apr 15, 2019 at 11:49 AM shane knapp <skn...@berkeley.edu> wrote: > 1) i absolutely do not want to test against more than two python > versions. consider my foot to have been put down on that. :) > > 2) i'll start testing against 2.3 and 2.4 now (last week was a bit crazy, > so i didn't get around to it). > > once i'm happy w/the 2.3 and 2.4 results, i'll follow up here and get the > upgrade schedule posted. > > On Wed, Apr 10, 2019 at 3:57 PM Sean Owen <sro...@gmail.com> wrote: > >> My guess is that branch 2.3, 2.4 still work with Python 3.6 even if >> we're not testing that. If we're only testing against 3.6, I guess we >> wouldn't detect if something broke for 3.4 which is still 'supported' >> in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway. >> This could be fine if we accept that small risk and it keeps the >> testing env from getting crazy; I don't actually want to test vs N >> Python versions anyway. >> >> On Wed, Apr 10, 2019 at 4:49 PM shane knapp <skn...@berkeley.edu> wrote: >> > >> > ok, i'll create a PR for each branch so we can test things that way, >> rather than backporting. >> > >> > the bigger question i have is how far back, branch-wise, are we willing >> to support w/regards to tests? >> > >> > On Wed, Apr 10, 2019 at 12:16 PM Sean Owen <sro...@gmail.com> wrote: >> >> >> >> In theory Spark 2.4 supports Python 3.4; would this mean it's now just >> >> tested vs 3.6? that's not out of the question, but can the older >> >> branches continue to test on older versions or is that super complex? >> >> >> >> On Wed, Apr 10, 2019 at 1:37 PM shane knapp <skn...@berkeley.edu> >> wrote: >> >> > >> >> > details here (see most recent comments for current state of things): >> >> > https://issues.apache.org/jira/browse/SPARK-25079 >> >> > >> >> > my PR for these changes: >> >> > https://github.com/apache/spark/pull/24266 >> >> > >> >> > we're doing this because we need to support arrow 0.12.1: >> >> > https://issues.apache.org/jira/browse/SPARK-27276 >> >> > >> >> > a couple of things: >> >> > >> >> > * i won't be switching things over until april 11th or 12th at the >> earliest. >> >> > * this change will require a short (~15-20min) downtime to switch >> over from 3.4 to 3.6. >> >> > * updating python will impact all active branches, so i will need >> to backport this pr: https://github.com/apache/spark/pull/24266 >> >> > >> >> > question: which other branches should i be applying this change >> to? definitely master and 2.4... also 2.3? 2.2? >> >> > >> >> > thanks in advance, >> >> > >> >> > shane >> >> > -- >> >> > Shane Knapp >> >> > UC Berkeley EECS Research / RISELab Staff Technical Lead >> >> > https://rise.cs.berkeley.edu >> > >> > >> > >> > -- >> > Shane Knapp >> > UC Berkeley EECS Research / RISELab Staff Technical Lead >> > https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
