I like Mark’s concept for deprecating Py2 starting with 2.4: It may seem like a ways off but even now there may be some spark versions supporting Py2 past the point where Py2 is no longer receiving security patches
On Sun, Sep 16, 2018 at 12:26 PM Mark Hamstra <m...@clearstorydata.com> wrote: > We could also deprecate Py2 already in the 2.4.0 release. > > On Sat, Sep 15, 2018 at 11:46 AM Erik Erlandson <eerla...@redhat.com> > wrote: > >> In case this didn't make it onto this thread: >> >> There is a 3rd option, which is to deprecate Py2 for Spark-3.0, and >> remove it entirely on a later 3.x release. >> >> On Sat, Sep 15, 2018 at 11:09 AM, Erik Erlandson <eerla...@redhat.com> >> wrote: >> >>> On a separate dev@spark thread, I raised a question of whether or not >>> to support python 2 in Apache Spark, going forward into Spark 3.0. >>> >>> Python-2 is going EOL <https://github.com/python/devguide/pull/344> at >>> the end of 2019. The upcoming release of Spark 3.0 is an opportunity to >>> make breaking changes to Spark's APIs, and so it is a good time to consider >>> support for Python-2 on PySpark. >>> >>> Key advantages to dropping Python 2 are: >>> >>> - Support for PySpark becomes significantly easier. >>> - Avoid having to support Python 2 until Spark 4.0, which is likely >>> to imply supporting Python 2 for some time after it goes EOL. >>> >>> (Note that supporting python 2 after EOL means, among other things, that >>> PySpark would be supporting a version of python that was no longer >>> receiving security patches) >>> >>> The main disadvantage is that PySpark users who have legacy python-2 >>> code would have to migrate their code to python 3 to take advantage of >>> Spark 3.0 >>> >>> This decision obviously has large implications for the Apache Spark >>> community and we want to solicit community feedback. >>> >>> >>
