I'd like to help with this Python 2 -> 3 migration if possible. We're nearly halfway through 2020 now -- is there currently anything stopping us from doing this migration at the moment? Is this the right time to do so?
On 2019/10/05 02:11:24, Valentyn Tymofieiev <valen...@google.com> wrote: > On Fri, Oct 4, 2019 at 11:02 AM Robert Bradshaw <rober...@google.com> wrote: > > > Thanks for holding this vote. Note that this is a pledge to remove > > support sometime in 2020, but no promises as to whether that will be > > January or December (though I hope sooner rather than later) > > > Right. > > > > > Valentyn, did you want to go ahead and make a PR adding Apache Beam to > > the python3statement page? > > > > Yes, I sent > https://github.com/python3statement/python3statement.github.io/pull/265. > > > > > On Mon, Sep 30, 2019 at 5:10 PM Valentyn Tymofieiev <valen...@google.com> > > wrote: > > > > > > As suggested and enthusiastically supported by several folks in this > > thread, I will send a vote to sign a pledge on http://python3statement.org > > on behalf of Apache Beam to discontinue Python 2 support in or before 2020. > > > > > > The motivation for signing the pledge is: > > > - to provide another signal to Beam users, and projects that depend on > > Beam that Beam Python 2 offering will soon sunset; > > > - to facilitate adoption of Python 3 by Beam users, developers, and > > runner maintainers; > > > - to facilitate adoption of Python 3 in wider Python ecosystem. > > > > > > See also http://python3stament.org for background behind this pledge > > and the list of projects which have already signed it. > > > > > > On Mon, Sep 23, 2019 at 4:45 PM Kyle Weaver <kcwea...@google.com> wrote: > > >> > > >> Re feedback collection, we already print a message: > > >> "You are using Apache Beam with Python 2. New releases of Apache Beam > > will soon support Python 3 only." > > >> When users run Python 2 pipelines. This might be a good place to > > provide additional info along with a place to send feedback (probably > > user@). > > While I'm sure not everyone out there reads their logs, I imagine this is a > > sure and easy way of reaching at least some Python 2 users. > > >> > > >> Kyle Weaver | Software Engineer | github.com/ibzib | > > kcwea...@google.com > > >> > > >> > > >> On Fri, Sep 20, 2019 at 10:28 AM Valentyn Tymofieiev < > > valen...@google.com> wrote: > > >>> > > >>> Thank you, Chad, for refreshing this conversation and adding the > > perspective of Python 2 users of Beam who have not(yet) completed the > > migration. My thoughts below. > > >>> > > >>> - It is in the best interest of everyone to ensure a smooth migration > > for Beam users. However a migration needs to happen since Python ecosystem > > is moving off of Python 2. > > >>> - Beam has a couple of dozen dependencies, and we cannot have an > > expectation that Python 2 versions of these dependencies will be maintained > > in 2020. > > >>> - BEAM-1251 should be closed, since it may communicate a signal that > > Beam does not support Python 3, while it does. Beam has first announced > > support of Python 3 in Beam 2.11.0, admittedly later than many mainstream > > libraries in Python ecosystem. > > >>> - I think Python 2 LTS release (if we continue them) may have critical > > bug fixes, but not new features, so we won't be backporting new features. > > >>> - Beam portability allows users to customize usercode runtime > > environment, and it should be possible for users to supply a Python 2 SDK > > harness container, should they have no other option. This would require a > > backported user-supplied version of Beam SDK that works on Python 2, > > although such SDK may become difficult/impractical to maintain for most > > users. > > >>> - There are several open issues related to Python 3, but they are > > improvements in nature, and we are steadily closing them off. I am not > > aware of any adoption blockers for Beam Python 3, specific to Beam. > > >>> - I have not heard of users reports who attempted but were not able to > > use Beam on Python 3. > > >>> - This does not mean that our offering is perfect, there may be errors > > and omissions that are yet to be discovered. However, it would be in the > > best interest of the Beam community to discover these issues earlier. A > > message that Beam will discontinue Python 2 support will encourage users to > > migrate, therefore I also support Beam signing > > https://python3statement.org. > > >>> - Having more usage statistics and feedback closer to 2020 can help us > > be more confident in deciding when to stop Python 2 support. > > >>> > > >>> On Thu, Sep 19, 2019 at 6:05 PM Ahmet Altay <al...@google.com> wrote: > > >>>> > > >>>> Thanks a lot for sharing your thoughts, I completely agree that we > > need to minimize the burden on our users as much as possible. Especially in > > this case when we are offering a robust python 3 solution just now. However > > I do share the same concerns related to dependencies and tool chains, It > > will be increasingly difficult for us to keep our code base compatible with > > python2 and python3 overtime. (To be very explicit, one of those > > dependencies is Dataflow's python pre-portability workers.) > > >>>> > > >>>> On Thu, Sep 19, 2019 at 5:17 PM Maximilian Michels <m...@apache.org> > > wrote: > > >>>>> > > >>>>> Granted that we just have finalized the Python 3 support, we should > > >>>>> allow time for it to mature and for users to make the switch. > > >>>>> > > >>>>> > Oh, and one more thing, I think it'd make sense for Apache Beam to > > >>>>> > sign https://python3statement.org/. The promise is that we'd > > >>>>> > discontinue Python 2 support *in* 2020, which is not committing us > > to > > >>>>> > January if we're not ready. Worth a vote? > > >>>>> > > >>>>> +1 > > >>>> > > >>>> > > >>>> +1 > > >>>> > > >>>>> > > >>>>> > > >>>>> On 19.09.19 15:59, Robert Bradshaw wrote: > > >>>>> > Oh, and one more thing, I think it'd make sense for Apache Beam to > > >>>>> > sign https://python3statement.org/. The promise is that we'd > > >>>>> > discontinue Python 2 support *in* 2020, which is not committing us > > to > > >>>>> > January if we're not ready. Worth a vote? > > >>>>> > > > >>>>> > > > >>>>> > On Thu, Sep 19, 2019 at 3:58 PM Robert Bradshaw < > > rober...@google.com> wrote: > > >>>>> >> > > >>>>> >> Exactly how long we support Python 2 depends on our users. Other > > than > > >>>>> >> those that speak up (such as yourself, thanks!), it's hard to get > > a > > >>>>> >> handle on how many need Python 2 and for how long. (Should we > > send out > > >>>>> >> a survey? Maybe after some experience with 2.16?) > > >>>> > > >>>> > > >>>> +1, we had some success with collecting information from users using > > Twitter surveys. > > >>>> > > >>>>> > > >>>>> >> > > >>>>> >> On the one hand, the whole ecosystem is finally moving on, and > > even if > > >>>>> >> Beam continues to support Python 2 our dependencies, or other > > projects > > >>>>> >> that are being used in conjunction with Beam, will also be going > > >>>>> >> Python 3 only. On the other hand, Beam is, admittedly, quite late > > to > > >>>>> >> the party and could be the one holding people back, and looking > > at how > > >>>>> >> long it took us, if we just barely make it by the end of the year > > it's > > >>>>> >> unreasonable to say at that point "oh, and we're dropping 2.7 at > > the > > >>>>> >> same time." > > >>>>> >> > > >>>>> >> The good news is that 2.16 is shaping up to be a release I would > > >>>>> >> recommend everyone migrate to Python 3 on. The remaining issues > > are > > >>>>> >> things like some issues with main sessions (which already has > > issues > > >>>>> >> in Python 2) and not supporting keyword-only arguments (a new > > feature, > > >>>>> >> not a regression). I would guess that even 2.15 is already good > > enough > > >>>>> >> for most people, at least to kick the tires and running tests to > > start > > >>>>> >> the effort. > > >>>> > > >>>> > > >>>> I share the same sentiment. Beam 2.16 will offer a strong python 3 > > offering. Yes, there are known issues but this is not much different than > > the known issues for rest of the python offering. > > >>>> > > >>>>> > > >>>>> >> > > >>>>> >> (I also agree with the sentiment that once we go 3.x only, it'll > > be > > >>>>> >> likely harder to maintain a 2.x LTS... but the whole LTS thing is > > >>>>> >> being discussed in another thread.) > > >>>>> > > >>>>> >> > > >>>>> >> On Thu, Sep 19, 2019 at 2:44 PM Chad Dombrova <chad...@gmail.com> > > wrote: > > >>>>> >>> > > >>>>> >>> Hi all, > > >>>>> >>> I had a read through this thread in the archives. It occurred > > before I joined the mailing list, so I hope that this email connects up > > with the thread properly for everyone. > > >>>>> >>> > > >>>>> >>> I'd like to respond to the following points: > > >>>>> >>> > > >>>>> >>>> I believe we are referring to two separate things with support: > > >>>>> >>>> - Supporting existing releases for patches - I agree that we > > need to give > > >>>>> >>>> users a long enough window to upgrade. Great if it happens with > > an LTS > > >>>>> >>>> release. Even if it does not, I think it will be fair to offer > > patches on > > >>>>> >>>> the last python 2 supporting release during some part of 2020 > > if that > > >>>>> >>>> becomes necessary. > > >>>>> >>>> - Making new releases with python 2 support - Each new Beam > > release with > > >>>>> >>>> python 2 support will implicitly extend the lifetime of beam's > > python 2 > > >>>>> >>>> support. I do not think we need to extend this to beyond 2019. > > 2 releases > > >>>>> >>>> (~ 3 months) after solid python 3 support will very likely put > > the last > > >>>>> >>>> python 2 supporting release to last quarter of 2019 already. > > >>>>> >>> > > >>>>> >>> > > >>>>> >>> With so many important features still under active development > > (portability, expansion, external IO transforms, schema coders) and new > > versions of executors tied to the Beam source, staying behind is not really > > an option for many of us, and with python3 support not yet fully completed, > > the window in which Beam is fully working for both python versions is > > rapidly approaching 2 months, and could ultimately be even less, depending > > on how long it takes to complete the dozen remaining issues in Jira, and > > whatever pops up thereafter. > > >>>>> >>> > > >>>>> >>>> The cost of maintaining Python 2.7 support is higher than 0. > > Some issues > > >>>>> >>>> that come to mind: > > >>>>> >>>> - Maintaining Py2.7 / Py 3+ compatibility of Beam codebase > > makes it > > >>>>> >>>> difficult to use Python 3 syntax in Beam which may be necessary > > to support > > >>>>> >>>> and test syntactic constructs introduced in Python 3. > > >>>>> >>>> - Running additional test suites increases the load on test > > infrastructure > > >>>>> >>>> and increases flakiness. > > >>>>> >>> > > >>>>> >>> > > >>>>> >>> I would argue that the cost of maintaining a python2-only LTS > > version will be far greater than maintaining python2 support for a little > > while longer. Dropping support for python2 could mean a number of things > > from simply disabling the python2 tests, to removing 2-to-3 idioms in favor > > of python3-only constructs. If what you have in mind is anything like the > > latter then the master branch will become quite divergent from the LTS > > release, and backporting changes will be not be as simple as cherry-picking > > commits. All-in-all, I think it's a lose/lose for everyone -- users and > > developers, of which I am both -- to drop python2 support on such a short > > timeline. > > >>>>> >>> > > >>>>> >>> I'm an active contributor to this project and it will put me and > > the company that I work for in a very bad position if you force us onto an > > LTS release in early 2020. I understand the appeal of moving to > > python3-only code and I want to get there too, but I would hope that you > > give your users are much time to transition their own code as the Beam > > project itself has taken. I'm not asking for a full 12 months to > > transition, but more than a couple will be required. > > >>>> > > >>>> > > >>>> What would be the ideal time frame for you? > > >>>> > > >>>>> > > >>>>> >>> > > >>>>> >>> thanks, > > >>>>> >>> -chad > > >>>>> >>> > > >>>>> >>> > > >>>>> >>> > > >>>>> >>> > > >
