> The Python 3 upgrade shouldn't be too difficult; happy to help out. Make > sure you duplicate your configuration so that CMake still works (happy to > help there also). Just message (privately or publicly).
Yeah I think shatil mentioned the source translation was rather straightforward, but the complexity is in the build dependencies (protobuf / grpc) and the scheduler / executor bindings (see below). > Do you require both Python 2 and 3 bindings to work, or can we make a clean > migration to 3? I ask because there are breaking changes in C/C++ bindings > going from 2 to 3, and supporting both will require conditionals rather > than replacements of code blocks here and there. Dependencies may also end > up incompatible between the two Python versions since some dropped Python > 2.7 support a long time ago. There are two areas of python code in our repo to consider: 1. Python code unrelated to scheduler / executor bindings (e.g. CLI, test runner, etc), we can go directly to Python 3. Don't need 2 + 3 compatibility AFAICT. 2. Python scheduler / executor bindings, this is an older API that we "replaced" with a newer HTTP API many years ago. I believe most users have since adopted the HTTP API, but within X we still use both the older CPython bindings and JNI bindings. At least for ourselves, we will need an upgrade story, e.g.: a. Ideally, python 2 and python 3 works using the same bindings / libmesos to simplify upgrades. b. Or, separate python 2 and python 3 bindings are available that work with the same updated libmesos, this allows users to potentially de-couple the scheduler / executor upgrade from their libmesos upgrade (upgrade libmesos first, then upgrade executor or scheduler). c. Or, the python bindings are upgraded to python 3 only (upgrade libmesos + executor/scheduler at the same time). Is (a) even possible? It seems the simplest from an upgrade perspective. If we choose (c), how will we do our python executor upgrade given we deploy libmesos and the executor binary to our fleet separately? On Sat, Jan 13, 2024 at 7:32 PM Qian Zhang <zhq527...@gmail.com> wrote: > Thanks Ben! I have let Shane know this to hold off on going into the attic, > and happy to help on the Cgroups v2 support :-) > > > Regards, > Qian Zhang > > > On Sat, Jan 13, 2024 at 8:48 AM Samuel Marks <sam...@offscale.io> wrote: > > > On latest commit 5109b9069b62510ab6d0cc0c78a9ed8327d3986a `fd -epy | wc > -l` > > shows 73 files. > > > > mesos/src/python/cli/src/mesos/http.py requires a trivial change to > support > > Python 2 & 3. Everything else in that dir is fine. > > > > mesos/src/python/cli_new/lib/cli/util.py requires two lines to be > > conditioned on Python 2/3 to resolve the now removed `imp` builtin > module. > > > > mesos/src/python/cli/src/mesos/futures.py needs cancel_futures > implemented > > or stubbed with a `NotImplementedError` thrown > > > > Then you're down to 17 files left incompatible with Python 2 & 3. > > > > mesos/support has 15 of these files; quickly reading `rg -Ftpy import` > > shows nothing interesting. Should be mostly trivial. Take care though to > > remove distutils; as that was removed in Python 3.12. > > > > mesos/src/examples/python contains the final 2 Python files. A cursory > > glance I can't see anything Python 2 & 3 incompatible. > > > > (happy to send the PR/patch or you can) > > > > Samuel Marks > > Charity <https://sydneyscientific.org> | consultancy < > https://offscale.io> > > | open-source <https://github.com/offscale> | LinkedIn > > <https://linkedin.com/in/samuelmarks> > > > > > > On Fri, Jan 12, 2024 at 6:46 PM Shatil Rafiullah <sha...@twitter.com> > > wrote: > > > > > Do you require both Python 2 and 3 bindings to work, or can we make a > > > clean migration to 3? I ask because there are breaking changes in C/C++ > > > bindings going from 2 to 3, and supporting both will require > conditionals > > > rather than replacements of code blocks here and there. Dependencies > may > > > also end up incompatible between the two Python versions since some > > dropped > > > Python 2.7 support a long time ago. > > > > > > On Fri, Jan 12, 2024 at 3:18 PM Samuel Marks <sam...@offscale.io> > wrote: > > > > > >> The Python 3 upgrade shouldn't be too difficult; happy to help out. > Make > > >> sure you duplicate your configuration so that CMake still works (happy > > to > > >> help there also). Just message (privately or publicly). > > >> > > >> Good to see interest in Mesos remaining, > > >> > > >> Samuel Marks > > >> Charity > > >> < > > > https://urldefense.com/v3/__https://sydneyscientific.org__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbSXruBk3U$ > > > > > >> | consultancy > > >> < > > > https://urldefense.com/v3/__https://offscale.io__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbSQf69uCY$ > > > > > >> | open-source > > >> < > > > https://urldefense.com/v3/__https://github.com/offscale__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbSR5dWm9A$ > > > > > >> | LinkedIn > > >> < > > > https://urldefense.com/v3/__https://linkedin.com/in/samuelmarks__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbS0q7dfzk$ > > > > > >> > > >> > > >> On Fri, Jan 12, 2024 at 6:01 PM Benjamin Mahler <bmah...@apache.org> > > >> wrote: > > >> > > >>> +user@ > > >>> > > >>> On Fri, Jan 12, 2024 at 5:55 PM Benjamin Mahler <bmah...@apache.org> > > >>> wrote: > > >>> > > >>> > As part of upgrading to CentOS 9 at X/Twitter, Shatil / Devin > (cc'ed) > > >>> will > > >>> > be working on: > > >>> > > > >>> > * Upgrading to Python 3 > > >>> > * Cgroups v2 support > > >>> > > > >>> > We will attempt to upstream this work for the benefit of other > users. > > >>> > > > >>> > In addition, we have several long-standing internal patches that > > should > > >>> > have been upstreamed but weren't. We currently deploy from an > > internal > > >>> > 1.9.x branch with these additional patches on top of OSS 1.9.x. To > > >>> simplify > > >>> > the above work since the code changes will be substantial > (especially > > >>> for > > >>> > cgroups v2), we'll try to work off master (or 1.11.x) and upstream > > our > > >>> > long-standing internal patches to minimize the delta between OSS > and > > >>> our > > >>> > internal branch. > > >>> > > > >>> > Let me know if folks have any questions. IMO it would be good for > > >>> users to > > >>> > hold off on going into the attic so that we can land this work at > > >>> least. > > >>> > > > >>> > Ben > > >>> > > > >>> > > >> > > >
