> > > I'm working on an "Essential Tools for Managing Python Development > Environments > <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>" > tutorial > Awesome! Did you consider conda envs? FWIW, we rely on conda envs > <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments> > in > the Data Engineering world to work around the lack of ability to > securely run docker images in production. > We didn't try pyenv, mainly because conda gets us more than just python :)
I tried conda a few years ago, but for my own use cases, I found it mostly got in my way. I know it's big with data and science folks though, so I will add it to an "other tools to be aware of" section. But now I'm curious about how conda enables running docker safely in production. :) For anyone interested in the (bewildering complexity of the current) Python packaging ecosystem, I can recommend this blog post by PyPA member and PSF fellow Pradyum Gedam: https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/ <https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/> -- Slavina Stefanova (she/her) Software Engineer - Technical Engagement Wikimedia Foundation On Mon, May 8, 2023 at 3:03 PM Andrew Otto <[email protected]> wrote: > > For Java, we run an instance of Archiva: https://archiva.wikimedia.org/ > > It's not a perfect approach but I think we can and should move in that > direction with all our other ecosystems > > Gitlab package registries may help us here! > > > > On Mon, May 8, 2023 at 8:59 AM Andrew Otto <[email protected]> wrote: > >> > Tangent: is it worthwhile to establish a consensus for best practices >> with package pinning and package management for Python projects in the >> Wikimedia ecosystem? >> Yes! That would be awesome. I have spent a lot of time floundering in >> this area trying to make decisions; it'd be nice if we had a good guideline >> established. >> >> > I'm working on an "Essential Tools for Managing Python Development >> Environments >> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>" >> tutorial >> Awesome! Did you consider conda envs? FWIW, we rely on conda envs >> <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments> >> in the Data Engineering world to work around the lack of ability to >> securely run docker images in production. We didn't try pyenv, mainly >> because conda gets us more than just python :) >> >> > Poetry is a modern lockfile-based packaging and dependency management >> tool worth looking into >> Poetry was nice, but I found that it wasn't as comprehensive (yet?) as ye >> old setuptools based stuff. I can't quite recall what, but I think it was >> around support for datafiles and scripts? But, this was 1.5 years ago so >> maybe things are better now. >> >> Thank you! >> >> On Fri, May 5, 2023 at 11:32 AM Slavina Stefanova < >> [email protected]> wrote: >> >>> Tangent: is it worthwhile to establish a consensus for best practices >>>> with package pinning and package management for Python projects in the >>>> Wikimedia ecosystem? When I last worked on a python project ( >>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing >>>> that we have so many different tools and approaches for doing these things, >>>> and seems like we'd benefit from having a standard, supported way. (Or >>>> maybe that already exists and I haven't found it?) >>> >>> >>> I'm working on an "Essential Tools for Managing Python Development >>> Environments >>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>" >>> tutorial that will be published to the wikis when ready. Maybe that could >>> be expanded upon? In my experience though, it can be hard to get people to >>> agree on following a standard, especially when there are so many different >>> options and many folks already have their favorite tools and workflows. But >>> it would be nice to have a set of recommendations to reduce the cognitive >>> load. >>> >>> -- >>> Slavina Stefanova (she/her) >>> Software Engineer - Technical Engagement >>> >>> Wikimedia Foundation >>> >>> >>> On Fri, May 5, 2023 at 5:18 PM Kosta Harlan <[email protected]> >>> wrote: >>> >>>> Tangent: is it worthwhile to establish a consensus for best practices >>>> with package pinning and package management for Python projects in the >>>> Wikimedia ecosystem? When I last worked on a python project ( >>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing >>>> that we have so many different tools and approaches for doing these things, >>>> and seems like we'd benefit from having a standard, supported way. (Or >>>> maybe that already exists and I haven't found it?) >>>> >>>> Kosta >>>> >>>> On 5. May 2023, at 13:51, Slavina Stefanova <[email protected]> >>>> wrote: >>>> >>>> Poetry is a modern lockfile-based packaging and dependency management >>>> tool worth looking into. It also supports exporting dependencies into a >>>> requirements.txt file, should you need that (nice if you want to >>>> containerize an app without bloating the image with Poetry, for instance). >>>> >>>> https://python-poetry.org/ <https://python-poetry.org/> >>>> >>>> -- >>>> Slavina Stefanova (she/her) >>>> Software Engineer - Technical Engagement >>>> >>>> Wikimedia Foundation >>>> >>>> >>>> On Fri, May 5, 2023 at 1:38 PM Sebastian Berlin < >>>> [email protected]> wrote: >>>> >>>>> A word of warning: using `pip freeze` to populate requirements.txt can >>>>> result in a hard to read (very long) file and other issues: >>>>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895 >>>>> . >>>>> >>>>> *Sebastian Berlin* >>>>> Utvecklare/*Developer* >>>>> Wikimedia Sverige (WMSE) >>>>> >>>>> E-post/*E-Mail*: [email protected] >>>>> Telefon/*Phone*: (+46) 0707 - 92 03 84 >>>>> >>>>> >>>>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <[email protected]> >>>>> wrote: >>>>> >>>>>> You can also create an empty virtual env, install all requirements >>>>>> and then do >>>>>> pip freeze > requirements.txt >>>>>> >>>>>> That should take care of pinning >>>>>> >>>>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister < >>>>>> [email protected]>: >>>>>> >>>>>>> For the general case of Python projects, I’d argue that a better >>>>>>> solution is to adopt the lockfile pattern (package-lock.json, >>>>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and >>>>>>> only update them when the new versions have been tested and are known to >>>>>>> work. pip-tools <https://pypi.org/project/pip-tools/> can help with >>>>>>> that, for example (requirements.in specifies “loose” dependencies; >>>>>>> pip-compile creates a pinned requirements.txt; pip-sync installs >>>>>>> it; pip-compile -U upgrades requirements.txt later; you check both >>>>>>> requirements.in and requirements.txt into version control.) But I >>>>>>> don’t know if that applies in your integration/config case. >>>>>>> >>>>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso < >>>>>>> [email protected]>: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> This is for python projects. >>>>>>>> >>>>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history> >>>>>>>> has released a new major version 2.0.2 which breaks the extremely >>>>>>>> popular >>>>>>>> requests <https://pypi.org/project/requests/> library. >>>>>>>> >>>>>>>> The fix is to pin urllib3<2 to prevent the new major version from >>>>>>>> being installed (example >>>>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini> >>>>>>>> ). >>>>>>>> >>>>>>>> https://phabricator.wikimedia.org/T335977 >>>>>>>> >>>>>>>> Upstream issue: https://github.com/psf/requests/issues/6432 >>>>>>>> >>>>>>>> >>>>>>>> Antoine "hashar" Musso >>>>>>>> Wikimedia Release Engineering >>>>>>>> _______________________________________________ >>>>>>>> Wikitech-l mailing list -- [email protected] >>>>>>>> To unsubscribe send an email to >>>>>>>> [email protected] >>>>>>>> >>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Lucas Werkmeister (he/er) >>>>>>> Software Engineer >>>>>>> >>>>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin >>>>>>> Phone: +49 (0)30-577 11 62-0 >>>>>>> https://wikimedia.de >>>>>>> >>>>>>> Imagine a world in which every single human being can freely share >>>>>>> in the sum of all knowledge. Help us to achieve our vision! >>>>>>> https://spenden.wikimedia.de >>>>>>> >>>>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. >>>>>>> V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg >>>>>>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt >>>>>>> für Körperschaften I Berlin, Steuernummer 27/029/42207. >>>>>>> _______________________________________________ >>>>>>> Wikitech-l mailing list -- [email protected] >>>>>>> To unsubscribe send an email to [email protected] >>>>>>> >>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Amir (he/him) >>>>>> >>>>>> _______________________________________________ >>>>>> Wikitech-l mailing list -- [email protected] >>>>>> To unsubscribe send an email to [email protected] >>>>>> >>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>>> >>>>> _______________________________________________ >>>>> Wikitech-l mailing list -- [email protected] >>>>> To unsubscribe send an email to [email protected] >>>>> >>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>> >>>> _______________________________________________ >>>> Wikitech-l mailing list -- [email protected] >>>> To unsubscribe send an email to [email protected] >>>> >>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>> >>>> >>>> _______________________________________________ >>>> Wikitech-l mailing list -- [email protected] >>>> To unsubscribe send an email to [email protected] >>>> >>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>> >>> _______________________________________________ >>> Wikitech-l mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> >>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >> >> _______________________________________________ > Wikitech-l mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
_______________________________________________ Wikitech-l mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
