Re: Challenges packaging Python for a Linux distro - at Python Language Summit
Hi Thomas (2021.05.12_23:06:45_+) > This looks great. Is there a video of it somewhere? The official blog post on it has been published: https://pyfound.blogspot.com/2021/05/the-2021-python-language-summit_23.html SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: Re: Challenges packaging Python for a Linux distro - at Python Language Summit
Jeremy Stanley writes: > For software development work, I compile my own Python interpreters > and libraries, because I need to develop against a variety of > different versions of these, sometimes in chroots, to be able to > target other distros and releases thereof. I keep all these in my > homedir or otherwise outside of normal system-wide installation > paths. Bear in mind that this isn't just a Debian problem, for > decades Red Hat has also advised programmers not to rely on their > /usr/bin/python for custom development because it is patched and > tuned specifically for running system applications packaged for > their distro and not intended as a general-purpose Python > distribution. There are tools that will help you do this. e.g. pyenv, asdf, etc. But, yes, for my own software development, I gave up trying to use Debian python packages years ago, because I found I was spending way more time trying to package the required dependencies (and dependencies of dependencies of dependencies) then I spending developing my application. This gets even worse if you want to support rpm based distributions. Plus the fact that I was becoming increasingly concerned that if I upgraded library python3-L for application A, I might accidentally break applications B, C, and D also, and testing/fixing everything at the same time not really feasible. So now I use asdf + poetry/virtualenvs now, plus Docker for distribution. As a result, I can upgrade the dependencies in each application one application at a time. If an application breaks, I just need to fix that one application, without breaking everything else at the same time. I can upgrade the OS distribution, and not be concerned (mostly anyway) that it will break my applications in weird and wonderful ways that I could not predict. This isn't so useful for distribution of Python based desktop applications, but I don't do a lot of that (and probably would be looking for a good off-the-shelf solution if I did). -- Brian May
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
Hi Luke, First, I'd like you to know I feel sorry to read how much you seem affected by what you describe below. Hopefully, you'll find a viable solution soon, and hopefully, we may help. However, please try to understand what others are telling. The solution you're looking for is probably not the one you expect, but I'm convince it does exist (ie: venv, probably, or better dependency management). See below. On 5/17/21 8:10 AM, Luke Kenneth Casson Leighton wrote: >> All the horrors that you are painting after this paragraph, are due to >> the fact that you aren't doing "apt-get dist-upgrade". I'm having a hard >> time understanding why you're both: >> - not doing "apt-get dist-upgrade" > > because precisely the difficulties encountered, and due to software > development > that critically requires more recent variants of what is typically in > debian/stable > i.e. usually a minimum of one even three years out-of-date. > > a dist-upgrade to debian / testing - a way to obtain the latest > variants of critical > software - frequently resulted in massive breakage. Massive breakage of what? Your own Python code? Well, it's up to you to stay current to whatever version is in Testing/Unstable. If you don't do it, then it's your responsibility, and you must live with the consequences of it. Yes, it's a massive undertaking to always stay up-to-date, and drains a lot of energy, though no choice if you're serious with what you're doing... > i quickly learned never to attempt that again given the massive disruption > it caused to me being able to earn money as a software engineer. So, to "earn money as a software engineer", you prefer your software to rely on older, probably buggy, and not security supported, dependencies? I'm sorry, but that's far from best practice, and it will bite hard anyways. Yes, dependency management is a hard work. Yes, it takes some of your time (sometimes massively). No, you can't avoid it... >> - complaining that it's breaking your system > > ah, you're missing the point by focussing on the background and context. Are you *SURE* that you're the only person that's right, and that the dozen experienced Debian Developers saying the opposite way are the wrong persons? I would strongly advise you to reconsider. Do you completely understand how "library transitions" are working in Debian? If not, I would strongly advise you to read about it. > i would prefer to solicit some > responses that acknowledge that this is an actual problem that needs > solving. Well, it feels like the only thing you accept, is the answer that *YOU* want, and not the one that others are telling you. Chances that it's going to happen are very thin, since we all know that what you're discussing is not the way Debian works. > Bryan's message unfortunately is a repeat of prior experiences > in reporting this issue over the years. Looks like a lot of people told you the same thing... Do you sometimes listen to what others tell? :) > Brian May wrote: >> A rolling type update might be convenient, but it is not supported by >> Debian, and has not been supported by Debian in sometime. There are >> complexities in such an arrangement, and it is difficult to test such >> arrangements will work as expected. Such an arrangement is not >> guaranteed or tested to work. > > Brian: as an experienced debian systems administrator and python > developer I'm not asking for help with either, and over the past 20 years > have successfully learned and deployed the techniques required to > keep a rolling system operational Apparently, you haven't... > like Thomas, you are missing the point by focussing on the context > and background material rather than focussing on the problem. We aren't at all missing the point that doing the way you're doing is often breaking your system. The problem: you're expecting that never doing "apt-get dist-upgrade" will handle library transitions, when this isn't the way Debian works. We're all telling you, but you do not want to listen, and insist that what you've been doing (wrongly) for 20 years is the correct way of doing things. Sorry, but it's not... You *will* experience breakage, and there's nothing we can do to save you. > this was also the tack taken by others when i explained the problem: > "it's your own fault, this isn't supported, therefore we don't have to > do anything, therefore it's not a bug, therefore the bugreport is > summarily dismissed". Which is 100% correct. > in my report, if you read it again carefully, i specifically stated that > *multiple debian maintainers* are receiving *multiple complaints* of > breakage and disruption due to the standard debhelper-python > build system not following the "safe" practice outlined by > numpy and sci-py. The problem isn't the build system, which is doing things the correct way. Any "arch: any" package linked against Python *must* be upgraded together with the interpreter itself when it transition from Unstable to Testing.
Re: Re: Challenges packaging Python for a Linux distro - at Python Language Summit
On 2021-05-17 07:10:39 +0100 (+0100), Luke Kenneth Casson Leighton wrote: > (apologies i forgot to say, please do cc me [...] Done. > a dist-upgrade to debian / testing - a way to obtain the latest > variants of critical software - frequently resulted in massive > breakage. > > i quickly learned never to attempt that again given the massive > disruption it caused to me being able to earn money as a software > engineer. [...] You're probably just going to see this as further confirmation of your opinion, or yet another person telling you that you're doing it wrong, but as someone who also writes rather a lot of Python programs for a living I learned long ago to not develop against the "system" on any platform. I use sid/unstable for my development systems, but I consider the python3 package in it to have two uses: Testing new packages of software which are targeting a future Debian stable release, and running other packaged things which are already part of Debian. For software development work, I compile my own Python interpreters and libraries, because I need to develop against a variety of different versions of these, sometimes in chroots, to be able to target other distros and releases thereof. I keep all these in my homedir or otherwise outside of normal system-wide installation paths. Bear in mind that this isn't just a Debian problem, for decades Red Hat has also advised programmers not to rely on their /usr/bin/python for custom development because it is patched and tuned specifically for running system applications packaged for their distro and not intended as a general-purpose Python distribution. -- Jeremy Stanley signature.asc Description: PGP signature
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
On 5/16/21 1:52 PM, Luke Kenneth Casson Leighton wrote: >> * One 3.x version at a time. Doesn't line up with cpython's support terms. > numpy and sci-py, the two "best known" debian python software > packages, have known about this for a long, long, time. they > quietly solved it by adding explicit building of **MULTIPLE** > versions of numpy-3.5, numpy-3.6, numpy-3.7, etc. etc. > i leave you with these insights. there *is* a solution: do what numpy and > sci-py do. I don't understand why you are pointing to numpy and scipy as good examples. The modules are not built in any different way from all the modules shipped in Debian. Matthias
Re: Re: Challenges packaging Python for a Linux distro - at Python Language Summit
(apologies i forgot to say, please do cc me, i am using the list archives reply-to links at https://lists.debian.org/debian-python/2021/05/msg00036.html) Thomas Goirand wrote: > All the horrors that you are painting after this paragraph, are due to > the fact that you aren't doing "apt-get dist-upgrade". I'm having a hard > time understanding why you're both: > - not doing "apt-get dist-upgrade" because precisely the difficulties encountered, and due to software development that critically requires more recent variants of what is typically in debian/stable i.e. usually a minimum of one even three years out-of-date. a dist-upgrade to debian / testing - a way to obtain the latest variants of critical software - frequently resulted in massive breakage. i quickly learned never to attempt that again given the massive disruption it caused to me being able to earn money as a software engineer. > - complaining that it's breaking your system ah, you're missing the point by focussing on the background and context. also, i didn't "complain" (despite it causing distress for some considerable number of years), i would have said, "i have a complaint" and stopped at that, rather than moved on to describe the details of what the root cause is. > Could you care to explain? This makes absolutely no sense. i could - but doing so is a distraction. i would prefer to solicit some responses that acknowledge that this is an actual problem that needs solving. Bryan's message unfortunately is a repeat of prior experiences in reporting this issue over the years. > By the way, since you're never doing "apt-get dist-upgrade", it means > your system is full of security issues that aren't getting fixed. > Hopefully, the computer system(s) you're talking about isn't connected > to internet, right? indeed. it's a development laptop on which i am critically reliant for financial income and for furthering free software. Brian May wrote: > A rolling type update might be convenient, but it is not supported by > Debian, and has not been supported by Debian in sometime. There are > complexities in such an arrangement, and it is difficult to test such > arrangements will work as expected. Such an arrangement is not > guaranteed or tested to work. Brian: as an experienced debian systems administrator and python developer I'm not asking for help with either, and over the past 20 years have successfully learned and deployed the techniques required to keep a rolling system operational, in order to keep my business operational and not lose money due to massive downtime either from having to transfer 10 million source code files from laptop to laptop, or from having to perform disaster recovery due to breakage. like Thomas, you are missing the point by focussing on the context and background material rather than focussing on the problem. this was also the tack taken by others when i explained the problem: "it's your own fault, this isn't supported, therefore we don't have to do anything, therefore it's not a bug, therefore the bugreport is summarily dismissed". in my report, if you read it again carefully, i specifically stated that *multiple debian maintainers* are receiving *multiple complaints* of breakage and disruption due to the standard debhelper-python build system not following the "safe" practice outlined by numpy and sci-py. i can only surmise that they probably don't want to say anything because the message being sent to them on summary closure of bugreports is that, well, "nobody cares". what i can say is that they're getting pretty fed up of constantly having to close bugreports, every time someone raises this issue, that's pretty clear from what they didn't say, if you know what i mean. once this is accepted as actually being a problem that needs solving, rather than being avoided because "it's not supported, you're on your own, not our problem", i am happy to help advise and discuss solutions. given that this has been ongoing for 10 years now i have been giving it some thought for a long, long time. however before getting to a discussion of solutions i would prefer to see acknowledgement that it is recognised as a problem that people actively wish to see fixed. otherwise i will have to wait patiently for the next disaster situation to occur, or, sadly, after 20 years of using debian, and remaining loyal to it despite maltreatment, i will have to find an alternative distro. what is stopping me from doing that is the severe financial consequences involved and risk to myself and my family due to my income being far below average. the downtime that would result is a huge risk, plus learning a new distro also compromises my effectiveness which also results in further lost income. what i am saying is: this is serious - i am effectlvely financially trapped and critically dependent on the decisions that you make, even though i am not paying you money for the work that you do. l.
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
Thomas Goirand writes: > All the horrors that you are painting after this paragraph, are due to > the fact that you aren't doing "apt-get dist-upgrade". I'm having a hard > time understanding why you're both: > - not doing "apt-get dist-upgrade" > - complaining that it's breaking your system > > Could you care to explain? This makes absolutely no sense. A rolling type update might be convenient, but it is not supported by Debian, and has not been supported by Debian in sometime. There are complexities in such an arrangement, and it is difficult to test such arrangements will work as expected. Such an arrangement is not guaranteed or tested to work. In short, if you want to upgrade to Debian version n, you really should be running entirely Debian n-1 before you start the upgrade. This is important. And you probably should upgrade everything in one go. As this is the only tested upgrade procedure. -- Brian May
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
On 5/16/21 1:52 PM, Luke Kenneth Casson Leighton wrote: >> * One 3.x version at a time. Doesn't line up with cpython's support terms. > > folks, deep breath here: this is much more important than the one line > summary suggests. > > for some background: i have been using debian since 1996 and python for > 20+ years, dating back to python 2.0. > > due to the massive amount of accumulated software (several million > development source code files) i run a rolling debian/testing system, > *never* do an "apt-get dist-upgrade", *always* simply perform an on-demand > install of a given package and let apt sort itself out even to the > point often of > having some innocuous package end up pulling in a new libc6-dev and > binutils. All the horrors that you are painting after this paragraph, are due to the fact that you aren't doing "apt-get dist-upgrade". I'm having a hard time understanding why you're both: - not doing "apt-get dist-upgrade" - complaining that it's breaking your system Could you care to explain? This makes absolutely no sense. By the way, since you're never doing "apt-get dist-upgrade", it means your system is full of security issues that aren't getting fixed. Hopefully, the computer system(s) you're talking about isn't connected to internet, right? Cheers, Thomas Goirand (zigo)
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
On Sun, May 16, 2021 at 12:52 PM Luke Kenneth Casson Leighton wrote: > to fix this, there needs to be a stable overlap where multiple compiled > versions > are installed and maintained. to fix the problem, the rule needs to be set: > > * any given version of python *MUST* cross over from one debian/stable > to another debian/stable. > * only when a given version of python has been in TWO versions of > debian/stable may it be considered for retirement / removal sorry, correction / clarification: any given version of python *and all python packages* must cross over from and be permanently and irrevocably available across two versions of debian/stable. this will sove in one fell swoop all of the issues that have been reported multiple times by multiple people for many years, creating a long-term reliable and stable distribution. you will be able to *SAFELY* do what you just did - back out of python 3.8 and deploy 3.9 - *WITHOUT* completely destroying people's systems because 3.7 was force-removed and they were given absolutely no choice in the matter. btw please, please, i am anticipating responses of the type "it's your problem, you shouldn't be using debian/testing, if you do you get everything you deserve". if that thought crossed your mind, please reconsider, and ask instead "why is this person using debian/testing for 20 years". l.
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
> * One 3.x version at a time. Doesn't line up with cpython's support terms.
folks, deep breath here: this is much more important than the one line
summary suggests.
for some background: i have been using debian since 1996 and python for
20+ years, dating back to python 2.0.
due to the massive amount of accumulated software (several million
development source code files) i run a rolling debian/testing system,
*never* do an "apt-get dist-upgrade", *always* simply perform an on-demand
install of a given package and let apt sort itself out even to the
point often of
having some innocuous package end up pulling in a new libc6-dev and
binutils.
generally this works extremely well: in 20 years the only really serious
problems occur are, yes, you guessed it: python related.
correction: the situation around python 2.2, 2.3 to 2.5 was perfectly fine,
i will touch on this again below.
now, i do hesitate to mention this, but what i am about to tell you, *multiple*
people including debian python package maintainers, have, over the years,
raised it *multiple* times on *multiple* separate occasions, and each and
every time they have been ignored and the bugreport shut down without
due consideration.
if you recall the situation with a version of ubuntu where they created two
versions of ubuntu where python 2.5 was the *only* version supported in
one, and python 2.6 was the *only* version of python available in the next,
anyone who performed an "apt-get dist-upgrade" had python REMOVED
from their running system, and, given that apt itself was critically dependent
on python at the time, people ended up with a permanently unuseable
system.
debian's python packaging is rushing ahead so fast at the moment that it
is in danger of running into something similar.
the key issue is that python-dev is an exclusive "narrow window" that, on
each upgrade, moves to one *AND ONLY ONE* version of python.
python3.6-dev, python3.7-dev, python3.8-dev, and now python3.9-dev.
this is not a problem for the majority of packages that are pure python.
it is however a REALLY SERIOUS problem for c-based modules, and
for python-boost applications it is absolute hell on earth (boost is
a long-term known thorn in the side of developers, and the python bindings
add a layer of extra special hell on top of that)
let's go through a scenario, going back in time somewhat
* developer (me) uses debian/testing which, nominally, is by now
mostly debian 9 packages, including (mostly) python 3.6
* debian / testing flips python-dev to python3.7-dev and builds
begin
* ALL PACKAGES WHICH DEPEND ON 3.6 NOW BREAK
due to the hard-coded numbering, *ANY* package containing
c-based modules has debian packaging which now says
"this package now EXCLUSIVELY depends on python3.7.so"
in that package, which is **NOT** numbered python3.7-{packagename},
it is named **PYTHON**-{packagename}, there is **ONLY**
a python3.7.so: the dependency on python3.6.so has been
**REMOVED**
that package, small as it is, ends up with a dependency cascade
that can result in literally HUNDREDS os packages being removed
and/or force-upgraded unnecessarily.
thus, i end up in a really serious situation, just like with ubuntu about
10 years ago, where the only option is to literally remove everything.
given that i am trying to do some development work on which i rely
for income, that is simply not an option to remove critical packages.
now, where it gets particularly bad is during the cross-over point
where some of the packages have not been re-built yet, and some
have. this happened very recently with the 3.7 to 3.8 cross-over:
an innocuous and simple package had a dependency cascade
where half the required packages - all of which were c-based -
had not even been built yet.
numpy and sci-py, the two "best known" debian python software
packages, have known about this for a long, long, time. they
quietly solved it by adding explicit building of **MULTIPLE**
versions of numpy-3.5, numpy-3.6, numpy-3.7, etc. etc.
this "rolling release" system basically solves absolutely everything,
and causes no problems whatsoever of any kind.
if i may repeat that again so as to emphasise the deep significance:
numpy and sci-py have *absolutely no problems whatsoever* when
it comes to these critical long-term rolling upgrades.
i cannot emphasise enough how serious this really is. it is
asking for trouble, and, more than that, has been reported multiple
times and been ignored each and every time.
unfortunately, very recently, it gets even worse.
i was extremely dismayed and alarmed, recently, to learn, on trying
to install a package, that there has been no "stabilisation time"
allowed, and that the latest version of debian/testing has *already*
moved on to python 3.9.
now, if the latest version of debian/testing had stayed with python 3.8,
there would be no problem. debian/testing would stabilise and become
debian/11, and there would be some peace at least for a while.
it gets worse.
one
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
On 5/13/21 1:26 AM, Stefano Rivera wrote: > Hi Thomas (2021.05.12_23:06:45_+) >> On 5/12/21 11:21 PM, Stefano Rivera wrote: >>> Matthias Klose gave a presentation at the Python Language Summit on the >>> Challenges packaging Python for a Linux distro. >>> [..] >> This looks great. Is there a video of it somewhere? > > No, there won't be videos published, only blog posts written. > > SR Matthias, if you read this: you *MUST* make such a presentation at the next debconf, *PLEASE* !!! Cheers, Thomas Goirand (zigo)
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
FWIW, if might come handy in the future, my 4c: > * What do we provide for scientific / data scientist use cases? - https://snapshot.debian.org/ is the unique service allowing to "go back in time" or just "freeze" the environment given a date. Very handy for reproducibility, collab, etc. Not possible AFAIK on pypi or even conda unless researcher prepared a full exhaustive list of frozen package==version@build nd_freeze from neurodebian-freeze assists in making use of that feature. I just stick it at the top of my Dockerfile/Singularity file recipes to make container itself as reproducible as possible, so later on I could add another component less likely affecting already existing ones. - wider arch support for extensions and non-python libraries/tools. ppc64el is gaining some momentum AFAIK in sci computing - better guarantees to achieve desired installation goal. examples of pip/conda failing to resolve depends are more numerous AFAIK. - integration and downstream testing at package build time and via https://ci.debian.net/ anyone who cares to not only "get it running" but have some assurance of correct operation (not junk-in-junk-out) should appreciate that. pypi has no concerns on that at all. conda is doing quite good job and does allow for some downstream testing. But it remains "more fluid", unlike a clear cut releases of debian with better guarantees for correct operation -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 WWW: http://www.linkedin.com/in/yarik
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
Hi Thomas (2021.05.12_23:06:45_+) > On 5/12/21 11:21 PM, Stefano Rivera wrote: > > Matthias Klose gave a presentation at the Python Language Summit on the > > Challenges packaging Python for a Linux distro. > > [..] > This looks great. Is there a video of it somewhere? No, there won't be videos published, only blog posts written. SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: Challenges packaging Python for a Linux distro - at Python Language Summit
On 5/12/21 11:21 PM, Stefano Rivera wrote: > Matthias Klose gave a presentation at the Python Language Summit on the > Challenges packaging Python for a Linux distro. > [..] This looks great. Is there a video of it somewhere? Cheers, Thomas Goirand (zigo)
