Re: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]
On 04/03/13 02:51 PM, Mark McLoughlin wrote: On Thu, 2012-12-06 at 07:06 -0800, Adam Williamson wrote: On Thu, 2012-12-06 at 15:30 +0100, Nicolas Mailhot wrote: IMHO use of software collections is a symptom of a badly run organisation not devoting enough cycles to maintain the software it uses, and hoping (as in wishful thinking) no problem will go critical before the product they built on top of those collections is end-of-lifed I completely fail to see how entities with that problem will manage to maintain the package number explosion creating software collections will induce. On the one hand, I agree completely - I think the 'share all dependencies dynamically' model that Linux distros have traditionally embraced is the right one, and that we're a strong vector for spreading the gospel when it comes to that model, and it'd be a shame to compromise that. On the other hand, we've been proselytizing the Java heretics for over a decade now, and the Ruby ones for a while, and neither shows any signs of conversion or just plain going away, so we may have to call it an ecumenical matter and deal with their models somehow. Sucky as it may be. I don't know, I'm a bit conflicted. It's interesting that you call out Java and Ruby folks as being heretics. I guess that means all is kosher with Python? Holy four month old thread revival, Batman! Why yes it does indeed mean that, since as you know, anyone in Fedora will tell you that I am _the_ person to go to for an encyclopaedic knowledge of software development, seeing as how I am the QA monkey, don't know how to write code, and don't develop any software ;) (That was a snarky way of saying, no, it doesn't mean that, it just means I was not aware of any issues like the one you describe below.) OpenStack is getting burned by API instability in some Python packages, so I've started a thread on Python's distutils-sig to try and guage the level of heresy amongst Python folks :) Two things I'm picking up from the thread: - A trend towards semantic versioning and, implicit in that, an acceptance of API breakages so long as the major number of a library version is incremented - Supporting the parallel installation of incompatible versions of libraries isn't seen as an issue because you can just use virtual environments ... which amounts to Python Software Collections. The combination of those two things suggests to me that the Python world will start looking a lot less sane to packagers - i.e. library maintainers breaking API compatibility more often and assuming we can just use SC or similar to have multiple incompatible versions installed. Well, that sounds pretty bad. You may sign my name to the Less Of This Nonsense petition with my blessing... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]
Hi, I find it sad that people are still arguing for the developer-oriented I only care about making application Y as easy to maintain on a wide variety of platforms as possible, and dismiss sysadmin security concerns as too inconvenient to follow, at the very same time one of the biggest proponents of this model, Oracle, is frantically trying to root out and eradicate all the old versions of its software due to exploitation in the wild of its security flaws. I'd think that would invalidate the approach pretty thoroughly (and to be fair Oracle inherited most of the mess from a Sun that didn't dare face developers with hard decisions. It is *no* coincidence that most problems are found in the java plugin, which was 'too hard' to open-source properly and that broke every single software project management rule in order to attract java developers). Are people still naïve enough to think shit only happens to the guy next door? When they'll have finally made every local app too unsafe to run, and forced everyone to use a daily-updated chrome, streaming apps from entities employing sysadmins that force their developers to update their deps in a timely manner, do they think their platform will still be relevant? Because this is what *I* am seeing in the market: slow pruning of entities unable to cope with modern security concerns, and hardening of you shall not take the easy developer path everywhere else. The longer you postpone security concerns the harder they are to handle, and the harder they are too handle the less competitive you get compared to others with better security hygiene. -- Nicolas Mailhot -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]
I'll interject my thoughts here (speaking just for myself): I think software collections are a great thing for us to provide tooling for and make easy for our users/consumers to use. That said, I don't think Fedora as a distribution should ever ship any of them. The tools/framework/etc, great. Actual collections. no. I guess I am just old, but I am pretty saddened by some of the development methodology used by some upstreams these days. The pendulum has been swinging toward bundle, release super fast, break things if we feel like it. I can only guess that this is going to bite people down the road when a widespread security issue hits a core component that lots of them bundle, or when they realize all their consumers no longer even use their newer releases (they just bundle the old ones). Anyhow, IMHO, Fedora as OS should ship one collection of integrated packages. We should try and unbundle where we can, and try and make things work, and try and educate upstreams. Just my 2 cents. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]
On Mon, 2013-03-04 at 22:51 +, Mark McLoughlin wrote: (following up with more thoughts from the distutils-sig thread) It started here: http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html and now we're talking about Software Collections here: http://mail.python.org/pipermail/distutils-sig/2013-March/020074.html Two things I'm picking up from the thread: - A trend towards semantic versioning and, implicit in that, an acceptance of API breakages so long as the major number of a library version is incremented - Supporting the parallel installation of incompatible versions of libraries isn't seen as an issue because you can just use virtual environments ... which amounts to Python Software Collections. I think this parallel installs issue is the key thing we (Fedora, OpenStack, etc.) need to help figure out. There will be incompatible updates to libraries and, like with e.g. gtk2 and gtk3, I think we'll need to support apps using different versions of the same library at the same time. Right now, Software Collections is the only way I can see that OpenStack can be packaged so we don't get screwed when an incompatible library update comes along. Nick Coghlan laid out some ideas for what could be done for Python which I summarised as: http://mail.python.org/pipermail/distutils-sig/2013-March/020082.html - the system has multiple versions of somedep installed under /usr somewhere - the latest version (2.1) is what you get if you naively just do 'import somedep' - most applications should instead somehow explicitly say they need ~1.3, -1.6 or ~-2.0 or whatever - distros would have multiple copies of the same library, but only one copy for each incompatible stream rather than one copy for each application This is a much happier prospect than either Software Collections or no support for parallel installs. It needs help to make it happen, though! Cheers, Mark. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]
On Thu, 2012-12-06 at 07:06 -0800, Adam Williamson wrote: On Thu, 2012-12-06 at 15:30 +0100, Nicolas Mailhot wrote: IMHO use of software collections is a symptom of a badly run organisation not devoting enough cycles to maintain the software it uses, and hoping (as in wishful thinking) no problem will go critical before the product they built on top of those collections is end-of-lifed I completely fail to see how entities with that problem will manage to maintain the package number explosion creating software collections will induce. On the one hand, I agree completely - I think the 'share all dependencies dynamically' model that Linux distros have traditionally embraced is the right one, and that we're a strong vector for spreading the gospel when it comes to that model, and it'd be a shame to compromise that. On the other hand, we've been proselytizing the Java heretics for over a decade now, and the Ruby ones for a while, and neither shows any signs of conversion or just plain going away, so we may have to call it an ecumenical matter and deal with their models somehow. Sucky as it may be. I don't know, I'm a bit conflicted. It's interesting that you call out Java and Ruby folks as being heretics. I guess that means all is kosher with Python? OpenStack is getting burned by API instability in some Python packages, so I've started a thread on Python's distutils-sig to try and guage the level of heresy amongst Python folks :) It started here: http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html and now we're talking about Software Collections here: http://mail.python.org/pipermail/distutils-sig/2013-March/020074.html Two things I'm picking up from the thread: - A trend towards semantic versioning and, implicit in that, an acceptance of API breakages so long as the major number of a library version is incremented - Supporting the parallel installation of incompatible versions of libraries isn't seen as an issue because you can just use virtual environments ... which amounts to Python Software Collections. The combination of those two things suggests to me that the Python world will start looking a lot less sane to packagers - i.e. library maintainers breaking API compatibility more often and assuming we can just use SC or similar to have multiple incompatible versions installed. I can see OpenStack upstream reacting to this by capping its required version range for each library it depends so that if the library does release an incompatible version, OpenStack sticks with the latest compatible version. If that happens, I think OpenStack packagers will need to look seriously at using Software Collections. Basically, we'd look to package and maintain our own stack of all the Python libraries we need above the core Python libraries. So, you'd have openstack-nova, openstack-glance, etc. all installed as normal in /usr, /var, etc. but they'd require python libraries from the openstack-grizzly SC like openstack-grizzly-python-eventlet which would be installed in /opt/fedora/openstack-grizzly/root/usr/lib/python. I'd appreciate it if someone else with a Fedora Python packaging background could look into this and, hopefully, explain how the discussion on distutils-sig isn't so terrifying after all. Cheers, Mark. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
[Fwd: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]]
Hey, Just forwarding it here so Python folks don't miss it on the main devel list. Thanks, Mark. Forwarded Message From: Mark McLoughlin mar...@redhat.com Reply-to: Mark McLoughlin mar...@redhat.com To: Development discussions related to Fedora devel@lists.fedoraproject.org Subject: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?] Date: Mon, 04 Mar 2013 22:51:31 + On Thu, 2012-12-06 at 07:06 -0800, Adam Williamson wrote: On Thu, 2012-12-06 at 15:30 +0100, Nicolas Mailhot wrote: IMHO use of software collections is a symptom of a badly run organisation not devoting enough cycles to maintain the software it uses, and hoping (as in wishful thinking) no problem will go critical before the product they built on top of those collections is end-of-lifed I completely fail to see how entities with that problem will manage to maintain the package number explosion creating software collections will induce. On the one hand, I agree completely - I think the 'share all dependencies dynamically' model that Linux distros have traditionally embraced is the right one, and that we're a strong vector for spreading the gospel when it comes to that model, and it'd be a shame to compromise that. On the other hand, we've been proselytizing the Java heretics for over a decade now, and the Ruby ones for a while, and neither shows any signs of conversion or just plain going away, so we may have to call it an ecumenical matter and deal with their models somehow. Sucky as it may be. I don't know, I'm a bit conflicted. It's interesting that you call out Java and Ruby folks as being heretics. I guess that means all is kosher with Python? OpenStack is getting burned by API instability in some Python packages, so I've started a thread on Python's distutils-sig to try and guage the level of heresy amongst Python folks :) It started here: http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html and now we're talking about Software Collections here: http://mail.python.org/pipermail/distutils-sig/2013-March/020074.html Two things I'm picking up from the thread: - A trend towards semantic versioning and, implicit in that, an acceptance of API breakages so long as the major number of a library version is incremented - Supporting the parallel installation of incompatible versions of libraries isn't seen as an issue because you can just use virtual environments ... which amounts to Python Software Collections. The combination of those two things suggests to me that the Python world will start looking a lot less sane to packagers - i.e. library maintainers breaking API compatibility more often and assuming we can just use SC or similar to have multiple incompatible versions installed. I can see OpenStack upstream reacting to this by capping its required version range for each library it depends so that if the library does release an incompatible version, OpenStack sticks with the latest compatible version. If that happens, I think OpenStack packagers will need to look seriously at using Software Collections. Basically, we'd look to package and maintain our own stack of all the Python libraries we need above the core Python libraries. So, you'd have openstack-nova, openstack-glance, etc. all installed as normal in /usr, /var, etc. but they'd require python libraries from the openstack-grizzly SC like openstack-grizzly-python-eventlet which would be installed in /opt/fedora/openstack-grizzly/root/usr/lib/python. I'd appreciate it if someone else with a Fedora Python packaging background could look into this and, hopefully, explain how the discussion on distutils-sig isn't so terrifying after all. Cheers, Mark. ___ python-devel mailing list python-de...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
Re: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]
- Original Message - On Thu, 2012-12-06 at 07:06 -0800, Adam Williamson wrote: On Thu, 2012-12-06 at 15:30 +0100, Nicolas Mailhot wrote: IMHO use of software collections is a symptom of a badly run organisation not devoting enough cycles to maintain the software it uses, and hoping (as in wishful thinking) no problem will go critical before the product they built on top of those collections is end-of-lifed I completely fail to see how entities with that problem will manage to maintain the package number explosion creating software collections will induce. On the one hand, I agree completely - I think the 'share all dependencies dynamically' model that Linux distros have traditionally embraced is the right one, and that we're a strong vector for spreading the gospel when it comes to that model, and it'd be a shame to compromise that. On the other hand, we've been proselytizing the Java heretics for over a decade now, and the Ruby ones for a while, and neither shows any signs of conversion or just plain going away, so we may have to call it an ecumenical matter and deal with their models somehow. Sucky as it may be. I don't know, I'm a bit conflicted. It's interesting that you call out Java and Ruby folks as being heretics. I guess that means all is kosher with Python? Python makes it a bit hard to install and use multiple parallel versions of libraries with just setuptools/distutils, so it's a bit more kosher :) Since I do both Ruby and Python packaging and I can compare them, it seems to me that the biggest difference is that Python folks aren't used to do specific version requirements (not judging what pros and cons that brings). OpenStack is getting burned by API instability in some Python packages, so I've started a thread on Python's distutils-sig to try and guage the level of heresy amongst Python folks :) It started here: http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html and now we're talking about Software Collections here: http://mail.python.org/pipermail/distutils-sig/2013-March/020074.html Two things I'm picking up from the thread: - A trend towards semantic versioning and, implicit in that, an acceptance of API breakages so long as the major number of a library version is incremented - Supporting the parallel installation of incompatible versions of libraries isn't seen as an issue because you can just use virtual environments ... which amounts to Python Software Collections. The combination of those two things suggests to me that the Python world will start looking a lot less sane to packagers - i.e. library maintainers breaking API compatibility more often and assuming we can just use SC or similar to have multiple incompatible versions installed. I can see OpenStack upstream reacting to this by capping its required version range for each library it depends so that if the library does release an incompatible version, OpenStack sticks with the latest compatible version. There have been similar issues with distro-packaging of Ruby applications using bundler for quite a while now [1], [2], so it's not something unseen. If that happens, I think OpenStack packagers will need to look seriously at using Software Collections. Basically, we'd look to package and maintain our own stack of all the Python libraries we need above the core Python libraries. So, you'd have openstack-nova, openstack-glance, etc. all installed as normal in /usr, /var, etc. but they'd require python libraries from the openstack-grizzly SC like openstack-grizzly-python-eventlet which would be installed in /opt/fedora/openstack-grizzly/root/usr/lib/python. I'd appreciate it if someone else with a Fedora Python packaging background could look into this and, hopefully, explain how the discussion on distutils-sig isn't so terrifying after all. IMHO that depends how the upstreams will actually use the new functionality. It can be used both for good (better semantics of versioning making it easier for packagers to discover API changes) or bad (too tight dependencies of packages, relying on fact that users will just use venv) from Fedora's POV. Cheers, Mark. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Regards, Bohuslav Slavek Kabrda. [1] http://lists.fedoraproject.org/pipermail/devel/2012-December/174912.html [2] http://lists.fedoraproject.org/pipermail/ruby-sig/2012-July/001073.html -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
