On 03/05/2013 08:53 AM, Mark McLoughlin wrote:
I'd appreciate it if someone else with a Fedora Python packaging background could look into this and, hopefully, explain how the discussion on distutils-sig isn't so terrifying after all.
As far as I can see, your concerns are valid in the near term, but manageable in the longer term. The Linux distros have tried to advocate system-wide dynamic linking for years, and ISVs have overwhelmingly responded by choosing not to support the platform, rather than by embracing dynamic linking. ISVs have instead voted with their feet by embracing Microsoft, Google and Apple, all OS vendors that explicitly encourage bundling of dependencies within an application. The Java and Ruby communities don't care, and the Python community doesn't actually care either (we're just a lot more conservative in general about making backwards incompatible changes in the first place). In the fight between easier security updates for system administrators and easier cross-platform and cross-OS-version support for developers, shared dependencies have lost, and lost comprehensively, amongst all but the largest software vendors.
It's *not* a coincidence that CPython publishes pre-built binaries for Windows and Mac OS X, but not for any Linux distro. Cross-distro packaging is just too hard, and not worth the effort, so we just publish the source archive and tell the individual distro communities "you figure it out" (and, to their credit, they generally do).
Even amongst the Linux community, the popularity of virtual appliances and virtual machines in general show that people recognise the serious architectural problems created by coupling nominally independent components together by forcing them to share dependencies. The response to that has not been "virtual machines are evil, because they bundle dependencies" (even though that's exactly what they do - they even bundle the underlying OS!), it has been, "we need to create the appropriate tools to easily update all these virtual machines when it is time to deploy a security fix".
Language specific virtual environments are no different: the response should NOT be "virtual environments are evil, we need to discourage developers from using them", it should be "what tools do we need to create to make it easy to deploy security fixes to language specific virtual environments?". When dependencies are bundled without metadata, that is indeed truly evil, as there's no way to know what needs to be updated to install a security fix. Virtual environments aren't like that - the necessary dependency information is there, the deployment tools to manage security updates just need to be written (and perhaps the behaviour of some existing tools adjusted to make it easier to deploy those fixes).
We can either dig in our heels, demanding that every language include support for dynamic linking against a version of a library at runtime without the distros doing any extra work (and demand that each individual upstream project do the necessary work to select the correct version at runtime), or we can try to create shared dependency systems that *don't suck* from the point of view of a single-developer ISV that wants to easily support arbitrary Linux distros (who may all be shipping different versions of the application's dependencies), as well as Mac OS X and Windows (who probably aren't shipping shared versions of the application's dependencies at all).
Specifically in the context of Python's virtual environments, Fedora and other distros definitely have scope to make contributions that improve the security update handling for system administrators, *without* compromising on ease of deployment for cross-platform developers. For example:
* A Python venv may include *.pth files in the venv's site-packages directory that add more directories to sys.path. This means it is possible to enhance pip and other installers to use a shared version store, *without* needing to use anything other than virtual environments to expose the appropriate versions of the shared dependencies to a given application. This can be done purely through installation tools and Python's existing import infrastructure *without* the application needing to do anything special. * Similar effects can also be achieved through symlinks (a *.pth based approach will likely be an easier sell upstream, though, since "doesn't work properly on Windows" is not an acceptable limitation when it comes to Python's packaging infrastructure)
Python upstream is heavily focused on cross platform development. If distros want to resolve sysadmin specific problems (such as handling security updates to shared dependencies), they need to bring a solution that doesn't make life harder for developers and also works on Mac OS X and Windows, or upstream will ignore you.
If you want me to flesh out the "shared versions for Python virtual environments" idea, so you can pitch it to the pip and virtualenv developers, I can certainly do that, but I don't have time to work on it, or advocate for it myself.
Regards, Nick. -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane Python Applications Team Lead Beaker Development Lead (http://beaker-project.org/) PulpDist Development Lead (http://pulpdist.readthedocs.org) _______________________________________________ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
