Re: packaging problems with python3.5 and python 3.6
On Jul 30, 2017, at 02:41, Steve Langasek wrote: >> https://wiki.debian.org/Python/LibraryStyleGuide where it states: >> You'll want to have at least the following build dependencies: >> ... >> python3-all >> but the >> python[,3]-all-dev >> are never mentioned. So sorry, I didn't know about them and searched >> for the required header files, and added the respective packages. > Ok, that was the first hit I got but I discounted it because it spoke of > packaging "libraries", not modules or extensions. “Libraries” is typically the term we’ve used to describe things you import from other things. In Python, everything you import is a module, and packages are just special kinds of modules, whereas extensions are module implemented in a compiled language instead of native Python. It may not be the best nomenclature, but it’s what we’ve been using for a long time. > Perhaps someone should make this wiki page a redirect to the python policy > manual? https://www.debian.org/doc/packaging-manuals/python-policy/ No, because LibraryStyleGuide *isn’t* policy, it’s an opinionated collection of best practices, and it is better suited for a wiki because those best practices do change over time. Policy is what you MUST do, and the barrier for changing policy is — and needs to be — higher. Cheers, -Barry signature.asc Description: Message signed with OpenPGP
Re: packaging problems with python3.5 and python 3.6
On Aug 2, 2017, at 15:34, Steve Langasek wrote: > > Except the python packaging policy already mentions build dependencies: > > "The python3-all-dev should be used when building extensions for any or all > Python 3 versions. The python-all-dev should be used when building > extensions for any or all Python 2 versions." > > > https://www.debian.org/doc/packaging-manuals/python-policy/ap-build_dependencies.html > > But this is then shadowed by the less precise/complete recommendation in the > wiki. Note that the LSG does link to the policy page and even says: "It is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it.” and goes on: "Instead, this page captures in example form, best practices for simple Python libraries that you might find on the Python Package Index.” Maybe packages containing extensions are simple enough to include on this page. It’s a wiki so, 1) it’s a living document; 2) contributions welcome! :) -Barry signature.asc Description: Message signed with OpenPGP
Re: MBF for deprecating Python2 usage
On Aug 3, 2017, at 17:57, Matthias Klose wrote: > > While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be > done > to deprecate Python2 usage within the distribution. It might not be possible > to > drop Python2 for the next release, but there are still too many issues with > packages. For now we identified some categories which need fixing. These are > documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting > in > more than 3000 bug reports. That's a high number, on the other hand we won't > make much progress if the issues are not named. My intent is to bring that up > in the Python BoF next week at DebConf and then filing these bug reports with > the user tags mentiond on the wiki page. Great to hear that you guys talked about it. Just a quick note that PEP 394 discussions have revived, lead by the Fedora folks. Please do check out the new thread, especially if you have opinions about what /usr/bin/python should do once Python 2.7 is EOL. https://mail.python.org/pipermail/linux-sig/2017-August/thread.html Cheers, -Barry signature.asc Description: Message signed with OpenPGP
Re: MBF for deprecating Python2 usage
On Aug 3, 2017, at 23:23, Scott Kitterman wrote: > > Read it. I remain completely convinced that /usr/bin/python pointing at a > python3 version is utterly wrong and a disservice to our users. Even after > we remove python2.7, people will be locally compiling it and using it for a > decade. I think it’s inevitable, and not doing so *eventually* will be a greater disservice. Sure, not today, maybe not in 2020, but I don’t think we’ll be having this discussion in 2025. If by then, 5 years after Python 2.7 EOL, users read on SO or some Learning Python site that they can just type ‘python' to get an interactive prompt, but it doesn’t work on Debian, they’ll think Debian is broken, not Python. So I think it makes sense to consider how the transition will work. You’re right that folks will still need Python 2 after 2020, and that their best option may be to build it themselves, but I don’t think they’ll be building and installing some old Debian package. Instead they may build from source, which means they’ll be installing it into /usr/local/bin not /usr/bin, and they’ll have to change their shebang lines anyway. And there will come a time when even Python 2.7 is unbuildable as toolchains and libraries evolve. Upstream will stop tracking those changes so even the upstream git repo won’t give you buildable source. There may be a bunch of third party forks, but I don’t think it’s our responsibility to ensure that Debian aligns with any of those hypotheticals. Folks needing to stay on Python 2 will probably also just elect to not upgrade their OS, so what we do in future releases won’t matter to them. Upstream and Debian already install a `python2` alias for `python`, and I think we need to join the chorus that in the future, the way to run Python 2 is via `python2`. Yes, people will have legacy stuff around for a long time, but it’s better to begin the transition now rather than make a major break years from now. > Such a change would be actively user hostile. > > When python2.7 goes away, /usr/bin/python should go too. Maybe in the short term (as Matthias suggests), but I’m not so sure it’s best to disappear /usr/bin/python forever. /usr/bin/python is not just the common shebang (and we should be actively transitioning to /usr/bin/python2 for that), but it’s also the command line UI for people wanting to learn Python, or just wanting to get an interactive interpreter to play with. There will be a day when, if they get a command not found when typing `python` at a shell prompt, they will wonder why Debian is broken. I think the transition should roughly be: * Update policy and tooling so that any scripts that require Python 2 use /usr/bin/python2 in their shebang. * Educate our users that they should be using `python2` for anything that has not been ported. * Port as much as possible to Python 3 (eventually, everything maintained in Debian) and /usr/bin/python3 in their shebang. * Plan for a date at which /usr/bin/python will point to Python 3. I know that’s the most controversial bit, but I do think that as time goes on and we’re past 2020, it will be the choice that gives our users the best experience. The discussion on linux-sig is a way to align the entire Python Linux ecosystem to the eventual goal, giving individual distros the leeway they need to time such transitions as they see fit to best serve their users. I’m hoping more Debian Pythonistas will join that discussion, otherwise the outcome will be heavily weighted toward other Linux distros and Debian may find itself without a voice in the matter. Cheers, -Barry signature.asc Description: Message signed with OpenPGP
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:43 AM, Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. Thanks for raising this issue Matthias. There are a couple of other points I'd like to make. This isn't /just/ a problem for Python experts or developers. Sometimes we recommend that Ordinary Users install Python from source, say to use an application that isn't supported by a distro's packaging system. There can be lots of reasons for this: maybe the app is in beta testing, maybe they want to try things out without affecting the system version, etc. Python developers may not be Debian experts and may not know about this system peculiarity. So when we tell people "just do configure; make; make install" we can actually do inadvertent damage to their Debian system. It also makes Debian the odd man out. Instructions we publish for every other *nix have to have a caveat or FAQ for Debian's (and derivatives) difference. These can go unnoticed until things break, and then we can get difficult to debug problem reports. An experienced helper will be conditioned to first ask "Are you on Debian or Ubuntu? Well, you have to do things differently there." This feature is difficult to discover, and highly surprising to any Python person, especially more so if they are not Debian experts. When I first hit this problem (and complained to Matthias ;), it took me several hours to figure out what was going on, where the hack was being inserted, and that it was actually intentional. A more fundamental question though is, why do this in the first place? What's the use case? What convenience are you providing users, and if that's overwhelmingly beneficial, can it be done in a way that doesn't go counter to the conventions of the upstream project? What confuses me is that this doesn't appear to be a convenience based on permissions, because on my Ubuntu box it takes root to install things in /usr/local just as it would to install things in /usr. Meaning, if I'm going to install an egg in either location, either manually or via easy_install, I still need to be root. Although this doesn't affect things /now/ I think Python 2.6 will allow for user- specific sys.paths so that you could for example install an egg in ~/.local/lib/python2.6 and have it just show up. I'm also uncomfortable with the way that Debian accomplishes this: it hacks Python's standard site.py. There really should be a better way to do this (I have to think about this a bit though). I would opt for Matthias's second option, and would be happy if there was a roadmap for fixing this problem. - -Barry P.S. System applications written in Python should /never/ use "#! /usr/ bin/env python". They should always hardcode /usr/bin/pythonX.Y. Otherwise you can break your system just by tweaking your $PATH. You should have to work harder to break your system . This is why setuptools rewrites the #! line when it installs scripts in packages. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR9aM8XEjvBPtnXfVAQKpBwP+OBvovf+MZsHTsU5N42TKqqHXi+IQwHKp rnCtqkBju+jYAWmJpE5BWHmYkBIyc9Me4N7tPMM04YYKwipub5tLEa1CUaHmtyWp 5sEHK2rlOJoluBvJy/S2NBgkSExw2RHmgkn5ZUb8IlLHO9HFh67Egc6vWG6Bg6xq GScfgB1bEfQ= =yk2u -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:58 AM, Wichert Akkerman wrote: - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. That just means you will break people's python at a later point again. That certainly can not be the right thing to do. Depends on how you look at it I guess. I see it as eventually fixing everyone's Python ;). Iff we really must come up with a way to make it a bit easier for people to install their own python version make it a system-wide switch in /etc/default/python2.x and keep the default behaviour as it is now so you will not break existing installations. I'm not sure what the "it" is you want to make a system-wide switch. Would that be which Python to use, or whether to add the sys.path hack? I don't know who's going to be at Pycon later this week, but if enough distro people are around, I'd be very happy to have a discussion about how to make this more consistent across distros, more amenable to users and developers, and how to make things less mysterious, confusing, or fragile. Given how many system tools are being built in Python now (a good thing!), there should be some agreement on the right way to do it. I'll be at Pycon and would be willing to chat about this at a BOF, over lunch, or over beers. :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR9aPNHEjvBPtnXfVAQJCzgP/QaadzyGxnvA6WqT8B3QsouSVvZvsmg9n LAOnoLYzZOhcLzSUlYiukPcU6DMX8HAA8AaAvwpiyCLmk6IZDGMJxDZH+R8u4+w7 9E4KRdt3yxa5nbcJAMa/fWzjpDTlXSecZBAHw7IAYngy55+fv7cJ7UvWgNqsPdIB h4mUoEU32CU= =I8kv -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 6:36 PM, Floris Bruynooghe wrote: On Tue, Mar 11, 2008 at 09:45:21AM -0400, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:43 AM, Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site- packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. Thanks for raising this issue Matthias. There are a couple of other points I'd like to make. This isn't /just/ a problem for Python experts or developers. Sometimes we recommend that Ordinary Users install Python from source, say to use an application that isn't supported by a distro's packaging system. There can be lots of reasons for this: maybe the app is in beta testing, maybe they want to try things out without affecting the system version, etc. Python developers may not be Debian experts and may not know about this system peculiarity. So when we tell people "just do configure; make; make install" we can actually do inadvertent damage to their Debian system. Problem is that both are very natural, having local packages being picked up by the system python in /usr/local/lib/python2.X/site-packages as well as being able to install a new python with a prefix of /usr/local. I asked quite a few Python people about this at Pycon 2008, and I raised it in a Python packaging session with Matthias and a few Fedora people. I actually think this is natural only for Debian users, and not the wider Python or Linux crowd. The Fedora people don't do this, nor does Gentoo AFAICT. It surprised quite a few Python folks I spoke with, even some that use Debian or Ubuntu. Given this choice I'd prefer the second option as well, it seems more natural then the first and an upgrade path is important (Wichert's argument about the first having problems with su is also important). It would be good if the distutils could be changed to install local packages there by default then, so as to provide the least surprises: just "python setup.py install" as root would do the correct thing. I agree that for Debian and Ubuntu, where this tradition is firmly ingrained, the second option makes the most sense. It lets you have the functionality you want without breaking the expectations of the wider Python community. Everyone can live together peacefully! :) I'm not sure your other suggestion make sense from a default distutils perspective. The reason is that distutils doesn't see the world that way, and it really doesn't know anything about the rest of the file system, other than the paths it gets from Python or that you tell it about. So having it install in Python's own site-packages by default still makes sense, but it should be easily configurable to do otherwise, either through command line switch or other mechanism. That's the case today though :). When looking at the grand scale of things the change for the system Python to look for local packages in /usr/local/python/lib2.X/site-packages (or /usr/local/python/site-packages2.X?[1]) is probably best done upstream so it is on all systems the same. The build system could do this when a prefix of /usr is passed to configure. I don't think it makes sense to get this into upstream because this is really a distro question. What about MacPorts installing into /opt/ local? What about if I install from source into ~barry/development/ python2.6? None of those should have anything to do with /usr/local. What confuses me is that this doesn't appear to be a convenience based on permissions, It has nothing to do with permissions. The sysadmin should never install things into /usr/ directly, /usr/local/ is their playground. This is Debian policy (which is fine), but I don't think all distros agree. I'm not a distro guy though. :) Mattias, didn't the Fedora guys say they were going to try to create a mailing list for discussing these kinds of cross-distro issues? I'm also uncomfortable with the way that Debian accomplishes this: it hacks Python's standard site.py. There really should be a better way to do this (
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 26, 2008, at 6:23 AM, Emilio Pozuelo Monfort wrote: Barry Warsaw wrote: This is Debian policy (which is fine), but I don't think all distros agree. I'm not a distro guy though. :) Mattias, didn't the Fedora guys say they were going to try to create a mailing list for discussing these kinds of cross-distro issues? There's http://lists.freedesktop.org/mailman/listinfo/distributions, you may want to raise this issue there. Thanks for the pointer. I'll keep an eye on this, but I don't have time for yet another mailing list. ;) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBR+pKpnEjvBPtnXfVAQI+2wQAiR+DmSgWzuWxFAw9T2TiztGouquKsqSC tD+CNuu7gF8HsjDKxdWltU2rIpxkrSVhPZSQPp5L3/4yL3LHAtANdKv5AyPQOAqz t7pSenmuyLdwcyuIzRQQRi3MrB9OSgnjfNHnBet6jw7H9o0RM+5pbakgo26xVIqt 2pyyQhDibLg= =NVYT -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: new dh_python proposal
I know I'm coming in the middle of this, and I'm sure I don't understand all the issues yet, but I'll chime in here anyway. :) On Jan 19, 2010, at 10:39 AM, Josselin Mouette wrote: >I’m also still wondering which exact problem this proposal is trying to >fix. Currently the only thing it does seem to address is a very specific >kind of issue that Loïc explained to me, which happens in Python-based >package managers: when you upgrade the Python packages the package >manager depends on, including a change in the default Python version (if >the default Python version doesn’t change, all modules remain available >during the upgrade, with both -central and -support), and the upgrade >process is interrupted, the package manager might not be able to >restart. There are other ways to fix this bug; the most obvious way >being to apply something similar to your proposal, but only to a very >small set of Python packages, namely those on which the package managers >depend. For the rest of the archive, I just don’t see the point. How many applications are affected by this kind of upgrade breakage? Have you considered cx_Freeze'ing them so they aren't subject to Python package breakage? cx_Freeze[1] is a tool that essentially creates a small binary executable, and zips all the dependent modules onto the end of the file. So the executable is actually a zip file, which Python can import from. The idea is that something like update-manager could be frozen so that it can run even if the modules it depends on have been uninstalled or otherwise broken. From what I understand, this doesn't solve all the problems in pycentral/pysupport, but it could solve an important subset of the problems being experienced during upgrades. -Barry [1] http://cx-freeze.sourceforge.net/ signature.asc Description: PGP signature
Re: Ideal directory structure?
On Jan 29, 2010, at 11:42 PM, Ben Finney wrote: > Umang writes: > >> What I am talking about is the upstream tarball that, when extracted >> into a directory, can run without being installed using distutils. > > If the package is designed to be installed by distutils, it seems obtuse > to expect it to work without using distutils. The point of distutils is > to provide a build process for preparing the package for use; before > that, it's not ready for use. Why not use something like virtualenv or buildout to create your development environment? Mailman 3 uses buildout, it's easy to set up and works great. -Barry -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Re: __file__ is a disease
On Jan 30, 2010, at 4:56 PM, Josselin Mouette wrote: > Maybe you don’t understand it is a plague, because you are not trying to > package the things you write with __file__ for a distribution. The > location of module files on the system should be a hidden implementation > detail. Because of __file__, it is not and this implementation detail > has to be exposed by packaging tools, restricting what they can do in > ways you don’t imagine, making it impossible to just abstract them > behind the “module” concept - which is the only one that should matter > for a programmer. Do you feel the same way about pkg_resources? > Going back to the topic, please try using autoconf, waf or even cmake to > distribute your modules. These tools were designed to abstract things > like filesystem locations and to generate everything needed at > installation time. Python-specific tools like setuptools are not able to > do that, not unless you bundle specific scripts with your packages. I've never used waf or cmake, but tools like autoconf and scons are not trivial systems to use, modify or debug. I don't think that they make life easier for a robust open source project. -Barry -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Re: __file__ is a disease
On Jan 31, 2010, at 01:09 AM, Josselin Mouette wrote: >pkg_resources allows to do things correctly, but I find it too complex; >it’s no wonder why people still hack stuff around __file__: it’s much >simpler. I think it's not so much that pkg_resources is complex, but that it has a lot of extra API that most people won't need. Hopefully the distutils-sig will distill the essential pieces of that and push it into the stdlib. I was trying to see if you objected to the specific explicit use of __file__ or the concept of co-locating data files with the source code, since that violates the FHS, but isn't really changed by use of pkg_resources. The advantage of using an API is that it can abstract away changes in policy or practice. >Basically, I’d say it tries to address the problem at the wrong level: >build/installation issues should be addressed by the build system, not >by the code. Except that this is where build/installation and code requirements intersect. The code at run time needs certain data files. It doesn't really care where they are, but it has to guarantee that it can find them. The installation system may care deeply where such files go so it has to communicate this to the code at run-time. Using an abstraction like pkg_resources helps bridge that gap. >> I've never used waf or cmake, but tools like autoconf and scons are >> not trivial systems to use, modify or debug. I don't think that they >> make life easier for a robust open source project. > >I beg to disagree. For a simple project, the autotools use files just as >light and simple as setuptools, and if you look at a few sample Python >projects using them, you will see the amount and the complexity of build >scripts is similar. Maybe. IME though, many folks who are very comfortable contributing Python code to a project run away in abject fear when asked to update autotools files. IMO, they're a necessary evil for complex C or C++ projects, but way too heavy for simple Python libraries or applications. >And as soon as you are not dealing only with pure Python code, the >difference becomes dramatic. For example the absence of pkg-config >support in setuptools makes it require specific code for each and every >dependency - except those that provide dependency information in the >Python-specific egg-info format, ha, ha. I guess when you're talking about more complicated projects, with mixes of various types of build artifacts, then the build system will necessarily have to be more complex. >There are huge projects that use the autotools without needing to jump >through hoops. A simple look at NumPy and its 1+ lines of specific >Python code just to be able to build the C pieces should be enough to >convince anyone to use a better tool. Maybe so, but that seems like an extreme example for the Python universe of stuff. Most libraries don't need anything nearly as complicated. >I’m not implying the autotools don’t have flaws: the insanity of libtool >bugs, the arcanes of m4 and the gazillions of broken m4 scripts that lie >around… But these flaws are about features that don’t even exist in >Python-specific tools. For most Python libraries and applications, they're YAGNI anyway. If there are specific problems with Python tools for the majority of straightforward Python libraries, let's fix those tools. But I think they mostly work, and where they don't, the distutils-sig is paving the way of the future. -Barry signature.asc Description: PGP signature
Re: Ideal directory structure?
On Jan 31, 2010, at 08:53 PM, Umang wrote:
>But I'll never find my data files in sys.prefix +
>path_i_asked_distutils_to_install_to. So what according to you, is the
>right way to go about distributing? This situation is definitely not ideal.
The Pythonic way (IMO) is to use pkg_resources:
http://peak.telecommunity.com/DevCenter/PkgResources
and especially the resource_string() and resource_stream() APIs. This means
if your data file is under mypkg.mydata (yes, you'll have an __init__.py in
your data directory), you can access your data file by either:
mydata = pkg_resources.resource_string('mypkg.mydata', 'foo.dat')
or
mydata_fileobj = pkg_resources.resource_stream('mypkg.mydata', 'foo.dat')
The module contains a lot of extra stuff that I find is rarely if ever used,
but might make for an interesting read. :)
Hope that helps,
-Barry
signature.asc
Description: PGP signature
Re: python2.6 vs python-json
On Feb 14, 2010, at 09:18 AM, Paul Wise wrote: >Unless they can be convinced to undo that, python-json is clearly at a Very unlikely that will happen. -Barry signature.asc Description: PGP signature
Re: Ensuring stable APIs for consumer applications
On Mar 15, 2010, at 08:47 PM, Jonathan Wiltshire wrote: >It occurs to me that there's very little guarantee of a stable API in >Python modules - less so in C modules, because upstream tends to be more >aware that this is a library, but in pure Python modules. There is nothing other than convention to prevent this from happening in pure Python modules. For those that are distributed in the standard library, I think you can be more comfortable with stable APIs. We take great pains not to change APIs between patch releases, and have policies for backward compatibility and deprecations between major releases of Python. This is not to say that we don't break things accidentally, but it should be considered a bug when it happens counter to Python project policy. >In lib* libraries, we (usually) have SONAMEs which provide some kind of >contract between the .so and the consuming application. Is there an >equivalent practice in pure Python modules that is implemented widely enough >to be useful? For third party libraries, well, to each its own. ;) You /hope/ that upstreams would use common practice of bumping major version numbers when they do this, and the best you can do is use packaging hints to decide which versions of the package you want. This is some support for this in tools like buildout, but that probably doesn't help you much. In Python 2.7 and 3.2, things will get better via the suite of distutils PEPs being adopted, but only in that you should at least be able to declare and manage versions in a more principled way. It will still be up to the upstream maintainers to do the right thing though. >The question came about because the upstream author of RedNotebook ships a >number of convenience copies, sometimes heavily modified, because he is >nervous about APIs changing suddenly and wasting his work. It's likely that >there will be more of these in the near future. Is there anything with which >I can reassure him and reduce the amount of duplicated code in this >application? What has he actually been burned by? Third party Libraries, standard libraries? I think in general, this is not a good practice and can make things more difficult in the long run. But it helps to know specifics. -Barry signature.asc Description: PGP signature
Re: interpreter / stdlib split
On Mar 29, 2010, at 08:39 AM, C.J. Adams-Collier wrote: >I'm pretty new to packaging python. Apologies in advance if I'm making >bad assumptions. I recently packaged IronPython. It can optionally use >the libraries from the standard python library. It doesn't seem that >these libs are distributed separately from the cpython interpreter. >Would it be worth the effort to split the interpreter and the standard >library into separate packages? I don't know whether jython is capable >of loading the library, but if it is, it seems reasonable to split >things up a bit. There is ongoing discussions in upstream Python-land about doing just this, for similar reasons. It seems the work for this has been stalled on upstream Python switching to Mercurial (which likely won't happen until after Python 2.7 is released this summer). The plan though is to split the standard library out of the core CPython repository so that it can be better shared among the alternative implementations. I don't know whether or how that might inform such a packaging split, but it's something to keep in mind. -Barry signature.asc Description: PGP signature
Re: How to properly provide packages for python3.x
On Apr 14, 2010, at 04:14 PM, Jakub Wilk wrote:
>python-central seems to have some (at least rudimentary) support for
>3.X.
>
>I was told that python-support does not support 3.X (see bug #573560).
I am hoping that none will be necessary. PEP 3147 is very close to acceptance
and merging into the py3k upstream trunk, perhaps as soon as this Friday. Of
course that means that symlink farms will no longer be necessary starting with
Python 3.2, but I would happily back port the essential parts of the patch to
3.1 or even 2.6 and 2.7 if folks here want it, and think it will help. I do
not think it would be difficult, and if it means simplifying Python
installations on Debian, I think it would be a big win.
(For backward compatibility, Pythons < 3.2 would need a -Xcachedir flag to
enable PEP 3147 style pyc files, as specified in the PEP. This means it would
be enabled by default only for system installed Python packages, but that
would still allow us to get rid of the symlink farm for apt packages.)
>>- how to do that properly? simply build with python3.x and then ship
>>the /usr/lib/python3.x/ dir?
>
>s|python3.x|python3.x/dist-packages|
>
>However, that would be wrong for pure-Python modules. If root imports
>such a module, it will be byte-compiled, but then *.pyc won't be removed
>on purge.
>
>On the other hand, almost the same can be said about modules for 2.X
>(even when managed by python-{central,support}): if root imports one
>with -O, .pyo will be created but never removed.
Isn't compileall used to create the pyc files? I think all pyc and pyo files
should be placed on the system by the time the package is installed.
-Barry
signature.asc
Description: PGP signature
Re: How to properly provide packages for python3.x
On Apr 14, 2010, at 07:01 PM, Josselin Mouette wrote: >Le mercredi 14 avril 2010 à 10:29 -0400, Barry Warsaw a écrit : >> >I was told that python-support does not support 3.X (see bug #573560). >> >> I am hoping that none will be necessary. PEP 3147 is very close to >> acceptance >> and merging into the py3k upstream trunk, perhaps as soon as this Friday. Of >> course that means that symlink farms will no longer be necessary starting >> with >> Python 3.2, but I would happily back port the essential parts of the patch to >> 3.1 or even 2.6 and 2.7 if folks here want it, and think it will help. I do >> not think it would be difficult, and if it means simplifying Python >> installations on Debian, I think it would be a big win. > >Even with PEP 3147, a helper is still necessary, at the very least to >generate dependencies. > >Furthermore - unless I’m missing something - this PEP doesn’t say >anything about supported Python versions by a given module. So even >without a symlink farm, there is at least some magic needed to indicate >which versions are supported, maybe in the .py files themselves. You're right about that. So a helper may still be necessary, but hopefully simpler. Would it make more sense to adapt python-support or python-central or to create a third tool (like we need yet another one :). -Barry signature.asc Description: PGP signature
Re: How to properly provide packages for python3.x
On Apr 14, 2010, at 09:26 PM, Piotr Ożarowski wrote: >[4] almost ready in pycompile, still waiting for >http://www.python.org/dev/peps/pep-0382/ (and 3147), though PEPs 382 and 384 are next on my list. -Barry signature.asc Description: PGP signature
Is it worth back porting PEP 3147 to Python < 3.2?
Apologies for the cross-post, but I want to make sure that everyone who cares about Python on both Debian and Ubuntu gets a chance to weigh in. On Friday, Guido approved and I landed the implementation of PEP 3147 on the py3k trunk (what will be Python 3.2). This allows multiple versions of Python to coexist on the same system without the need for symlinks to handle pyc file incompatibility. http://www.python.org/dev/peps/pep-3147/ This will be officially supported in Python starting with 3.2. It will not however be available in upstream Python 2.7. The PEP does recognize however that distros may want to back port the feature to get its advantages now. Although I have not yet tried to do so, I think the essential elements of the PEP should be fairly easy to back port to Python 2.6, 2.7 and 3.1. The question is, should we do this? As the PEP outlines, my preference would be to back port it but *not* enable this by default in any Python < 3.2. Instead, the back port would add a -Xenablecachedir switch, and an associated $PYTHONENABLECACHEDIR environment variable that would have to be used to enable PEP 3147 pyc paths. Of course, this would have to be supported in python-support and python-central too, but I believe Piotr has started working on this. With the back port it means that any Python modules installed by apt will not need symlinks in order to share both the source and pyc locations. Normal Python developers working on their own code however would still get traditional pyc file locations by default (but of course, there would be nothing stopping them from setting the above switch/envar). It's possible that Debian and Ubuntu will come to different decisions about what to do, but it would be nice if we could reach consensus. I would be willing to do the work to back port the essential bits to earlier Pythons, or at least spend some time to see whether it is feasible. I know that given the release schedule of Python 2.7, it's possible that Ubuntu 10.10 could include 2.6, 2.7, and 3.1, though I don't think that's been decided yet. I'm not sure how Python 2.7's scheduled release date of 2010-06-26 matches Debian's release schedule. (Given that lenny was released on 2010-01-30 I'm guessing it will be a while.) Before I start working on this though, I'd like to get your opinion as to whether it is worth it. Thanks, -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 20, 2010, at 06:50 AM, Omer Zak wrote: >My take of the situation: >Yes, please backport PEP 3147 to at least Python 2.7. >The rationale: we'll need to support both Python 2.x and Python 3.x for >several years, and it will be nice if the same library package can be >made to support both 2.x and 3.x. > >It would also be nice to define a way to specify that a tree of Python >2.7 scripts are to be compiled into Python 3.x bytecode, by specifying >that the compilation process has first to run 2to3 >(http://docs.python.org/py3k/library/2to3.html#to3-reference) on the >Python scripts and actually compile the 2to3's output. If we can come up with a reasonable way to make that work, then I'd be in favor of back porting the PEP to Python 2.x. Has anybody run a systematic test to see which of the Python packages that are available in Debian work in Python 3 with an automatic 2to3 conversion? Probably the trickiest part will be working out the dependences (i.e. if a package you're dependent on isn't 2to3'able, then you aren't either). I do think that if we back port to 2.7 we should also back port to 2.6. While I don't yet know what the default Python version will be for Ubuntu 10.10, I can certainly imagine it might be 2.6, with 2.7 available but not default. If so, then it definitely makes sense to back port to both. If 10.10 includes only Python 2.7, then sure, we'll only back port to that version. I guess the same kind of systematic test for package compatibility with Python 3 would be useful to run for compatibility with Python 2.7. Does such a thing exist? -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 20, 2010, at 10:14 AM, Piotr Ożarowski wrote: >[Omer Zak, 2010-04-20] >> My take of the situation: >> Yes, please backport PEP 3147 to at least Python 2.7. >> The rationale: we'll need to support both Python 2.x and Python 3.x for >> several years, and it will be nice if the same library package can be >> made to support both 2.x and 3.x. > >you cannot (in most cases) share 2.X and 3.X Python code, so adding it >to 2.7 and not to 2.6 doesn't make sense Just thinking out loud: How insane would it be to think of an import hook that could do it at run time? Or, a compilall switch that would do it when the pyc file was generated? You'd need some way to specify that a particular module could/should not be auto-converted, or conversely if automatic support were more the exception, some way to say it can be done. I wonder if something in the debian/[control|rules] file could trigger that. >I doubt we'll touch 2.X packages in Debian Squeeze and in Squeeze+1 >we'll most probably have only 1 Python 2.X version (if any) so I don't >see a point in backporting it in Debian. Good to know, thanks. >If you want to ship two 2.X Python versions in Ubuntu, you could use my >new dh_python for that (I hope to finish it soon), but you'd have to >convert *all* packages to it and lets face it, python2.5 and python2.6 >transitions in Ubuntu (at least in universe) were... well not even >close to Debian's quality and these transitions didn't require that much >work... Sorry, I don't really know the history of any of that so I can't comment. But I would like to know more about your new dh_python, what changes it would require, etc. Where can I find it, or information about it? -Barry signature.asc Description: PGP signature
Re: Skip Python 2.6 and use 2.7 as default in Squeeze?
On Apr 21, 2010, at 02:43 PM, Piotr Ożarowski wrote: >> Moreover AFAICT 2.7 is the most compatible-with-the-previous-version >> Python release in the last 16 years > >did you hear about relative imports in 2.7? (enormous transition for us) Are you sure about this? Despite what PEP 328 says, I'm not sure that change actually happened. I can find no reference to this change in Misc/NEWS on trunk, nor in the What's New for Python 2.7, and my own (limited) testing shows that absolute imports have not been enabled by default. I could be missing something of course; I'll double check with python-dev. -Barry signature.asc Description: PGP signature
Re: Skip Python 2.6 and use 2.7 as default in Squeeze?
On Apr 21, 2010, at 08:33 PM, Piotr Ożarowski wrote: >[Piotr Ożarowski, 2010-04-21] >> should be easy to check (by creating 2 simple .py files and running >> them using python2.7 from experimental). I'll check it later tonight. > >It's not enabled (fails only if I import absolute_import from >__future__) Right, thanks for confirming (also confirmed on python-dev). It won't make it into Python 2.7, so I've updated the PEP to remove the sentence stating that absolute imports would be enabled. Yay! One less thing to worry about during 2.6->2.7. :) -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 20, 2010, at 04:57 PM, Scott Kitterman wrote: >I think it is difficult to know for sure what the future will hold. If the >backport is not technically complex, I think a backport with the default to >off would be a nice tool in the box is things go differently than we plan. >Other than the effort involved, I don't see a downside, but I have no idea >how hard this would be to do. It may not be worth porting the entire patch, but I think the core interesting bits won't be too difficult. The one tricky extra thing to implement is switching it on only when -Xenablecachedir is given. In Python 3.2, it's always enabled. We'll see how things go. If Ubuntu 10.10 is going to include multiple versions of Python 2.x, then I think it makes sense. With one 2.x and one 3.x it's not quiet as clear. -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 21, 2010, at 12:01 AM, Piotr Ożarowski wrote: >2to3 is not reliable, at least not for now. I agree that there's no way we can just enable it by default. Too many upstream packages need modifications to work in Python 3. However, for those that are Python 3-ready, it's a useful tool. The question then is whether we want to enable it for those packages that it works with. >Example: even today I backported Python 3 related patch from tip and upstream >*did* test it with 2to3 and python3.X before releasing Python 3.X compatible >version. The second python3-* package I maintain doesn't provide Python >extension for now, as it didn't even build for 3.X version; module (that had >to use custom 2to3 rules, BTW) uses it optionally, though. If it doesn't >always work with modules tested with python3, I'm pretty sure it will not >always work with modules written when python3.X wasn't even released upstream >so runtime conversion is not acceptable, IMHO. > >Every maintainer has to check the module before providing python3-foo >package and that requires manpower Ubuntu doesn't have. That's >why Python transitions in Ubuntu didn't work that well in the past (I >don't recall a single one completed before Debian, and Ubuntu started >with 2.6 few years before us...) How much of the transition testing is automated? It would be very interesting for example, to have a test framework that could run any combination of Python packages against various versions of Python, and get a report on the success or failure of it. This may not be a project for the distros of course - I think upstream Python would be very interested in something like this. For example, a tool that grabbed packages from the Cheeseshop and tested them against different versions would be cool. If snakebite.org ever gets off the ground, that might be the best place to put something like this together (though we'd care less about OSes that aren't Debian and Ubuntu). >> But I would like to know more about your new dh_python, what changes it >> would require, etc. Where can I find it, or information about it? > >it will ship symlinks instead of creating them at runtime (and use >list of exceptions instead of shipping a list of files to compile) in >"old" mode and in PEP3147 mode, paths like: > > /usr/share/python3/foo/__init__.py > /usr/share/python3/foo/bar.py > /usr/share/python3/foo/3.2-/bar.py > /usr/share/python3/3.1-3.5/baz.py Those are the symlinks, right? E.g. /usr/share/python3/foo/3.2-/bar.py would symlink to /usr/shared/pyshared/bar.py? I like the idea of shipping symlinks, but I thought they were created at runtime to handle the case where a new version of Python was installed after the package was installed. Is that the case, and if so, wouldn't all the packages have to be updated when a new version of Python is supported? >using __path__ to choose the right source file >(like in the example[1] I gave you the other day). Note that right now >it requires changing __init__.py file and that's where namespace PEP >comes in (and new .pth files). Cool. >See threads on debian-python mailing list with "dh_python" in Subject >for more info, some things changed since then (and I'm too sleepy right >now to list them :-P), though :). I'll look those up. >What changes it will require? Well, I plan to provide wrappers that will >emulate dh_pycentral/dh_pysupport's API, but as recent >site-packages -> dist-packages transition showed - every package has to >be checked by its maintainer (there are lots of corner cases not easy to >predict like depending on helper tool's internal paths) > >Right now only pycompile[2] and pyclean[3] are available in public. > >[1] http://people.debian.org/~piotr/__path__.tar.gz >[2] http://people.debian.org/~piotr/pycompile >[3] http://people.debian.org/~piotr/pyclean Thanks for the links. I'd like to help in any way I can. -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 22, 2010, at 01:47 PM, Scott Kitterman wrote: >Based on my experience with the Python 2.5 and 2.6 transitions, I'd really >prefer to have Ubuntu follow Debian rather than lead. Ideally, yes I'd like the same thing, but I'll need help because I'm so new to Debian. So if it makes sense for Ubuntu and doesn't harm Debian, then it would be acceptable to patch Python in Debian even if it doesn't help much (because Debian won't have multiple versions of Python 2.x)? -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 21, 2010, at 12:03 AM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-04-20] >> If 10.10 includes >> only Python 2.7, then sure, we'll only back port to that version. > >why do you want to backport it to 2.X for a single python2.x package? It would only make sense if we wanted to share source between multiple versions of Python. So if 10.10 includes only Python 2.7 and we do not think it's feasible or worth it to also share (many) packages with Python 3, then the back port isn't worth it. -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On Apr 26, 2010, at 06:35 PM, Nicolas Chauvat wrote: >Nice to see someone of the core python team taking part in >distribution development. Well, it's official now - I've joined the Ubuntu platform team at Canonical, so I'm very keen on helping to solve problems with Python on Debian and Ubuntu. >On Thu, Apr 22, 2010 at 01:52:11PM -0400, Barry Warsaw wrote: >> How much of the transition testing is automated? It would be very >> interesting >> for example, to have a test framework that could run any combination of >> Python >> packages against various versions of Python, and get a report on the success >> or failure of it. This may not be a project for the distros of course - I >> think upstream Python would be very interested in something like this. For >> example, a tool that grabbed packages from the Cheeseshop and tested them >> against different versions would be cool. If snakebite.org ever gets off the >> ground, that might be the best place to put something like this together >> (though we'd care less about OSes that aren't Debian and Ubuntu). > >Unfortunately, Logilab does not have a much man-power to offer to set >this up at the moment, but would something like >http://apycot.hg-scm.org/ fit your description of a test framework ? That's for continuous integration of Mercurial, right? >We also have it running at logilab.org and cubicweb.org of course: >http://www.logilab.org/view?rql=testconfig&vid=summary >http://www.cubicweb.org/view?rql=testconfig&vid=summary > >As you can see with these second and third links, tests include >lintian and piuparts checks. > >Is it something like this that you had in mind? Yes. What are you using to drive this? I'm not really up on CI tools, but Hudson has been getting a lot of buzz. http://hudson-ci.org/ What I like about your display is that a failure in one area does not necessary mean a failure elsewhere. That way you can better see the overall health of the package. What I have in mind is defining a set of best practices, embodied as much as possible in tools and libraries, that provide carrots to Python developers, so that if they adhere to these best practices, the can get lots of benefits such as nearly automatic and effortless packaging in Debian and Ubuntu. It's things like 'python setup.py test' just working, and it has an impact on PyPI, documentation, release management, etc. These best practices can be opinionated and simple. If they cover only 80% of Python packages, that's fine. Developers would never be forced to adhere to them, but it would be to their advantage to do so. -Barry signature.asc Description: PGP signature
Re: continuous integration/testing for python packages [Was: Is it worth back porting PEP 3147...]
On Apr 27, 2010, at 10:01 AM, Nicolas Chauvat wrote: >[discussion started at >http://lists.debian.org/debian-python/2010/04/msg00046 >should we continue or trim some of the cc'ed lists ?] Is there a better place to discuss Python packaging issues that are of interest to both Debian and Ubuntu? Are all Ubuntu developers who care about shared Python concerns generally members of debian-python? I don't know these things. :) >We are using http://www.logilab.org/project/apycot that is GPL >software mainly developed and maintained by Logilab, but slowly >reaching out to a larger audience. There's also buildbot of course, which I guess it the granddad of Python CI tools, kind of. >It uses a web framework to store the information in a db and provide a >web user interface, plus slave testing bots running on one or more >hosts that get the next task from the queue, execute it and store the >results in the db. Are there things like a API (REST or otherwise) for pulling data out of apycot? >> What I like about your display is that a failure in one area does not >> necessary mean a failure elsewhere. That way you can better see the overall >> health of the package. > >You may find interesting the following blog posts about apycot and >ways to display its information http://bit.ly/9dZQQE > >> as nearly automatic and effortless packaging in Debian and Ubuntu. > >We tried fully automatic packaging of Python programs years (8?) ago >and did not succeed for distutils and setuptools were too far away >from Debian packaging concerns. > >Introducing in mypackage/__pkginfo__.py and mypackage/setup.py all the >information needed to generate the debian/* files without the need to >modify them eventually meant more or less copying their whole content, >for their is actually not much to generate. It also meant using a less >efficient toolchain because of the added conversion step. I'm not familiar with __pkginfo__.py, but I do agree that setup.py makes automated packaging difficult. We need a declarative syntax that can be consumed by more tools, which is why I'm so excited about Tarek's work in distutils-sig. >We moved to having tools that check the consistency of the information >provided by __pkginfo__ and debian/* files and make it easier to build >the Debian packages. These tools are >http://www.logilab.org/project/logilab-devtools Is there a wiki or online documentation documenting these tools, or is it all in the source? >Packaging a piece of Python software now requires a bit of (easy) work >at first, but following releases only need one or two commands. And >all the dh_python* helper scripts reduced that work even further. Is that easy work manual or automated? What does it take to Debianize random-simple-pypi-package? (By that I mean "run a script" or "inspect setup.py and write the debian/*" or "...?". >> What I have in mind is defining a set of best practices, embodied as much as >> possible in tools and libraries, that provide carrots to Python developers, >> so >> that if they adhere to these best practices, the can get lots of benefits >> such >> ... >> It's things like 'python setup.py test' just working, and it has an >> impact on PyPI, documentation, release management, etc. These best >> practices can be opinionated and simple. If they cover only 80% of >> Python packages, that's fine. Developers would never be forced to >> adhere to them, but it would be to their advantage to do so. > >Sounds good to me :) Right now, it's just an idea. There are a few existing attempts at this, so it's worth looking at what exists first. -Barry signature.asc Description: PGP signature
Re: Python2.4 has been removed, now what for Zopistas and Plonistas? [ADDENDUM]
On Apr 27, 2010, at 06:32 PM, Toni Mueller wrote: >I'm not sure about how much the changes in the infrastructure would be >an obstacle, or prevent this from happening (didn't check yet), and I >don'tknow how the group thinks about the political and/or personal >issues involved, either. What I do see is that it will most likely >almost impossible to provide security support for Python 2.4 if >upstream decides to stop providing these (Barry?). Officially, Python 2.4 is EOL. The last release was 2.4.6 on 2008-12-19 and it was a source-only release. Even security patches will only be provided if there's someone motivated enough to do it, which for a release as old as 2.4 is doubtful. -Barry signature.asc Description: PGP signature
Re: Python talks at DebConf
On May 04, 2010, at 04:20 PM, Richard Darst wrote: >I was looking through the talks submitted to DebConf, and noticed >there weren't very many Python related talks. Given the amount there >is to discuss related to Python in Debian, it would be great to see >some more submissions. I wish I could make it. The location is fairly convenient, but the timing just doesn't work with my summer plans. :( -Barry signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On May 05, 2010, at 11:54 AM, Piotr Ożarowski wrote: >What do you think about backporting it to Python 3.1 in Squeeze? > >I think it's worth to backport it to python3.1 even if it will be the >only 3.X version supported in Squeeze as this should ease the transition >to python3.2 in Squeeze+1 and give some time to various tools to adopt >to PEP3147 changes. > >There are not so many packages that support Python 3 for now and Lenny >doesn't have it at all so this transition shouldn't be too problematic. It sounds like a good idea. A few questions: * Would you enable __pycache__ by default? * If so, would you provide a command line option to disable it? The easiest thing of course would be a high fidelity back port of exactly what's in py3k: it's always on and you can't disable it. I bet that would back port fairly easily. It's a bit more work to change that behavior but users of Python 3.1 might be surprised by the difference from upstream (the same questions apply to any back port to Python 2.x). -Barry signature.asc Description: PGP signature
Re: Python talks at DebConf
On May 08, 2010, at 10:55 AM, Piotr Ożarowski wrote: >The only reason I got from Ubuntu for doing transitions outside Debian >and allowing Debian to do it later (and forcing us to fix after them) >is... "because you are slow". All technical reasons (like relative >imports in 2.6) were easy to prove wrong. I'm sure Ubuntu guys are not >crazy and do not break universe on purpose (main is always well tested) >so I'd love to hear their reasons. I know nothing about the history of this so I won't comment on it. Note that today is the first day of the Ubuntu Developer Summit for Ubuntu 10.10. On Thursday we are going to have a session to discuss the roadmap for Python on Ubuntu and what version(s) we will ship by default in 10.10. I invite your constructive input in this thread about issues that you'd like to see discussed. https://wiki.ubuntu.com/FoundationsTeam/Specs/MaverickPythonVersions Cheers, -Barry signature.asc Description: PGP signature
Python versions for Ubuntu 10.10 (Maverick Meerkat)
Now that the Ubuntu Developer Summit is over, I'll be able to catch up on this mailing list. However I wanted to take a moment to summarize the session we had on Python versions for Maverick Meerkat (Ubuntu 10.10). First, let me thank the DPL Stefano Zacchiroli for coming to UDS-M and representing the Debian community. It was really great meeting and having a chance to talk with him. He's encouraged me to apply for DD status and I think I'll take him up on that challenge. :) It's also wonderful to see the commitment he and Ubuntu Community Manager Jono Bacon have made in making our two communities real partners in all this fun and excellent work we're doing. My own personal principle for decisions regarding Python on Ubuntu are that we (Debian and Ubuntu) should always work together and try to reach consensus where possible. We can also recognize that Ubuntu and Debian may ultimately make different decisions, but they should be one of timing rather than substance. What I mean by that is that we can use the basic fact of our different release schedules to our advantage. With Ubuntu's time-based and LTS releases, we can often be more aggressive in our decisions, but always with a mind toward pushing work upstream so that everyone can benefit. I have to admit that I will not always know the best way to do that, and I'll look to you for help. Thanks especially go to Scott Kitterman who knows both communities well, is well versed in Python, and happens to live not far from me. (Time for a BACON PIG anyone? :) So, on to Python... I've started a wiki page which I'll use to collate all the work on Python versions for Ubuntu 10.10: https://wiki.ubuntu.com/MaverickMeerkat/TechnicalOverview/Python In summary, we would really like Maverick to ship Python 2.6, 2.7, and 3.2 (beta) with Python 2.7 as the default. Because 3.2 will not be released upstream by then, we'll get pre-approval for SRUs to get us to Python 3.2 final when available. I think that because there are few libraries and applications that require Python 3, we can be relatively safe with that part of the decision. In order to switch to Python 2.7, we need to do some work first. We need to know how compatible the packages in main and universe are with Python 2.7. So to that end we're going to do several tests, gather data, and make a final decision based on that data. We came up with a requirement of 100% build and import success on Python 2.7 before Maverick feature freeze on August 12, 2010. We're going to start with a PPA on Launchpad to build up the Python 2.7 stack so that everyone can join in with testing and fixing build and import problems. We're also going to investigate ubuntuwire.com to help us out with this if Launchpad's build system isn't up to the task. I think we identified about 1200 packages that need to be rebuilt and Michael Vogt has a system for testing the importability of newly built packages. My plan is to automate as much of the work as possible so that we can re-use what we've built for future Python transition, and I hope that you'll help me ensure that this work is relevant and beneficial to Debian. The key thing to remember is that even if we cannot make it to Python 2.7 for Ubuntu 10.10, none of this work is wasted. We know that we'll all be on Python 2.7 sooner or later, so at the very least this will get us closer for Maverick+1, and lays the groundwork for squeeze+1. Cheers, -Barry signature.asc Description: PGP signature
Re: Python versions for Ubuntu 10.10 (Maverick Meerkat)
On May 18, 2010, at 10:52 AM, Kumar Appaiah wrote: >On Tue, May 18, 2010 at 11:38:23AM -0400, Barry Warsaw wrote: >> First, let me thank the DPL Stefano Zacchiroli for coming to UDS-M and >> representing the Debian community. It was really great meeting and having a >> chance to talk with him. He's encouraged me to apply for DD status and I >> think I'll take him up on that challenge. :) It's also wonderful to see the >> commitment he and Ubuntu Community Manager Jono Bacon have made in making our >> two communities real partners in all this fun and excellent work we're doing. > >I don't know how many Debian Python people are directly involved with >Python upstream, but it would be fantastic to have someone in the >intersection of both sets. Not to exclude anyone, but I know that at least Matthias and myself are both core Python developers. -Barry signature.asc Description: PGP signature
Re: Python versions for Ubuntu 10.10 (Maverick Meerkat)
On May 18, 2010, at 02:30 PM, Scott Kitterman wrote: >As is usual for discussions involving more than one person, we didn't all >leave the room with quite the same understanding. I think 2.7 as default is a >stretch goal at most. We want 2.7 as a supported version, but I didn't sense >a lot of appetite for pushing hard for 2.7 as default. These things happen. :) Still, I'd like to make 2.7 for Maverick the goal, but of course, we'll obviously make that decision only once we've gather enough information. >We did make the assumption that by the time 2.7 is released Debian will be in >freeze. Yep. I heard rumors of a June-ish freeze, but I guess nothing official has yet been decided. >Another important point that I think deserves more emphasis is that we agreed >that the testing done in Ubuntu for 2.7, will also be done against the Debian >archive as well. There was a strong consensus in the group that Ubuntu should >take great care to avoid doing anything that would accrue a technical debt >that has to be paid in Debian (I found this a notable change from similar >discussions in previous release cycles). I completely agree. I've added some emphasis to the previously mentioned wiki page. Thanks, -Barry signature.asc Description: PGP signature
Re: Python versions for Ubuntu 10.10 (Maverick Meerkat)
On May 19, 2010, at 12:42 AM, Sandro Tosi wrote: >Hi Barry, >thanks for your interest for Debian too :) Sure thing! While I don't regularly run Debian I do have the ability to run quite a few VMs on my main development boxes, so it should be pretty easy for me to test things. I just hope you'll be patient with me asking lots of dumb questions. You can always tell me to RTFWP (RTF Web Page :). >On Tue, May 18, 2010 at 23:50, Barry Warsaw wrote: >> On May 18, 2010, at 02:30 PM, Scott Kitterman wrote: >>>We did make the assumption that by the time 2.7 is released Debian will be in >>>freeze. > >Even if it won't be in freeze yet, it will soon to be, so it doesn't >change that much > >> Yep. I heard rumors of a June-ish freeze, but I guess nothing official has >> yet been decided. > >Nothing written in stone, but near june yes. Funny that it's the >python2.6 transition what's currently delaying the freeze.. Bummer. What exactly is the hold up? I do remember seeing those threads on the mailing list, but I didn't quite understand what needs to be done, or the implication that it's a blocker for Debian. >I don't think we can include 2.7 in Debian supported versions for >squeeze, maybe just as a "playground" for squeeze+1. What does it mean to include Python 2.7 as a "playground"? Is there a possibility that squeeze+1 could switch to Python 2.7, if we were able to make the switch in Maverick and push all necessary changes up to Debian? I guess it would be the folks on this mailing list making that decision, right? Is there a wiki page or email thread that describes exactly what you guys had to do for the Python 2.5 -> 2.6 transition, and why it was so painful? IOW, something like a "lessons learned" page? If it's not recorded then it's hard to learn from what you've already been through, and work on ways to improve the situation this time around. >>>Another important point that I think deserves more emphasis is that we agreed >>>that the testing done in Ubuntu for 2.7, will also be done against the Debian >>>archive as well. There was a strong consensus in the group that Ubuntu should >>>take great care to avoid doing anything that would accrue a technical debt >>>that has to be paid in Debian (I found this a notable change from similar >>>discussions in previous release cycles). >> >> I completely agree. I've added some emphasis to the previously mentioned >> wiki >> page. > >Ideally transitions should start in Debian and lend "for free" in >ubuntu, but I understand that for 10.10 we can't make Debian the >environment for 2.7 transition :) Right. It does seem preferable to get new stuff and changes into Debian first, and let them automatically sync to Ubuntu. In fact I'm going to work on getting some of my own Python packages into Debian. However, in this case I think we'll be making lots of changes in Ubuntu first, so it's important to define the process for getting those changes back into Debian. I don't yet fully understand the work involved (your other messages imply it's /a lot/ of work ;), but I'm sure I'll be painfully aware of it as this process continues. I'm willing to be naively ambitious for now because I think the work won't get wasted (since we know Python 2.7 has to come someday), and will eventually benefit both Debian and Ubuntu. -Barry signature.asc Description: PGP signature
Re: Python versions for Ubuntu 10.10 (Maverick Meerkat)
On May 19, 2010, at 11:00 AM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-05-18] >> We can also recognize that Ubuntu and Debian may ultimately >> make different decisions, but they should be one of timing rather than >> substance. What I mean by that is that we can use the basic fact of our >> different release schedules to our advantage. > >sure, as long as Debian is not affected (and Ubuntu doesn't waste too >much of our work) So, how can we make sure that doesn't happen? IOW, how can I begin to experiment with a Python 2.7 transition in a way that will benefit Debian as well? >> With Ubuntu's time-based and >> LTS releases, we can often be more aggressive in our decisions, > >we have experimental for aggressive changes... experimental is overlayed on top of unstable, right? experimental's got Python 2.6.5 right now, but once that lands in squeeze does it make sense to get Python 2.7 into experimental? >> https://wiki.ubuntu.com/MaverickMeerkat/TechnicalOverview/Python > >* how many Ubuntu developers at least read PEP 3147? Probably only a handful. I'd guess that only a handful of upstream Python core devs have read it too though. ;) OTOH, I think most Python developers, and Python packages, will not be affected by PEP 3147 since most don't care about where their pyc files live. Some will of course and we will have to fix them or ask their authors to fix them. Still, I suspect that the intersection of packages that care *and* are ported to Python 3 is probably ridiculously small right now. ;) That would change of course if we backport to Python 2. >* how many of them will work on huge changes that packages need in order > to work with PEP3147? I don't know. What exactly needs to change? Not the packages themselves, but the packaging? How much of that can be automated? >* will Ubuntu force Debian to use these changes (you will *have to* > introduce them in 10.10 if you want to use python3.2 on desktop)? I hope not *force*. I hope we can do a lot of the work and make it available to Debian. >FYI: I've been there, I tried to fix 2.6 related bugs in Ubuntu without >possibility to test them on Debian, it's not fun at all and I will not >do it again (since I got not nice comments and delays in return) Well, if we understand now the types of problems that can come up, then hopefully we can coordinate fixes for them in a much more positive way. I still don't have a good understanding of what - outside of Python 2.7 compatibility itself - needs to change. BTW, if we cannot feasibly do both (provide Python 2.7 and switch to Python 3.2) then I'd rather focus on Python 2.7, since it will be released by Ubuntu 10.10 and 3.2 will not. -Barry signature.asc Description: PGP signature
Re: Python versions for Ubuntu 10.10 (Maverick Meerkat)
On May 19, 2010, at 01:05 PM, Piotr Ożarowski wrote: >[Piotr Ożarowski, 2010-05-19] >> we have experimental for aggressive changes... > >unless you wanted to use Debian experimental, debian-python mailing list >and our help since the beginning and later sync it in Ubuntu (if you >decide it's ready and we will not want to make further changes). >If yes, then I'm all for it. That sounds like a good way to go. If we have success doing it in Debian experimental first, and are able to sync experimental to Ubuntu (for the Python stuff), then I'm all for it too. -Barry signature.asc Description: PGP signature
Re: Python and python3 as separate runtime systems
On May 26, 2010, at 05:43 PM, Scott Kitterman wrote: >I think it makes sense for them to be kept separate at runtime. > >This would mean separate python-foo and python3-foo binaries where both are >supported from the same source. > >It would mean that binaries should not depend on both Python 2 and Python 3 >interpreters. +1 -Barry signature.asc Description: PGP signature
Re: Python versions for Ubuntu 10.10 (Maverick Meerkat)
On May 23, 2010, at 03:58 PM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-05-22] >> So, how can we make sure that doesn't happen? IOW, how can I begin to >> experiment with a Python 2.7 transition in a way that will benefit Debian as >> well?' > >Simply avoid doing transitions with significant changes in either upstream >code¹ or in interpreter packaging² as otherwise the transition will be >made by few overloaded developers who have to update lots of packages really >urgently (just few days ago I noticed yet another "fix" made by Python >core developer this time) or by people who don't use³ Python at all. > >[¹] f.e. changes in API, absolute imports in 2.7 would be one of those, >right now I don't see such changes in 2.7, but I didn't play much >with it yet Contrary to what PEP 328 implies, absolute imports are *not* the default in Python 2.7. Default import semantics will only change in Python 3. >[²] f.e. --prefix or dist- vs. site-packages change I don't foresee any such additional packaging changes of this type for Python 2.7. >[³] without seeing the bigger picture (i.e. not being a long term >maintainer for given package), it's really easy to make a mistake, >even if a package builds fine on all architectures, passes all tests >(with 100% test coverage), is lintian and piuparts clean, module imports >fine, works fine... you still can end with bugs like LP:418280 (and I >can easily imagine breaking lots of other packages by such bugs). >Note that I assume Ubuntu maintainer *did* check all these things >before uploading. Of course, that tells me that there's a bigger problem here. :) IIUC, you're saying the Ubuntu maintainer probably did all the right things and yet stuff was still broken. If that's the case, then it seems like there's no other way to find these breakages short of just trying to make the transition happen and waiting for bug reports. LP #418280 does seem like a process bug: a package that previously was unbundled got added to the stdlib, making the separate package obsolete. The one I can think of for Python 2.7 would be argparse. >> experimental's got >> Python 2.6.5 right now, but once that lands in squeeze does it make sense to >> get Python 2.7 into experimental? > >testing, unstable and experimental have Python 2.6.5 right now, the only >difference is experimental has 2.6 set as default since few days. 2.7 is >in experimental as well, it's not marked as supported, though. Adding >2.7 to the list of supported Python versions will make it a lot easier >to build packages with 2.7 support (f.e. in experimental enabled >pbuilder) and test it in unstable (with 2.7 installed from experimental) What do we need to do to make Python 2.7 supported in experimental? >> >> https://wiki.ubuntu.com/MaverickMeerkat/TechnicalOverview/Python >> > >> >* how many Ubuntu developers at least read PEP 3147? >> >> Probably only a handful. > >sorry, but I don't think it's enough to do the python3.2 transition in >few months (please note that we (I?) were told that that f.e. pyshared >related changes in python-central were tested really well by Ubuntu >developers) While I'd love to be bold and get Python 3.2 in Ubuntu 10.10, if we're time constrained I personally think Python 2.7 is a higher priority, because the latter will have been released upstream by then, but the former will not. It will already be problematic to put 3.2 in Maverick because we'll have to get pre-approval for SRUs to match final release. Note that doesn't mean we still shouldn't begin to do the testing and preparation for Python 3.2. Making it a goal doesn't make it a certainty (for this cycle anyway :). >> I'd guess that only a handful of upstream Python >> core devs have read it too though. ;) > >I think most upstream Python core devs do not try to use two (or more) >different Python versions with hundreds of 3rd party (sometimes really >old) modules and applications, so they probably do not care much about >issues that PEP3147 tries to solve. I don't know. I think many upstreams try to test with multiple versions of Python, but I don't know if they do it simultaneously, how much they rely on the platform's Pythons, etc. I'm also not quite sure what the point of the question is. ;) But you're probably right that most upstreams don't care too much about the central problem that PEP 3147 solves (though *lots* of people seem to want the directory de-cluttering aspects of it). >> OTOH, I think most Python developers, >> and Python packages, will not be affected by PEP 3147 since most don't care >> about wher
PEP 384 (was Re: Bits from dh_python2 author ;-)
On May 16, 2010, at 02:21 PM, Piotr Ożarowski wrote: >What's missing to have full PEP3147 support? >* PEP 384 implementation (will allow us to share (most?) .so files) I have a concern about this. While I understand the motivation, I'm not sure implementing PEP 384 will have any practical help in any kind of reasonable time frame. The reason being, that I think it's highly unlikely existing extension modules will be rewritten to use the ABI, or if they do, it will be a long time coming. Now perhaps I'm wrong, and that having the ABI available in Python 3.2 will make it so that anybody porting their extension module to Python 3 will naturally use it, and it will be a good opportunity to adopt the ABI. But I also think developers are (rightly!) lazy and will do the minimal amount to port their code, which will not include using the ABI. Of course the old adage of "there's more new code unwritten than there is existing code" will eventually come to our rescue, but again, not in any time frame that will be useful for us (maybe when my son is the FLUFL :). Matthias has also suggested getting the ABI version number in the .so filename and making the dynamic loader more intelligent. Kind of like a PEP 3147 for shared libraries. That would still mean we'd need to ship multiple .so's for every supported Python, so we'd lose the disk space/bandwidth advantage, but it might make for less hacky solutions to finding the right .so to load. I don't know. I'm skeptical that PEP 384 is worth the effort, but I'm open to other opinions. -Barry signature.asc Description: PGP signature
Re: Python talks at DebConf
On May 10, 2010, at 01:23 PM, Piotr Ożarowski wrote: >Why I think derivatives should not add new versions? >* because it's mostly chasing numbers - I'm pretty sure there are not > more than 10 packages that require Python >= 2.6 and are not easy to > port to 2.5 in Ubuntu 10.04, >* because when you have to convert hundreds of packages, without > checking them carefully (most packages in Ubuntu don't have maintainer > assigned to them) you end up with "fixes" like: > - disabling tests, > - breaking perfectly valid XS-Python-Version or debian/pyversions, > - hardcoding "-I /usr/include/python2.6" in debian/rules (yes, 2.5 was >still in supported when I saw it) > or no fixes at all (>100 packages that FTBFS, ignoring broken > XS-Python-Version or debian/pyversions, packages that build > correctly, pass all tests... and do not work[1]), >* because new version often means changes in helper tools (cdbs, > debhelper, python-central, python-support) and you're risking the > situation where we will not like your implementation and will rewrite > them in incompatible way (and that will mean you will have to rewrite > them again), >* because we're supporting upgrades from oldstable only (do you know how > many packages in Ubuntu are suffering from missing/too many > Conflicts/Replaces/Provides: pythonX.Y-foo?) (this argument is > actually semi related, as you cannot do much if we will drop support for > one of versions and you still support it in LTS) >* because of crazy ideas like implementing "include-symlinks" in > python-support or using virtualenv in Debian packages as workarounds ;-P These are all really good reasons; thank you for outlining them! Let's assume that when adding a new version requires changes such as you've described above, we have discussed them here and have reached consensus on the approach. It may be true that we disagree on the specific implementations, but let's further assume that our intentions are all good (i.e. that we *want* to work together and forge a common path). The question then becomes, if Ubuntu makes the decision to leap onto a new Python version before Debian does, and we take on the burden of achieving consensus and doing much of the work, how can we ensure that Debian gains the benefit of it in a way that works with your development process? What I mean is, let's say we have to change something in a helper tool. We want that change to happen at least also in Debian, if not first in Debian, but because of our different release cycles, where would we put it? Ubuntu already sync's with Debian, but does it make sense to for example, put changes that are required in Ubuntu first, into Debian experimental? At least that way, Debian will gain the benefit of it when they catch up. If experimental isn't the right place for that, what is? Or is there no place that we can sync changes from Ubuntu to Debian for things like this? -Barry signature.asc Description: PGP signature
Re: Python talks at DebConf
On May 18, 2010, at 11:42 PM, anatoly techtonik wrote: >On Mon, May 10, 2010 at 2:23 PM, Piotr Ożarowski wrote: >> >> Why I think derivatives should not add new versions? >> * because it's mostly chasing numbers - I'm pretty sure there are not >> more than 10 packages that require Python >= 2.6 and are not easy to >> port to 2.5 in Ubuntu 10.04, > >Backporting is a waste of time, Python 2.6 adds new tasty useful >functions that I use for my packages and do not want to support Python >2.5. I'm in a similar situation with my own packages, most of which these days are spun out of my Mailman 3 work. They will only support Python 2.6 and it is a waste of time to try to back port them. >I won't add them to Debian, because: > >1. No Python 2.6 But maybe they can make it to unstable or experimental or very soon now squeeze . >2. I still do not know how >3. Non-Python toolchain is obscure >4. I do not feel like to wasting time learning Debian Policy Tomes >5. I already know virtualenv These are all problems that both Debian and Ubuntu suffer from, IMO. I wouldn't be surprised if just about every Linux distribution suffers from it too. We have huge documents filled with policies, lots of wiki pages, gobs of tools, and enormous amounts of lore in people's heads. It all makes for a very difficult navigation for the newbie. Truth is, Python has this problem too. TOOWTDI is a nice slogan, but not really true in reality. It's just that we know what we know and once that newbie is shined off of us, we forget what we didn't know. ;) >> * because when you have to convert hundreds of packages, without >> checking them carefully (most packages in Ubuntu don't have maintainer >> assigned to them) you end up with "fixes" like: >> - disabling tests, > >Which tests can fail on newer Python versions? I though newer Pythons >are backward compatible, except Py3k. They are *mostly* compatible, but they cannot be fully 100% backward compatible. Modules and features go through a well defined deprecation cycle, __future__ imports get turned on by default, etc, etc. Why just today we had a discussion in #debian-python about string exceptions(!). Those have been deprecated for at least 3.5 years and yet there's still (what Piotr, 100+?) packages in Debian that still use them. (I can understand the confusion about this. Reading PEP 352 would imply that string exceptions will only be deprecated in Python 2 and not removed until Python 3. However, the What's New in Python 2.6, and empirical evidence says that they were removed in Python 2.6. I honestly don't remember that decision being made.) >> - hardcoding "-I /usr/include/python2.6" in debian/rules (yes, 2.5 was >> still in supported when I saw it) >> or no fixes at all (>100 packages that FTBFS, ignoring broken >> XS-Python-Version or debian/pyversions, packages that build >> correctly, pass all tests... and do not work[1]), > >Looks like a major failure of Debian to be a common base for >derivative distributions for Python apps. If there are such problems, I hope that Debian can find ways to make this easier for derivatives. This relates to my previous question about how best to sync work done for Ubuntu for Python 2.7 back into Debian. >> * because new version often means changes in helper tools (cdbs, >> debhelper, python-central, python-support) and you're risking the >> situation where we will not like your implementation and will rewrite >> them in incompatible way (and that will mean you will have to rewrite >> them again), > >That's why helper tools should be Python based and crossplatform, like >the Python itself. And there should be OOWTDI. :) -Barry signature.asc Description: PGP signature
Re: Python talks at DebConf
I know I'm a broken record on this, and I (currently ;) have very little power to do much about it other than *talk*, but... On May 11, 2010, at 10:18 PM, Piotr Ożarowski wrote: >Why am I mentioning Ubuntu at all? Because all decisions about Python in >Debian are made there. My own personal hope is that *decisions* about Python that affect Debian and Ubuntu are made here, even if the implementation, and timing of the availability of those decisions, come in Ubuntu first. I think that's an important distinction to make. These kinds of skew problems stem, IMHO, from the fundamentally different release cycles of upstream Python, Debian, and Ubuntu. Python releases new versions "when ready" but approximately time-based to every 18 months. Debian releases strictly "when ready"[1], and Ubuntu releases every 6 months. These are very difficult differences to reconcile. -Barry [1] Please do not read into this any implied judgement about the different cycles. signature.asc Description: PGP signature
Re: Is it worth back porting PEP 3147 to Python < 3.2?
On May 06, 2010, at 04:47 PM, Lino Mastrodomenico wrote: >2010/5/5 Barry Warsaw : >> users of Python 3.1 might be surprised by the difference from upstream > >It might be useful mentioning somewhere that the best way to detect if >the Python implementation used supports PEP 3147 is: > >import imp >hasattr(imp, 'get_tag') > >(or something similar) and not checking for an hard version number. That's a good point, thanks. I'll add that to the PEP. -Barry signature.asc Description: PGP signature
Re: packages that raise string exceptions
On May 28, 2010, at 01:33 PM, Piotr Ożarowski wrote: >Jakub Wilk found the problem and prepared a list of packages affected: >http://people.debian.org/~jwilk/tmp/string-exceptions.ddlist > >anyone volunteers to check them and report bugs? > >(string exceptions are not allowed in Python >= 2.6) Some of these are surprising (e.g. bzr?! - it has to be Python 2.6 compatible because it comes with Lucid), but they could just be out of date versions. One other thing to keep in mind is that the upstream developers may not be fully aware of the problem. If the exception only gets raised in exceptional circumstances, and those are not tested, then the app or library may not in practice see the failure very often. -Barry signature.asc Description: PGP signature
Re: packages that raise string exceptions
On May 28, 2010, at 04:35 PM, Jakub Wilk wrote: >* Barry Warsaw , 2010-05-28, 10:21: >>>Jakub Wilk found the problem and prepared a list of packages affected: >>>http://people.debian.org/~jwilk/tmp/string-exceptions.ddlist >>> >>>anyone volunteers to check them and report bugs? >>> >>>(string exceptions are not allowed in Python >= 2.6) >> >>Some of these are surprising (e.g. bzr?! - it has to be Python 2.6 compatible >>because it comes with Lucid), but they could just be out of date versions. > >"It works is Ubuntu, so it must be fine" is a horrible fallacy. :) :), but that's not exactly what I'm saying. The fact that bzr works in Lucid on Python 2.6 tells me that something about the analysis is incorrect. Maybe it's that the check was done on an older version of bzr: lenny has 1.5 but squeeze has 2.1. The latter is certainly more modern. Also, because I know that bzr is a very well unittested piece of software, and that the tests are done on Python 2.6 (because that's what's in Ubuntu 10.04), it can't have many if any string exceptions. Sanity check with a core bzr developer (and a quick manual scan of the code) seems to confirm this. So I think we need more details. -Barry signature.asc Description: PGP signature
Re: using experimental as a playground
On May 28, 2010, at 01:24 PM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-05-28] >> What I mean is, let's say we have to change something in a helper tool. We >> want that change to happen at least also in Debian, if not first in Debian, >> but because of our different release cycles, where would we put it? Ubuntu >> already sync's with Debian, but does it make sense to for example, put >> changes >> that are required in Ubuntu first, into Debian experimental? > >... and announce it on proper mailing list (packages from experimental >have different Pin (~priority) by default) Cool, please help me with that. What's the proper mailing list? I don't know much about how to use Debian experimental, but I'm willing to learn! > >> At least that >> way, Debian will gain the benefit of it when they catch up. If experimental >> isn't the right place for that, what is? Or is there no place that we can >> sync changes from Ubuntu to Debian for things like this? > >I think experimental (if unstable cannot be used) is the right place. >If there's something wrong with these changes, bugs will be reported >pretty soon. Cool. -Barry signature.asc Description: PGP signature
Re: packages that raise string exceptions
On May 28, 2010, at 07:21 PM, Jakub Wilk wrote: >Let's try what says grep on the Lucid's vesrion: > >$ grep -Er "raise\s+[\"']" --include='*.py' bzr-2.1.1/ >bzr-2.1.1/bzrlib/bundle/bundle_data.py:raise 'Value %r has >no colon' % info_item >bzr-2.1.1/bzrlib/tests/test_hooks.py:"a ChangeBranchTipParams >object. Hooks should raise " > >(The latter is a false positive, of course.) Nice catch. Filed as LP #586926. -Barry signature.asc Description: PGP signature
DPMT and PAPT request
I realize that I've never formally requested addition to DPMT and PAPT. Since I maintain both upstream Python applications and modules, and since I'm on the core Python development team, I would like to request admission to both Debian teams. I'll start off by maintaining the various packages that I author, independently, as spin-offs from the GNU Mailman project, and various others that I use and hack on. Of course, I'm also very interested in helping to maintain Python itself. For reasons I'm not quite sure of, my Alioth user id is warsaw-guest. I've requested membership through the respective groups on Alioth. Thanks, -Barry signature.asc Description: PGP signature
Re: DPMT and PAPT request
On May 29, 2010, at 11:18 PM, Bernd Zeimetz wrote: >On 05/28/2010 09:10 PM, Barry Warsaw wrote: >> I realize that I've never formally requested addition to DPMT and PAPT. >> Since >> I maintain both upstream Python applications and modules, and since I'm on >> the >> core Python development team, I would like to request admission to both >> Debian >> teams. I'll start off by maintaining the various packages that I author, >> independently, as spin-offs from the GNU Mailman project, and various others >> that I use and hack on. Of course, I'm also very interested in helping to >> maintain Python itself. > >Welcome to the teams, glad to have you on board! Thanks! >> For reasons I'm not quite sure of, my Alioth user id is warsaw-guest. I've >> requested membership through the respective groups on Alioth. > >That will be fixed as soon as you are a Debian Developer. You should make sure >you become one quickly! :) :) I do plan on whipping a few of my own libraries into shape and working my way to DD-ship. >The reason behind the -guest is that the accounts from the debian ldap are >synchronized with alioth, so it is necessary to have a way to distinguish >between DDs and non-DDs, especially to ensure that an account is not used on >alioth already when a new DD requests to have the name. > >Hope that explains it :) It does, and thanks. -Barry signature.asc Description: PGP signature
Re: Possible Mass Bug Filing: String Exceptions Removed in Python 2.6
On Jun 05, 2010, at 10:43 PM, Scott Kitterman wrote: I filed a bug on bzr and it was fixed, though I don't know if it's released yet or not. It was a corner case anyway so probably not a show stopper. -Barry signature.asc Description: PGP signature
Re: Policy for "Specifying Supported Versions" for Python3
On Jun 18, 2010, at 03:45 PM, Piotr Ożarowski wrote: >[Luca Falavigna, 2010-06-18] >> Il 18/06/2010 14.37, Scott Kitterman ha scritto: >> > XS-Python-Version: >= 2.x >> > XS-Python3-Version: >= 3.x > >+1 (but with XB-Python-Version, i.e. no XB-Python3-Version) Agreed. This aligns with the decision to separate the python2 and python3 stacks by essentially treating them as different interpreters. Maybe we should have called Python 3 "Phyton" instead. :) -Barry signature.asc Description: PGP signature
Re: Policy for "Specifying Supported Versions" for Python3
On Jun 18, 2010, at 04:30 PM, Julian Andres Klode wrote: >Another reason against this would be that it will look stupid once there >is no Python 2 anymore. If it's one thing I've learned in all my years is that it's virtually impossible not to look stupid 10 or 15 years later . I mean, have you *seen* those haircuts? Seriously though, make the best decisions for today. Python 2 will be around for a long time. -Barry signature.asc Description: PGP signature
Re: Policy for "Specifying Supported Versions" for Python3
On Jun 20, 2010, at 04:28 PM, Scott Kitterman wrote: >> I'm going to declare rough consensus around this approach and I'll have a >> Python policy patch for review shortly. I haven't had time to read this through yet, but I recently posted some information related to Python on Debian and Ubuntu and requested off-list feedback. One of the more interesting messages I got was from someone who was trying to install a package that has been ported to Python 3, is available on Ubuntu for both Python 2 and 3, and wanted one command to install both binary packages. He was using Synaptic but I don't think that matters. It seems to me that the right way to handle this would be meta-packages that included dependencies on both the Python 2 and Python 3 version of the underlying packages. Maybe we should consider supporting this, and coming up with an agreed-upon naming scheme. E.g. for Python package 'foo', we'd have: * python-foo - the binary package for foo in Python 2 * python3-foo - the binary package for foo in Python 3 * python-2and3-foo - for the meta package that installs both of the above "python-2and3-foo" is probably a crappy naming convention. 1) Is this a good idea? 2) Can you suggest a better name? -Barry signature.asc Description: PGP signature
Re: Policy for "Specifying Supported Versions" for Python3
On Jun 21, 2010, at 06:30 PM, Scott Kitterman wrote: >I think most people install Python modules and extensions as dependencies of >applications they care to use. For Python developers that actually care >about such things, I think it's better that the just install both manually. I agree about the former. I'm just posing the question, I'm not sure I have a strong opinion about the latter. For developers, I guess 'apt-get build-dep' gets close, but it doesn't seem quite right. >If we maintain a standard that if in Python you import foo, then the Python >package name is python-foo and the Python3 package is names python3-foo, I >would think this is manageable. I think that adding this metapackage would >impose a lot of complexity on packagers and/or python helper maintainers, >bloat the Packages.gz file signficantly, and probably provide confusing >search results. > >I'm not sure what the best answer is, but I'm not sure there is one that's >even good. Maybe the answer isn't in adding more package dependencies, but instead in a tool that you could wrap around apt. E.g. if I wanted Python package foo installed for all installed Python versions, I think it wouldn't be too difficult to write a little helper that could map from Python module name to python-foo and python3-foo binary package names, doing the apt-get install for you. Does that sound more reasonable? -Barry signature.asc Description: PGP signature
Re: Policy for "Specifying Supported Versions" for Python3
On Jun 22, 2010, at 12:13 AM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-06-21] >> I haven't had time to read this through yet, but I recently posted some >> information related to Python on Debian and Ubuntu and requested off-list >> feedback. One of the more interesting messages I got was from someone who >> was >> trying to install a package that has been ported to Python 3, is available on >> Ubuntu for both Python 2 and 3, and wanted one command to install both binary >> packages. He was using Synaptic but I don't think that matters. > >you mean like `debi foo_1.2-3_amd64.changes` or >`apt-get install python-foo python3-foo`? I think the OP wanted something more like the latter. -Barry signature.asc Description: PGP signature
versioned .so files (over in python-dev)
For those of you not closely following python-dev, I've made a proposal for versioned .so file naming for Python 3.2. It would allow us to place extension module .so files for different Python versions (or really, Python builds, e.g. 3.2/3.3, debug, UCS2/4, etc) in the same directory without naming collisions. The thread starts here: http://mail.python.org/pipermail/python-dev/2010-June/100998.html Except as it specifically impacts Debian, it's probably best to comment on the proposal over in python-dev. -Barry signature.asc Description: PGP signature
Re: versioned .so files (over in python-dev)
On Jun 24, 2010, at 11:58 PM, Jakub Wilk wrote: >What's the point in making distutils produce versioned .so? Such a change is >going to break lots of packages for exactly zero benefit: Why so? >helper tools will need to do unversioned->versioned renames anyway, in order >to handle non-distutils build systems. Such helper tools will still need to know what versioned names Python will import. Having it in distutils will make the common case easy. It does indicate that external tools will need to know what names a particular Python build supports. It probably shouldn't be hardcoded. -Barry signature.asc Description: PGP signature
Re: Specifying Supported Python Versions - Round 2
On Jun 30, 2010, at 04:58 PM, Scott Kitterman wrote: >On Wednesday, June 30, 2010 04:51:38 pm Piotr Ożarowski wrote: >> [Scott Kitterman, 2010-06-30] >> >> > For Python3: >> > >> > 1. A new field called X-Python3-Version: It does not support lists of >> > versions (e.g. (3.0, 3.1)). Acceptable values are a single version (e.g >> > 3.1), greater than or equal to a version (e.g. >= 3.1), or strictly less >> > than a version (e.g. << 3.2). Versions 2 or less will raise an error. >> > >> > 2. There is no #2. If your build system uses py3versions -r, then you >> > need X-P3-V, if it's not there, an error will be raised. If it doesn't >> > use py3versions -r, then it's between the maintainer and their build >> > system. The field is not mandatory. >> >> why? If py3versions is invoked in debian/rules, then there definitely is >> at least one python3-* binary package. Why do you want to make this >> field required? I'd make it optional and assume all 3.X versions if >> X-P3-V is not set. > >I'm trying to minimize the amount of implicit magic we do. > >There was strong consensus for Python 2 that "all" wasn't a great idea for XS- >P-V and so we point people away from it. No implicit all seem the logical >next step for Python 3 were we have more freedom to make things work the way >we want. > >If there's some consensus for an implicit "all" in Python 3, I won't object. The question that comes to mind is why Python 2 and Python 3 would have different rules? If there's a good reason (e.g. it cleans up a backward compatibility wart), then cool. If not, then consistency between the two might make a reasonable default. -Barry signature.asc Description: PGP signature
Re: Specifying Supported Python Versions - Round 2
On Jun 30, 2010, at 06:02 PM, Scott Kitterman wrote: >I don't want to break a lot of packages. Both implicit and explicit >all are widely used in Python packages. I think we have more freedom >to do the "right" thing for Python 3. That's what I expected (i.e. "cleans up a wart"). +1 Thanks, -Barry signature.asc Description: PGP signature
Documenting Python Debuntuisms
We had a report in upstream Python from a user who was trying to find information about dist-packages. He did a Google search and didn't find any definitive official explanation of this Debuntuism. His suggestion was to add a note to the official Python documentation, but that doesn't seem quite right to me. I think it would be better to have official documentation on wiki.debian.org[1] about issues of interest to Python developers on Debian and Ubuntu. This would be a user-oriented set of pages, not a developer set of pages. Things like dist-packages and probably that python-setuptools gives you distribute would be two things to start with. I'm sure there are others. So: do you think this is a good idea? If so, where should this information go? I'm not sure Teams/PythonAppsPackagingTeam or Teams/PythonModulesTeam are really the best place for this. Suggestions? -Barry [1] I'd probably add a page to wiki.ubuntu.com, mostly pointing it at the wiki.debian.org page, but filling in anything where Ubuntu deviates from Debian. Not sure yet where that would go either... signature.asc Description: PGP signature
Re: Documenting Python Debuntuisms
On Jul 13, 2010, at 10:43 AM, Asheesh Laroia wrote: >On Mon, 12 Jul 2010, Barry Warsaw wrote: > >> We had a report in upstream Python from a user who was trying to find >> information about dist-packages. He did a Google search and didn't >> find any definitive official explanation of this Debuntuism. His >> suggestion was to add a note to the official Python documentation, >> but that doesn't seem quite right to me. >> >> I think it would be better to have official documentation on >> wiki.debian.org[1] about issues of interest to Python developers on >> Debian and Ubuntu. This would be a user-oriented set of pages, not >> a developer set of pages. Things like dist-packages and probably >> that python-setuptools gives you distribute would be two things to >> start with. I'm sure there are others. >> >> So: do you think this is a good idea? If so, where should this >> information go? I'm not sure Teams/PythonAppsPackagingTeam or >> Teams/PythonModulesTeam are really the best place for this. > >As a random bystander, yes, I think a page on wiki.d.o is great idea. >I think that if you're not sure where to put the page, just live the >wiki dream and put it in any random place. And then once the page gets >traffic and content people will figure out the *right* place for it. I like the JFDI attitude Asheesh! I've done a quick pass through * http://wiki.debian.org/DebianPython * http://wiki.debian.org/Python to clean things up a little, and to begin to document in the former a few deviations from upstream Python that a Debian user might encounter. At least that puts a stake in the ground. Let the wikiwars begin. -Barry signature.asc Description: PGP signature
Re: Documenting Python Debuntuisms
On Jul 13, 2010, at 11:05 PM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-07-13] >> * http://wiki.debian.org/DebianPython >> * http://wiki.debian.org/Python > >I removed some really old pages with "Python" in the URL Nice! >What do you think about renaming all /DebianPython.* to /Python\1 >so f.e. >http://wiki.debian.org/DebianPython -> http://wiki.debian.org/Python >http://wiki.debian.org/DebianPython/Policy -> >http://wiki.debian.org/Python/Policy (with old pages redirecting to >new ones) +1 -Barry signature.asc Description: PGP signature
Re: Documenting Python Debuntuisms
On Jul 14, 2010, at 10:56 AM, Suno Ano wrote: >Yes, that is a common question people have when they want to know how >exactly things are handled in Debian. It was the same for me so I >created > >http://www.markus-gattol.name/ws/python.html#why_has_debian_dist-packages_directories Hi Suno. Thanks for writing this page; I have a few comments on some of the details. First, when Python searches for a module to import, only sys.path is consulted (modulo other import hooks). It doesn't really treat $PYTHONPATH or the cwd any differently. It does initialize sys.path from $PYTHONPATH and it does (sometimes) include a special marker (the empty string) first on sys.path to indicate the cwd, but once Python's machinery gets going $PYTHONPATH is ignored. Second, my recollection about *why* Debian's Python >= 2.6 uses dist-packages instead of site-packages is different. Long before becoming more active here, I raised a problem (I think on this list, and with Doko) that some folks were having because of conflicts surrounding the FHS. Debian put /usr/local/lib/pythonX.Y/site-packages on the system Python's sys.path because it interprets the FHS to allow/encourage/require system administrators to install non-standard stuff to /usr/local. Generally, I agreed with this, but this conflicted with 15 or so years of established practice in upstream Python. A from-source install of Python by default goes into /usr/local and *also* has /usr/local/lib/pythonX.Y on sys.path. This means that if you install Python from-source, as many Python developers do through the default cmmi build, and system administrators do achieve Python builds isolated from critical system resources, you could break your system Python by installing incompatible packages into what you thought was an isolated environment. Yes, this happened to me :) but I wasn't alone. Because it was unlikely upstream Python was going to change (e.g. to install from source by default into /opt) or that Debian Python policy was going to change, Doko came up with a clever compromise of changing the system Python's /usr/local claim to be dist-packages instead (mirrored to /usr/lib space for convenience and consistency). This seemed like a perfectly workable solution, which I still agree with, despite the mild surprise factor for seasoned non-Debian-versed Python developers. Anyway, I think it would be useful to elaborate on this in the Debian Python wiki, though I won't have time to add this for now. Cheers, -Barry signature.asc Description: PGP signature
Re: Numpy API change?
On Jul 21, 2010, at 04:17 PM, Yaroslav Halchenko wrote: >But assuming that in longer run we agree on how we invoke unittesting for >Python modules we ship[...] I propose this be spelled: 'python setup.py test'. In my copious spare time , I'm working on code, documentation, and infrastructure to make this the preferred way of testing Python modules and applications. You don't *have* to conform, but we'll put out big carrots for you if you do. For example, we could have a health check icon on Cheeseshop for all newly uploaded packages which showed coverage and success of the above command. -Barry signature.asc Description: PGP signature
Re: Numpy API change?
On Jul 22, 2010, at 10:00 AM, Yaroslav Halchenko wrote: > >On Thu, 22 Jul 2010, Barry Warsaw wrote: >> >But assuming that in longer run we agree on how we invoke >> >unittesting for Python modules we ship[...] > >> I propose this be spelled: 'python setup.py test'. > >does setup.py test just do unittesting? or + doctesting, regressions, >examples, ... ? It depends on the package. IOW, this command should run whatever core set of tests the package defines. >doesn't it re-build module first? It builds it, but shouldn't rebuild it. >do you advocate also 'module.test()' may be for testing already >installed module whenever setup.py is no longer there? That's an interesting question. I'd rather see runpy supported so you can do things like 'python -m foo --test', but I haven't thought about it (or experimented) in detail yet. >Altogether setup.py test sounds like a viable solution for testing >within build tree (thus rebuilding the module/extensions); and I did >see some projects to provide similar targets (some times called >differently, e.g. nosetest, although 'test' sound better ). > >But it doesn't provide means for testing of already built/installed >module. Somewhat native approaches for this: > * having module.test, so we could just >python* -c 'import module; module.test()' See above. > but that implies upstream willing to do so or our duty to hack it up > * using >nosetests module > but that is not guaranteed to work. Although nose's discovery is > nice etc, project which is not 'nose aware' might get all kinds of > side-effects while invoking tests this way. > on the other hand >nosetests --with-doctests ... > is very nice, but once again, some projects might not care about > doctests So, I would personally let the upstream decide, and in fact make it their responsibility to support the interface. As I said, I'd put some good carrots under their noses so that they'll *want* to support it, and I'd add whatever is necessary to make it easy (i.e. improve doctest discovery). Of course there's nothing preventing DPMT from adding those hooks. I'm really just proposing a common command line interface that Python modules and applications are encouraged to support. >So, altogether, all those approaches require all upstreams to agree >upon the way things get tested... which I doubt would happen. Well, as I said, if we make opinionated decisions, promote best practices, and provide incentives, many -- and I think most -- will. You're never going to get everyone to do anything, but that's okay, they just won't get the big shiny Happy Icon on Cheeseshop, etc. >To account for heterogeneity among the projects why instead we just >not agree upon few debian/rules targets, e.g. > >debian/rules test-available-* > to run unittest battery (possibly with doctests and > whatelse) with module.test() or > nosetests or whatever upstream's native way. I think that's fine too. What I have in mind is really Debian (or actually distro/platform) independent, so for example if snakebite really ever exists, we could run the tests on all available platforms. Then debian/rules would usually just invoke 'python setup.py test' or 'python -m foo --test', but that could be overridden for the outliers. > Rule might need to 'cd debian' or some other directory to assure > that local source tree is not the one getting tested. > > -* is there for a python version to be used ease test-* below > >debian/rules test-available > to run against all supported python versions > >and, dependent on successful build > >debian/rules test-* > which possibly just adjust the PYTHONPATHS and call corresponding > $(MAKE) test-available-$* >debian/rules test > to test them all > >Then for the validation of the packages which could just use >test-available and test targets within extracted source package Sounds good to me! -Barry signature.asc Description: PGP signature
Re: Documenting Python Debuntuisms
On Jul 15, 2010, at 06:55 PM, Markus Gattol wrote: > Barry> This means that if you install Python from-source, as many > Barry> Python developers do through the default cmmi build, and system > Barry> administrators do achieve Python builds isolated from critical > Barry> system resources, you could break your system Python by > Barry> installing incompatible packages into what you thought was an > Barry> isolated environment. > >You mean because the source install of Python installs into >/usr/local/lib/python3.1/site-packages in addition to /usr/local? It >still does if I recall correctly. Well, from-source installs of Python have a default --prefix of /usr/local, so yes it all goes in /usr/local and /usr/local/lib/pythonX.Y/site-packages is on sys.path. I'm not sure that's exactly what you meant by "in addition to" though. Thanks for the updates Markus. -Barry signature.asc Description: PGP signature
Python 2.7 status on Ubuntu Maverick
Having just returned from a platform rally in Prague, I thought I would update folks on the progress of enabling Python 2.7 as a supported -- but not default -- Python in Ubuntu Maverick (what will be 10.10). I'd also like to take this opportunity to solicit help in getting us to the point where we can make the decision to enable or not enable. Doko has built a toolchain PPA which provides Python 2.7 among other things: https://edge.launchpad.net/~doko/+archive/toolchain/+packages There are currently a few build failures here but Python 2.7 isn't one of them. I think (but for various reasons atm can't check) that this is essentially the Python 2.7 package from Debian experimental. On top of that, I have a PPA with just python-support, python-central, and python-defaults which add Python 2.7 as a supported version: https://launchpad.net/~pythoneers/+archive/toolchain2.7 Finally, I have a PPA that includes all the other packages which build-dep on python-all and python-all-dev: https://launchpad.net/~pythoneers/+archive/py27stack4/+packages As you can see there are a few build failures here, but at least one of them is bogus I think[1]. I'm still evaluating the others and would appreciate any additional input or feedback you might be able to provide. Of course, this only helps with extension module packages, since as you know, any incompatibilities with pure-Python packages can't be determined until install time. One of my next steps is to create a Maverick VM and install as much of the Python rdeps as possible, collecting information on what fails to byte-compile. Here are the steps you can take if you want to experiment with the Python 2.7 stack on a Maverick system: (as sudo): add-apt-repository ppa:doko/toolchain add-apt-repository ppa:pythoneers/toolchain2.7 add-apt-repository ppa:pythoneers/py27stack4 apt-get update apt-get upgrade apt-get install python2.7 Please ping me on irc or email if you have any thoughts on the ftbfs in the py27stack4 PPA. Cheers, -Barry [1] I think the subversion failures are bogus. I built the package in an sbuild environment and it succeeded just fine. The PPA failure looks to me like a test suite failure where a socket could not be opened. The PPA build machines have outgoing networking restrictions and I think this is what we're hitting. I'm not positive, but it seems related. I'm not sure how to work around that short of disabling those tests in a (temporary) patch. signature.asc Description: PGP signature
Re: Python 2.7 status on Ubuntu Maverick
Hi Jakub, Thanks very much for looking at these. I have some dumb questions. :) On Jul 26, 2010, at 10:54 PM, Jakub Wilk wrote: >* Barry Warsaw , 2010-07-26, 15:38: >>https://launchpad.net/~pythoneers/+archive/py27stack4/+packages >> >>As you can see there are a few build failures here, but at least one of them >>is bogus I think[1]. I'm still evaluating the others and would appreciate any >>additional input or feedback you might be able to provide. > >gnome-python-desktop, kdebindings, and twisted-* are depwaits; obviously >nothing to do with Python 2.7. Okay, I see this. Nothing we can do about it now. >libvigraimpex, pygtk, qscintilla2 failed to build because their >build-dependencies didn't provide modules for Python 2.7. I'm not sure I understand what this means, or what if anything we can do about it. Does this mean that some of the packages that these three build-dep on also need to be built for Python 2.7? If so, then how would I go about finding out which ones and adding them to the PPA? One thing I notice is that libvigraimpex and pygtk, but not qscintilla2 build-dep on python-numpy, which as you say below is fixed in numpy 1.4.1 (modulo the "funny" issues ;). I don't know if that's related to the failures here, but I guess we should try to get python-numpy 1.4.1 into the stack (which would fix its ftbfs too). The qscintilla2 failure is mysterious to me, so any light you could shed on that would be appreciated! I'll look at this in more detail too. >I can reproduce FTBFSes of pyao, pygobject, vte by rebuilding Ubuntu >packages in sid environment; therefore these failures are not >python2.7-specific. Also, sid versions of these packages can be built >just fine (in sid). Does that mean the problem is related to the Ubuntu packaging of these packages? pyao: 0.82-2.1ubuntu4 vs 0.82-3 pygobject: 2.21.4-0ubuntu3 vs 2.21.4-1 vte: 1:0.24.3-1ubuntu2 vs 1:0.24.3-1 I looked at the pyao failure and it sure is weird. It's a compilation failure that I can reproduce in my mav sbuild too. I'll look at these in more detail. >Failure of python-numpy is related to Python 2.7. This bug seems to be >fixed in Numpy 1.4.1, so again Debian is not affected. (We have funnier >issues with Numpy instead. ;>) Looks like 1:1.4.1-4 is in unstable so i guess we should request a sync to get that fix? A few other things that Scott, Doko, and I discussed in IRC: * I will setup a PPA and request syncs for universe packages that build-dep on python-all* * Doko might want to enable py2.7 in experimental to get the benefit of testing in Debian * We will make a final decision for Maverick by feature freeze (12-aug) Thanks everyone. -Barry signature.asc Description: PGP signature
Re: Python Testing -- should be there uniformity?
On Jul 27, 2010, at 09:48 PM, Yaroslav Halchenko wrote: >well, both "setup.py test" and "module.test()" sound like reasonable >interfaces to adhere to. Yes, especially if 'python -m module.test' were an available command line interface. The former could be promoted for in-development testing and the latter could be promoted for deployed package testing (i.e. where you wouldn't have a setup.py). Probably the best place to continue discussing this is in the TIP mailing list, except perhaps as it applies specifically to Debian. -Barry signature.asc Description: PGP signature
Testing Python modules (was Re: Numpy API change?)
On Jul 27, 2010, at 05:56 PM, Yaroslav Halchenko wrote: >On Thu, 22 Jul 2010, Barry Warsaw wrote: >> In my copious spare time , I'm working on code, documentation, >> and infrastructure to make this the preferred way of testing Python >> modules and applications. You don't *have* to conform, but we'll >> put out big carrots for you if you do. > >do you have somewhere any documentation available on this >infrastructure so we could point upstreams to follow? I don't at the moment, but I really need to put such a wiki page together. It's just gotten pushed down on the stack. :( >are you recommending nose as the testing framework? what internal >organization of the modules to support it do you recommend... i.e. in a few >modules I saw (and we somewhat adopted in our projects) having I'm currently in favor of unittest2, but nose has some nice features. When unit2's plugin architecture is done, I think that'll probably be the way I want to go. >project/data -- minimalistic data sets (if) necessary for >testing/examples project/testing -- tools >project/tests-- actual unittests which rely on project.testing It's an open question, and I know there are a million different ways people structure their tests! I generally have project/testing for support code, project/testing/data for test data sets, project/tests for unittests and project/docs for doctests. >alternative strategy is to take advantage of nose discovery allows >to keep unittests closer to respective code, so /tests is not needed >any longer... True. I like separating my tests into submodules, and I don't personally like in-docstring doctests, so I'm biased toward those decisions. -Barry signature.asc Description: PGP signature
Python testing (was Re: Numpy API change?)
On Jul 28, 2010, at 12:23 AM, Sandro Tosi wrote: >anyhow, since I'm at it: please don't force ANY testing tool; I kinda >like unittest2, and it's available in python2.7 stdlib, and it's also >backported to 2.4-2.6 (and even packaged for debian), and I don't want >to be forced to use nose for my upstream development. Hi Sandro, This isn't about forcing anything on anyone! This is Python we're talking about so despite the Zen, there will always bee a million ways to do things. :) Really this is about helping build a common meme for simple and common tasks, and giving beginners a clear bright path to start their journey on. There will always be fun things to explore out in the wilderness, but sometimes you just want the simple path, and beginners can be overwhelmed by the choices. Experienced developers will always be able to do whatever they want! -Barry signature.asc Description: PGP signature
Re: Python 2.7 status on Ubuntu Maverick
On Jul 29, 2010, at 03:35 PM, Jakub Wilk wrote: >* Jakub Wilk , 2010-07-28, 18:35: >>looking at sip4-qt3 build log, even the PPA version provides only >>>modules for 2.6, which looks like a bug in sip4-qt3. > >Actually, it's python-defaults in toolchain2.7 PPA, which does not >support python2.7. Yay. Looks like a resync pulled a newer version that blew away my py27 enabled package. This is depressing :( -Barry signature.asc Description: PGP signature
Re: Testing Python modules (was Re: Numpy API change?)
On Jul 30, 2010, at 11:16 AM, Nicolas Chauvat wrote: >On Thu, Jul 29, 2010 at 11:23:05AM -0400, Barry Warsaw wrote: >> True. I like separating my tests into submodules, and I don't >> personally like in-docstring doctests, so I'm biased toward those >> decisions. > >I'd say in-docstring doctests are good at documentation rather than >extensive testing. Well, *doctests* in general should always be considered documentation first (IMHO). They're testable documentation sure, but testing is not their primary function in life. We've had lots of debates among developers about the pros and cons of doctests, but my own personal feeling is that separate-file doctests can be excellent at providing system documentation (that can be the source of generated formats via great tools like Sphinx), and can provide wide - but not total - system coverage. IOW, doctests are documentation and are a complement to traditional Python unittests, not a replacement. I might differ with other developers on where to draw the line between doctests and unittests, but that's all minutia. ;) I don't personally like in-docstring doctests much. I find that more compact and stylized API documentation better in that precious spot (e.g. epydoc style markup). Again, I think there is lots of room for different opinions! Cheers, -Barry signature.asc Description: PGP signature
Re: Will DPMT be ok maintaining a package that could potentially build other language bindings?
On Jul 30, 2010, at 07:55 AM, Clint Byrum wrote: >[bump] > >Anybody? I have little street cred to speak for DPMT, but I would be hesitant about it. I personally have little Ruby or Lua experience and wouldn't know how to fix problems in those bindings. OTOH, I don't have a better suggestion. :( -Barry >On Jul 27, 2010, at 1:37 AM, Clint Byrum wrote: > >> So I recently uploaded python-libgearman into the svn repository, >> but bzed pointed out that it didn't build from the swig bindings, >> and so really wasn't acceptable to upload to Debian. >> >> I agree, and now I have a source package that builds from the >> upstream swig sources, but this begs the question as to whether or >> not DPMT wants to maintain something that, long term, should also >> build lua and ruby bindings. >> >> Here is the branch with the packaging in it: >> >> https://code.launchpad.net/~clint-fewbar/+junk/gearman-interface-packaging >> >> If it seems ok and DPMT doesn't mind having it in the repository >> when we add ruby/lua bindings, then I'll check it in, otherwise I >> will seek more broad sponsorship for the package. >> >> Thanks! >> -Clint >> >> -- >> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org >> with a subject of "unsubscribe". Trouble? Contact >> listmas...@lists.debian.org Archive: >> http://lists.debian.org/35a4ee6e-e98f-426c-88ee-f6c50213a...@ubuntu.com >> > > signature.asc Description: PGP signature
Re: Python 2.7 status on Ubuntu Maverick
On Jul 27, 2010, at 12:33 AM, Jakub Wilk wrote: >In sid there are two packages with python2.7-specific bytecompilation >errors: mgltools-viewerframework and python-jaxml. Both packages do >something like: > >import sys >sys.__debug__ = something > >which is a syntax error in Python 2.7. This one's really interesting. % python2.6 Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> __debug__ = 1 File "", line 1 SyntaxError: can not assign to __debug__ >>> import sys >>> sys.__debug__ = 1 >>> % python2.7 Python 2.7.0+ (release27-maint:83294, Jul 30 2010, 15:49:51) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> __debug__ = 1 File "", line 1 SyntaxError: cannot assign to __debug__ >>> import sys >>> sys.__debug__ = 1 File "", line 1 SyntaxError: cannot assign to __debug__ >>> So assignment to global __debug__ has been a SyntaxError since 2.6 (at least), but it now appears that the parser is more strict. It's not that __debug__ is a reserved word though because assignment to e.g. sys.print gives you a different SyntaxError. I don't see a specific item describing this in the Python 2.7 Misc/NEWS file, nor can I find a bug in the tracker for this issue, so I'm not sure if it's intended behavior or not. I think I'll raise this on python-dev before working on a patch for Debian. -Barry signature.asc Description: PGP signature
Re: Packages whith “except” overwriting builtins
On Aug 04, 2010, at 03:21 PM, Jakub Wilk wrote: >* Paul Wise , 2010-08-04, 08:31: >>> There are pychecker, pyflakes, and pylint in Debian. >>> This specific case raises a warning in pylint, if I'm not mistaken. >> >>Thanks for the info, I've added these package names to the >>DebianMentorsNet wiki page listing feature wishlists for >>mentors.debian.net/lintian-extras. > >Frankly, I find none of these suitable for running automatic checks: >- pychecker doesn't support some syntax constructs (#577070); >- signal/noise ratio of pylint is close to zero; >- pyflakes looks promising, but doesn't even detect the bug we are >currently discussing. None really are, at least not on their own, and not for an automated gatekeeper without heavy hacking. They can be useful tools, but I've never seen any moderately complex bit of Python code ever be completely clean. It would be difficult to get one configuration that could be used across the board. -Barry signature.asc Description: PGP signature
Re: Python BoF at DebConf10 - summary
On Aug 14, 2010, at 10:11 PM, Piotr Ożarowski wrote: >Summary of discussions in the Python BoF meet at DebConf10 Thanks for the update. Being not far from my back yard, I was hoping to make it but it fell within family summer vacation. >Squeeze will ship with Python 2.5, 2.6 (as default) and 3.1. Note that we will *not* be supporting Python 2.7 in Maverick, but it will be available. I will follow up with more details hopefully later today, but I think we have a solid foundation and should be able to enable it as the default in NN. The PPAs provide good feedback for build-dep failures, and you've posted some lists of byte-compilation failures for 2.7. Having another 6 months for Ubuntu, and working through and with debian-python to get fixes upstream, into Debian and syncd to Ubuntu for 11.04 will hopefully better align Ubuntu with Debian, and make this as smooth a transition as possible. The recommendations that came out of Debconf look pretty good to me. I especially like the promotion of unit tests during build time, and would like to help with this (it's been discussed here before but I've been too busy to work on it much). Maybe tox[1] can help us here. -Barry [1] http://codespeak.net/tox/ signature.asc Description: PGP signature
Re: common issue: setlocale handling?
On Aug 22, 2010, at 10:16 PM, Floris Bruynooghe wrote: >Using the standard warnings module this would only happen once for >each python script. This sounds like the more sensible behaviour, but >it sounds like it should be fixed upstream so that the unguarded >setlocale just shows this behaviour. If everyone here agrees with >that maybe someone should ask python to consider this? My guess is that you'd get a lot of push back from folks in python-dev. Won't a change like this have the potential to produce confusing, wrong, or hard to track down bugs? This kind of implicit behavior change seems to run counter to the "explicit is better" mantra. Note too that there's no way this would change before Python 3.2. I'm not qualified to personally champion this upstream, but I do encourage you to submit a bug at bugs.python.org if you feel strongly about it. If you can't or don't want to bring it to the attention of python-dev, I will do so in a neutral way. I strongly discourage Debian from deviating from upstream here. -Barry signature.asc Description: PGP signature
Re: Skip Python 2.6 and use 2.7 as default in Squeeze?
On Sep 01, 2010, at 12:33 AM, Marian Sigler wrote: >> Given how much work is required to change the default Python, does it >> make sense to just skip Python 2.6 and use 2.7 as the default Python >> version in Squeeze? >What has emerged here? I see that it won't be the default, but will it >be at least included at all? > >For most python users (I assume) it will be of great benefit already if >just pure python without all those packages debian provides would be >available (also) in version 2.7. Correct me if I'm wrong, but since Squeeze is frozen, it's too late to add support for Python 2.7, let alone make it the default. Technically, I think we're probably not yet ready to fully support Python 2.7 anyway, but we're perhaps not far. I wanted to get Python 2.7 as an officially supported version for Ubuntu 10.10, but that didn't happen either. It will be available, but not officially supported. I do expect that we'll turn on support very early in the Ubuntu 11.04 development cycle, and if things go well, I'll push for making it the default. In order to get here, I've done a lot of work analyzing, building, and fixing packages for Python 2.7, starting with the Ubuntu main repository and moving on to universe. We still have a handful of packages in main that ftbfs for 2.7, and a greater number in universe, but the total count is not insurmountable. There are still some outstanding questions though, such as how to test pure-Python packages which don't get built and we only find incompatibilities at install time. I'm working on some tools to gather these metrics and track development. All of this work should flow back into Debian, if it doesn't already come from there first. For example, numpy 1.4 fixes the ftbfs for Python 2.7. I basically did a sync from sid and built it in my Python 2.7 enabled chroot, then uploaded it to my Python 2.7 enabled PPA. Anything that I have to fix explicitly (e.g. the Subversion bindings) I'm trying to first work with upstream, and then get the changes into Debian, from which will flow into Ubuntu. For example, I have changes for python-defaults, python-support, and python-central to enable Python 2.7 support. Perhaps we can get those into unstable soon? I'm just now catching up after summer vacations, but plan on updating my tools and wiki pages to coordinate this effort. I'll interact mostly on this mailing list, but occasionally on ubuntu-devel for Ubuntu specific stuff (if there is anything). All help folks want to lend to the effort will be greatly appreciated! Cheers, -Barry signature.asc Description: PGP signature
Re: Skip Python 2.6 and use 2.7 as default in Squeeze?
On Sep 01, 2010, at 10:17 AM, Paul Wise wrote: >On Wed, Sep 1, 2010 at 6:33 AM, Marian Sigler >wrote: > >>> Given how much work is required to change the default Python, does >>> it make sense to just skip Python 2.6 and use 2.7 as the default >>> Python version in Squeeze? >> What has emerged here? I see that it won't be the default, but will >> it be at least included at all? > >As you can see here, python2.7 is only available in experimental: > >http://packages.qa.debian.org/p/python2.7.html > >As we are now in the squeeze freeze period it is unlikely that >python2.7 will enter squeeze: > >http://lists.debian.org/debian-devel-announce/2010/08/msg0.html > >> For most python users (I assume) it will be of great benefit already >> if just pure python without all those packages debian provides would >> be available (also) in version 2.7. > >backports.org exists for packages needed by users of stable/squeeze >that weren't yet ready for inclusion in testing/squeeze at the time of >the freeze. It would probably be a good idea to use it for python3.2 >and python2.7 packages once squeeze is out. What do you think about merging my changes to make Python 2.7 a supported version in experimental, either before or after squeeze is released? I guess once squeeze is out, it should probably go in testing though. I'd love to get 2.7 supported in debian soon-ish. It'll make it much easier to coordinate changes I'll have to make anyway in natty. -Barry P.S. Are any of y'all coming to UDS-N in Orlando Florida USA next month? If so, I'd love to have a face-to-face session or two on how we can improve collaborations on Python development issues between the two distros. signature.asc Description: PGP signature
Re: Skip Python 2.6 and use 2.7 as default in Squeeze?
On Sep 02, 2010, at 08:43 PM, Piotr Ożarowski wrote: >[Barry Warsaw, 2010-09-02] >> What do you think about merging my changes to make Python 2.7 a >> supported version in experimental, either before or after squeeze is >> released? I guess once squeeze is out, it should probably go in >> testing though. > >I'd do it already, but I'm waiting for release managers' answer to >this¹ mail. If they'll tell us to use experimental instead of unstable, >python2.7 in the list of supported Python versions would be problematic > >[¹] http://lists.debian.org/debian-release/2010/08/msg01107.html Cool. In the meantime, what's the best way to make my changes available to debian-python for review and/or merging? Right now, they're in these three Bazaar branches: https://code.launchpad.net/~barry/ubuntu/maverick/python-defaults/py27add https://code.launchpad.net/~barry/ubuntu/maverick/python-support/py27 https://code.launchpad.net/~barry/ubuntu/maverick/python-central/py27 Let me know so I can prepare for the inevitable :) -Barry signature.asc Description: PGP signature
Fw: Python packaging, dependencies, upstream facilities
Robert Collins has an interesting use case, though I'm not sure about his proposed solution. This probably touches on upstream and Debian packaging, so in the spirit of starting a discussion, I forward his pvtmsg here for debate (with his permission). -Barry Begin forwarded message: Date: Fri, 20 Aug 2010 09:22:20 +1200 From: Robert Collins To: Barry Warsaw Subject: Python packaging, dependencies, upstream facilities So, I'm going to give you a use case that debian packages suck at for python (they don't for C) and how I see a path-to-a-solution. If you were to make this happen, it would be oh so lovely. The use case - bare essentials, there are many versions of this. Deploy 2 versions of Launchpad to the same host at the same time, when an incompatible change to a dependency has happened. E.g. zope does a release that moves an attribute; our old version of LP would no longer work, and our new version of LP has other changes that mean it won't work on the old zope. How does this suck? If you upgrade the old zope, the running process is now running without the python files it loaded. Ugh. If you don't upgrade the old zope, you can't run the new LP. Ugh. What we do today, you know: we don't use debian packages for the dependencies. Which means we need a whole parallel dependency structure, tools to manage them, etc etc. Path to a solution: use an API marker analgous to the ABI markers C libraries have. Incompatible changes to a package bump the package *name*. e.g. python-zope.publication2.3 to python-zope.publication2.4 Compatible changes don't: python-zope.publication2.3-2.3.0ubuntu0 to python-zope.publication2.3-2.3.1ubuntu0 Both packages would provide python-zope.publication, but not conflict with each other. Software built on these that needs a specific version of the API would depend on python-zope.publication2.4 or whatever. And a debhelper script, like the C ABI one, could look at what build deps are installed to automate *that*. Problem is, stock python, has no API versioning concept. 'import foo' -> there can only be one. So one needs someline like 'Require(2.3)' or however the black magic setuptools stuff is spelt to be available. Specifically: - python-packageN and python-packageN+Y must be coinstallable - neither can use /site-packages/package - importing just 'package' should get the highest versioned one unless one is already loaded - using Require or something should get the highest versioned one matching your needs unless one is already loaded. Get this working, and the use case becomes: Install both versions of the dependency. Profit -Rob signature.asc Description: PGP signature
Re: Python packaging, dependencies, upstream facilities
On Sep 21, 2010, at 10:30 AM, Piotr Ożarowski wrote: >[Robert Collins, 2010-09-20] >> Path to a solution: use an API marker analgous to the ABI markers C >> libraries have. Incompatible changes to a package bump the package >> *name*. e.g. >> python-zope.publication2.3 to python-zope.publication2.4 >> Compatible changes don't: >> python-zope.publication2.3-2.3.0ubuntu0 to >> python-zope.publication2.3-2.3.1ubuntu0 > >Gustavo does something like this with CherryPy (python-cherrypy and >python-cherrypy3 conflict with each other due to the same namespace, >though). I maintain python-jinja and python-jinja2 but my upstream >*did* change module name after breaking API (I will remove python-jinja >after releasing Squeeze). Of course, this happens all the time. Witness unittest2, distutils2, urllib2, httplib2 and the list goes on. It's ugly and inconvenient, but in many ways easier to manage and at least we *understand* how it works! >I'm not sure we should try to solve this. IMHO we should try to >convince upstreams that breaking API/ABI so often is a bad thing >instead. Good luck. :) >Please note that: >* 3rd party Python module authors break API or ABI a lot, and I mean > A LOT (sic!). Even if we'll just Provide: python-foo-API, that will > produce a lot of new virtual packages and a lot of confusion, >* something as simple as changing --install-lib causes many Python > modules to break (hardcoded paths, things like > `os.path.join(os.path.dirname(__file__), '..', '..', 'share', > 'foo')`, etc.) so playing with sys.path will require patching even > more files, >* adding new source package (which is the easiest way to support one > more upstream version at the same time) requires going thru NEW which > would make ftp-masters very unhappy (dozens of NEW packages every > month), >* maintaining 2 versions of the same module means twice as many work >for > both Debian maintainer and security team >* ... > >I see only one sane way to fix the problem - changing Python >interpreter to recognize API from filenames, like foo.1.py foo.2.py >foo.2.3.py (with `import foo <= 2` as valid syntax) and let upstream >authors decide when to bump it, just like C guys do, but that's a >topic for python-devel mailing list... It would be, but I'm skeptical. In general, libraries tend to come in (Python) packages these days and you're generally not going to mix APIs in files within the same package, unless the package is itself managing multiple APIs. It might be something that distutils2 can attack, either through its metadata files or by encoding the API in package directory names. -Barry signature.asc Description: PGP signature
Re: Python packaging, dependencies, upstream facilities
On Sep 21, 2010, at 03:02 PM, Clint Byrum wrote: >In the java world, they use maven because it handles this for them. >They create a maven spec file that says "I need libX, libY, and >libZ (v1.1)". maven, during the build, goes out and finds libX and >libY's latest versions, then finds the closest match to libZ v1.1. >These are placed in jars in the classpath for the project, and viola >it "just works". Tools like buildout and virtualenv exist to address issues like this. Buildout's goal in life for example is to be able to version control repeatable built environments. It plays quite nicely with Python's own package management system (PyPI, aka Cheeseshop), but poorly with the operating system's package system. This isn't specific to Debian; a Python developer on Fedora would feel the same pain. There have been sessions at probably the last 3 or 4 Pycon sprints to try to address this, but it's difficult. I do think that the work that Tarek and distutils-sig is doing will get us closer, and the more Debian developers that are involved in the process, the better for us. I guess, if it were easy it would be done by now. ;) >But, if you can get the order of the library loading path right, >then this structure solves Robert's original desire: > >/usr/share/pyshared/foo1 >/usr/share/pyshared/foo2 >/usr/share/pyshared/foo => foo2 >/usr/share/pyshared/consumer-of-foo1/lib-packages/foo >=> /usr/share/pyshared/foo1 My sense is that there are two aspects to any possible solution. One is at the OS. Debian needs to support installing multiple versions of a package easily, whether that's by putting the version number in the package name as I think Robert suggests (but I don't like) or some other mechanism. But it shouldn't be doing things like symlinking 'foo' to 'foo2' because only the applications themselves know what versions they are compatible with. This is why buildout has a configuration file that you can use to specific explicitly the library versions you want (and let the dependency management algorithms figure out the rest). It also gives you an isolated environment so that you don't care what the OS has. So something like a configuration file that would ship with the pyfoo app that said "give me foo-1, bar-2.3, and baz-7.0" would set up the interpreter's run-time environment so that those package versions are mapped to "import foo", "import bar", and "import baz". >* What about instances where a dependent-library of consumer-of-foo1 >also wants to 'import foo' but needs foo2? Now I have to make sure >the entire chain works with foo1. How can I do that? IMO, you can't. >* How do I efficiently and reliably prepend that lib-packages >directory only when using consumer-of-foo1 > > >While I think Robert described it as "TERRIBLE" when I suggested >it the other day, the way that pylons does this, I think its at >least simple and understandable. > >For working on the CLI, pylons simply spawns a shell that sets >PYTHONPATH/PYTHONHOME. Likewise, one is required to do so when >running their particular pylons based app as a web application. >This allows you to run easy_install or anything else, inside that >CLI, and the libraries are installed local to the application. I don't like that sub-shell approach much myself, but I've had to do it in a previous life to properly select C++ shared libraries for a compiled application. Of course, this was on Solaris and many years ago. ;). I sympathize with Robert when he calls those approaches "terrible". virtualenv, buildout, and similar are at least less one-off hacky approaches, and the latest version of buildout I think can optionally be told to use system Python installed packages if they match the requested version number. There are folks who think that *every* Python application should essentially have its own isolated environment. I've held the opinion that it's dangerous to use the system Python both for essential system operations (as Debian, Ubuntu, and Gentoo among others do), and also for deployment of applications on those operating systems. I've seen situations in the past that trying to craft a proper environment for a particular app broke the OS. >Its not necessarily ideal, but it shows the great lengths one must >go to to have multiple versions of libraries. Yep. No doubt it's a crappy situation we find ourselves in today. Why not plan on coming to Pycon 2011 and sprinting with the distutils-sig? :) -Barry signature.asc Description: PGP signature
Re: common issue: setlocale handling?
On Sep 25, 2010, at 01:22 PM, Paul Wise wrote: >On Mon, Aug 23, 2010 at 8:17 PM, Barry Warsaw wrote: > >> My guess is that you'd get a lot of push back from folks in >> python-dev. Won't a change like this have the potential to produce >> confusing, wrong, or hard to track down bugs? This kind of implicit >> behavior change seems to run counter to the "explicit is better" >> mantra. >> >> Note too that there's no way this would change before Python 3.2. >> >> I'm not qualified to personally champion this upstream, but I do >> encourage you to submit a bug at bugs.python.org if you feel >> strongly about it. If you can't or don't want to bring it to the >> attention of python-dev, I will do so in a neutral way. > >I missed your mail due to an evil mail provider, but it looks like >doko has filed a bug about this: > >http://bugs.python.org/issue1703592 > >Some particularly insightful comments by cjwatson there. Yep. Unfortunately I don't think upstream can do anything but improve the documentation for Python 2.7. As I followed up in the bug report, I'm +0 for adding a convenience function to Python 3.2 that swallows the exception. -Barry signature.asc Description: PGP signature
Fw: Call for proposals -- PyCon 2011
Hi folks. Most of you probably have seen this, but just in case, I hope it's okay to forward to this list. It would be awesome have a nice representation of debian-python at the conference. Some talk ideas: * explaining to folks how Python is delivered on Debian. This would include things like our customizations (dist-packages), default versions, dh*/python-support/central/etc., the functioning of DPMT and PAPT * intro to packaging Python libraries and applications for Debian, including how to get new packages reviewed and sponsored * PEPs 3147, 3149 and how they and other technical developments will improve the Python experience on Debian, and what still needs to be done (e.g. libraries only compatible with one version of Python 3). * The uneasy marriage between Cheeseshop and apt and some possible solutions from a DD's perspective Now I'm probably not the best person to give these talks (certainly not *all* of them :) so if you agree that Atlanta is very nice in March and have talk ideas, I encourage you to propose one! If you have other ideas, let us know. It would be nice to have some coordination, I think. Also: definitely plan on coming to the sprints. They're the free part of the conference (not counting travel, room and board ;) and always a great way to spend some face time with other Python hackers on difficult problems. -Barry Begin forwarded message: Date: Thu, 23 Sep 2010 21:35:22 -0400 From: Jesse Noller To: python-announce-l...@python.org Subject: Call for proposals -- PyCon 2011 Call for proposals -- PyCon 2011 -- <http://us.pycon.org/2011/> === Proposal Due date: November 1st, 2010 PyCon is back! With a rocking new website, a great location and more Python hackers and luminaries under one roof than you could possibly shake a stick at. We've also added an "Extreme" talk track this year - no introduction, no fluff - only the pure technical meat! PyCon 2011 will be held March 9th through the 17th, 2011 in Atlanta, Georgia. (Home of some of the best southern food you can possibly find on Earth!) The PyCon conference days will be March 11-13, preceded by two tutorial days (March 9-10), and followed by four days of development sprints (March 14-17). PyCon 2011 is looking for proposals for the formal presentation tracks (this includes "extreme talks"). A request for proposals for poster sessions and tutorials will come separately. Want to showcase your skills as a Python Hacker? Want to have hundreds of people see your talk on the subject of your choice? Have some hot button issue you think the community needs to address, or have some package, code or project you simply love talking about? Want to launch your master plan to take over the world with Python? PyCon is your platform for getting the word out and teaching something new to hundreds of people, face to face. In the past, PyCon has had a broad range of presentations, from reports on academic and commercial projects, tutorials on a broad range of subjects, and case studies. All conference speakers are volunteers and come from a myriad of backgrounds: some are new speakers, some have been speaking for years. Everyone is welcome, so bring your passion and your code! We've had some incredible past PyCons, and we're looking to you to help us top them! Online proposal submission is open now! Proposals will be accepted through November 10th, with acceptance notifications coming out by January 20th. To get started, please see: <http://us.pycon.org/2011/speaker/> For videos of talks from previous years - check out: <http://python.mirocommunity.org/category/conferences> For more information on "Extreme Talks" see: <http://us.pycon.org/2011/speaker/extreme/> We look forward to seeing you in Atlanta! Please also note - registration for PyCon 2011 will also be capped at a maximum of 1,500 delegates, including speakers. When registration opens (soon), you're going to want to make sure you register early! Speakers with accepted talks will have a guaranteed slot. Important Dates: * November 1st, 2010: Talk proposals due. * December 15th, 2010: Acceptance emails sent. * January 19th, 2010: Early bird registration closes. * March 9-10th, 2011: Tutorial days at PyCon. * March 11-13th, 2011: PyCon main conference. * March 14-17th, 2011: PyCon sprints days. Contact Emails: Van Lindberg (Conference Chair) - v...@python.org Jesse Noller (Co-Chair) - jnol...@python.org PyCon Organizers list: pycon-organiz...@python.org -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/ signature.asc Description: PGP signature
Thoughts on the transition to Python 2.7
Python 2.7 is now supported in experimental. Along with the opening of Ubuntu 11.04 for development, it's time to revisit our transition to Python 2.7. It's something I'm keen on doing, and committed to working on. It's imperative to me that we do this transition so that both Debian and Ubuntu can benefit, and certainly I do not want to dump any work on Debian developers that I'm not willing to help with myself. Of course, any and all assistance will be greatly appreciated. Plus, I'll need sponsors! :) I want to make sure everyone is on the same page for the overall plan so that even if Ubuntu gets released with Python 2.7 support first (due to different release schedules), Debian will have everything it needs when the time for Wheezy comes. My vision for this transition is that it happens as much as possible in experimental, with syncs back to natty. I think there are two basic buckets of work: * Python 2.7 compatibility We need to check both build and installation (i.e. for pure Python packages) compatibility for 2.7. Fedora has largely completed their transition and I've spoken to Toshio who pointed me at their repository. They've submitted patches upstream, so hopefully it will be a "simple" matter of sync'ing to the right upstream version, or grabbing the Fedora changes. We do want to give proper attribution to the Fedora hackers where appropriate of course. I've already done this for cheetah for natty. Once I actually fix that ftbfs, I'll look for a sponsor for experimental. * dh_python2 transition Piotr and Matthias are keen on transitioning all of our pycentral/pysupport packaging to dh_python2, and I think we should try to do that as we touch packages for 2.7 support, though for a first pass, I think we should do this only where it's easy to change. For difficult transitions, let's concentrate first on getting 2.7 support, and noting that the package needs to get dh_python2'd later. There will be plenty of time to clean up the stragglers. python-support and python-central both need updates for Python 2.7. I have patches for those changes ready, but they'll need a sponsor. I want to document the plan, any issues or concerns we need to keep in mind, and progress on a page under wiki.debian.org/Python so that everyone can follow along and hopefully participate. Why is it important to add Python 2.7 support now? For several reasons: * Python 2.6 is no longer in upstream maintenance support. Only security fixes will be accepted and it is doubtful that there will be a full 2.6.7 release (there may be a source-only security release, but only if necessary). * Fedora has already nearly completed their transition to Python 2.7, though not yet released. We don't want to fall behind the curve! * Wheezy will include Python 2.7 as the default (and possibly only) version, so we might as well start the transition work now to give it plenty of time to bake. Why should we begin the switch to dh_python2 now? * TOOWTDI :) Yes it will be work to touch every package, which is why I suggest doing that opportunistically, while making sure python-support and python-central changes get us to Python 2.7 in the meantime. But dh_python2 is the future[*] so we might as well start getting there. If any of you are planning on coming to UDS-N next week, I look forward to meeting you and talking about how we can best make this transition happen. Eagerly awaiting your comment, -Barry [*] though it's odd to say "the future" where Python 2 is concerned. ;) signature.asc Description: PGP signature
Re: Thoughts on the transition to Python 2.7
On Oct 21, 2010, at 06:35 AM, Jakub Wilk wrote: >* Barry Warsaw , 2010-10-20, 19:30: >>We need to check both build and installation (i.e. for pure Python package >s) >>compatibility for 2.7. > >What is so special about pure-Python packages? I mistyped. I meant to say that Python file compatibility cannot be fully determined until byte-compilation time, which happens at installation. That is, unless the package has a test suite that runs at build time and fully touches all .py files. That doesn't even address correctness against Python 2.7, which is much harder. That's one reason why I think all packages should have test suites and all test suites should be run at build time - *and* the build should fail if the test suite fails (something I'm not sure happens all the time). -Barry signature.asc Description: PGP signature
Re: Thoughts on the transition to Python 2.7
On Oct 21, 2010, at 09:15 AM, Luca Falavigna wrote: >Il 21/10/2010 1.30, Barry Warsaw ha scritto: >> * Python 2.7 compatibility >> >> We need to check both build and installation (i.e. for pure Python packages) >> compatibility for 2.7. > >We should analyze which differences between 2.6 and 2.7 are relevant for >the most packages we ship in the archive, in order to avoid runtime >errors or, worse, logic ones which are quite impossible to notice. This >way, maintainers are warned about potential issues with 2.7, and can act >accordingly with proper fixes, not hacks. > >IIRC, 2.6 -> 2.7 doesn't introduce a lot of differences, so this task >should be easier than previous transitions, but I believe it has to be >accomplished in advance to provide guidance to our fellow maintainers. Agreed. While we can use the What's New document to give us a guide, I think as we start to add 2.7 support, we'll find a small list of actual problems that folks need to be aware of. I once went through the What's New to identify things I *thought* would be a problem, but I didn't capture that list. Now we have a place: http://wiki.debian.org/Python/27transition I'll try to do another pass through the What's New to write down some gotchas, but I may not have time before UDS, so please feel free to edit the page. >> Fedora has largely completed their transition and I've >> spoken to Toshio who pointed me at their repository. They've submitted >> patches upstream, so hopefully it will be a "simple" matter of sync'ing to >> the >> right upstream version, or grabbing the Fedora changes. We do want to give >> proper attribution to the Fedora hackers where appropriate of course. I've >> already done this for cheetah for natty. Once I actually fix that ftbfs, >> I'll >> look for a sponsor for experimental. > >Is there a quick way to collect/display these patches? Something like >BTS usertags, or something similar? Not that I'm aware of. What I did with cheetah was, 1) merge the latest upstream; 2) if that still fails, navigate to the Fedora project and see if they have a Python 2.7 patch. It might serve us well to have consistent naming for any Python 2.7 patch we add to our packages. Suggestions? >> * dh_python2 transition >> >> Piotr and Matthias are keen on transitioning all of our pycentral/pysupport >> packaging to dh_python2, and I think we should try to do that as we touch >> packages for 2.7 support, though for a first pass, I think we should do this >> only where it's easy to change. For difficult transitions, let's concentrate >> first on getting 2.7 support, and noting that the package needs to get >> dh_python2'd later. There will be plenty of time to clean up the stragglers. > >I agree. Moving to dh_python2 should (usually) be a matter of a >semi-automated process, so I'd focus more on providing support for 2.7 >and, as the last step, pushing dh_python2 as helper. +1. If it can be semi-automated, can we script it? >> I want to document the plan, any issues or concerns we need to keep in mind, >> and progress on a page under wiki.debian.org/Python so that everyone can >> follow along and hopefully participate. > >Fully agreed. Awesome. See above for the page link. -Barry signature.asc Description: PGP signature
Porting to Python 3
Ultimately of course we all want to land on Python 3. In conversation with Toshio Kuratomi and David Malcolm of the Fedora project, we feel that it's in all of our best interest to pool our resources. I think the distros are on the front lines of this transition, and working together across the distros is the best way for us all to get there. To that aim, there are two resources I want to make you aware of if you are interested in helping. The first is the python-porting mailing list: http://mail.python.org/mailman/listinfo/python-porting The second is the wiki page for coordinating porting activities: http://wiki.python.org/moin/PortingToPy3k Right now there is information on porting C code, Python code, and other helper information. Please join the effort! -Barry signature.asc Description: PGP signature
Re: Thoughts on the transition to Python 2.7
On Oct 21, 2010, at 06:50 PM, Jakub Wilk wrote: >If you care only about byte-compilation, installing a package is >overkill. It's way easier and faster just to unpack the .deb (dpkg-deb -x) >and run `python2.7 -m compileall /path/to/unpacked/stuff`. Nice tip, thanks. I've added it to the wiki page. -Barry signature.asc Description: PGP signature
Re: Wheezy plans
On Oct 18, 2010, at 05:33 PM, Josselin Mouette wrote: >If you mean “port all the scripts and applications of the default >installation from python to python3”, then that’s a goal that could be >achieved for wheezy. Especially if we work with other Python porters: http://mail.python.org/mailman/listinfo/python-porting -Barry signature.asc Description: PGP signature
Re: Wheezy plans
On Oct 22, 2010, at 07:52 PM, Julian Andres Klode wrote: >Tell that the Arch people: >http://www.archlinux.org/news/python-is-now-python-3/ > >Yep, they switched /usr/bin/python to Python 3.X I heard that Gentoo has done it too, but I have not verified that. -Barry signature.asc Description: PGP signature
Re: RFS: Didjvu, Djvusmooth, Ocrodjvu, Pybtex
On Nov 05, 2010, at 09:46 PM, Stefano Rivera wrote: >#!/usr/bin/make -f >%: > dh $@ [...] > >For new packages, you should probably consider using dh_python2 instead >of python-support. The eventual plan is to migrate all Python packages >to it. Which should be as easy as adding "--with python2" to the end of the dh line. -Barry signature.asc Description: PGP signature
Re: python sample packages?
On Nov 08, 2010, at 02:53 PM, Paul Elliott wrote: >Sorry if this is a faq, but are there any hellow world >debian sample packages that could be used as a starting point? I have a very simple extension module that I use to debug Python build experiments. I could debianize it and make it available for example. https://code.launchpad.net/~barry/+junk/stupid -Barry signature.asc Description: PGP signature
Re: python sample packages?
On Nov 08, 2010, at 04:16 PM, Yaroslav Halchenko wrote: >there is also py2dsc from python-stdeb if we go for automation (I've >not tried it yet though :-/) While I'm a big fan of python-stdeb, I should also point out that at UDS-N, James Westby laid out a plan and an architecture for a tool to make it easy to "debianize" many different types of packages, not just Python ones. There would be common parts and language-specific parts. Thus a lot of what python-stdeb does would land in this tool and we (Pythoneers) could write the plug-in to parse the setup.py and write the rules file. I don't think anything's been actually written yet, but James was (IIRC) going to set up a forum to discuss it and a code repository somewhere. -Barry signature.asc Description: PGP signature
