I uploaded a Python 3 port of ubiquity today, which I've been dealing with on and off for a while. It was surprisingly easier than I expected, although there were a few interesting roadblocks. I mostly embarked on this to get some practice in writing Python 3 code, with a side order of thinking that it would be a good idea to attack the stack of things to port from the top down rather than the bottom up, in case any of the work became unnecessary as a result.
I'd done a Python 3 port before, namely germinate (assisted at the time by http://python3porting.com/), so I had a rough idea of the sort of way I find it useful to attack these problems. In particular, my preference is definitely to avoid using 2to3 except as a sort of first cut at a to-do list. A great deal of the problem can be demolished simply by porting to a sufficiently modern Python 2 style first. Thus, I started with a few old standbys: print function, 'except Exception as value', module renamings, and so on. To start with, I found I could either guess randomly at common problems or run 2to3 in its default report-only mode, find a single category of problem (for instance, "map returns an iterator rather than a list"), grep for it through the whole codebase, and think of a way to make all occurrences valid in both Python 2 and 3. After a while I got to the point where it was worth adding --python2 and --python3 options to ubiquity's test suite so that I could try both (the test runner re-execs itself, so I couldn't just run it under different interpreters), and could continue until the test suite passed. A few specific notes on things I did in this stage: * The test suite used things like mock.patch("__builtin__.open"). I defined a helper like this: if sys.version >= '3': def builtin_patch(name): return mock.patch("builtins.%s" % name) else: def builtin_patch(name): return mock.patch("__builtin__.%s" % name) * gettext.install only takes unicode=1 (or unicode=True or whatever) in Python 2; that's unnecessary in Python 3. The neatest approach I found was: kwargs = {} if sys.version < '3': kwargs['unicode'] = 1 gettext.install(domain, LOCALEDIR, **kwargs) * If you're using python-apt, you *must* port entirely to the 0.8 API. python-apt tolerated this under Python 2, but the old API is compiled out under Python 3. /usr/share/python-apt/migrate-0.8.py may be of some partial help. * It's probably not news to anyone that you have to get your binary vs. text data model solid when porting to Python 3. There was one wrinkle I hadn't thought of, though. ubiquity uses subprocesses quite a bit, and they return binary data by default. Initially I tried .decode(), but after a while I realised that you can pass universal_newlines=True to subprocess.Popen (etc.) to get text output directly; this is much neater, works under both Python 2 and 3, even improves non-Unix support under Python 2 if you care ;-), and I recommend this approach. There were a couple of exceptions in ubiquity, either where requiring straight-up binary data or where dealing with text that's potentially in mixed encodings indicated by field names. * Three-arg raise is particularly awful for compatibility, because the Python 2 form is an uncatchable SyntaxError in Python 3; you have to use exec() to work around this. ubiquity only had one instance of this, so I used six.reraise(). * pyflakes got upset with functions/methods defined two ways depending on sys.version, so I had to add some exclusions. * python-libxml2 hasn't been ported. If you're currently using this, consider whether you can just use something in the standard library instead, rather than the larger python(3)-lxml. I switched ubiquity over to xml.etree.cElementTree, and aside from the expected footprint-related virtues of using the stdlib, it actually ran faster in our case. * I had to port PyICU to get the test suite working. This had been done upstream, but required packaging. * Be careful with what 2to3 says about list/iterator/view-related changes on dictionary methods and similar. Its conservative approach is to add list() more or less everywhere, but if you're actually using .items() etc. only as an iterator, you can leave it as-is. Watch out for cases where you modify the dictionary while iterating over it, though; in that case you'll indeed need list(). * I know others (including Barry) advocate 'from __future__ import unicode_literals'. I found this a good approach in tests, but it makes me nervous in library code since it could easily end up changing your API. I'd say only attempt this if you have exceptionally good test coverage. Now, ubiquity's test suite isn't everything it might be, although I was pleased to find that it got me most of the way. However, I still had to attack some frontend/backend glue code, and the KDE frontend is currently untested (any volunteers?). PyKDE4 had been ported upstream, but required packaging; thanks to Philip Muškovac for reviewing and uploading my patch there. There were then a few other things to fix, including calling sip.setapi("QVariant", 1) until I figure out what's going on with the new QVariant API, and joyously discovering that most bits of PyQt4 finally return ordinary Unicode strings in Python 3 rather than messing about with QString objects. But, finally, it all works at least in my tests. I expect there'll be a bit of shakedown time, but once things have settled I anticipate the main benefit being that we stop having failures only in languages that use non-ASCII characters, which has been a headache for us in the past. I also expect to be able to drop the compatibility code once everything is working and it's clear that we're past the point of no return in using only Python 3 on the desktop image. I definitely felt a tipping point here: once I'd ported a couple of packages, my approach to subsequent ones has been to go through all the changes I made for previous ports and duplicate each of them, which really speeds things up. Plus, of course, each library helps another batch of packages. Now that both GTK (via PyGI) and PyKDE are usable, it should be possible to attack quite a few multiple-frontend programs in Ubuntu; so please do! -- Colin Watson [[email protected]] -- ubuntu-devel mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
