Hello, This is my second report on the work progress on a project PyPI to Debian Repository Converter mentored by Piotr Ożarowski.
Work: ----- Over the past two weeks I’ve worked mainly over planning the holistic algorithm of my program, detailed designing of plugins’ API and making the first attempts to implement it, introduction adjustments to previously stored functions related to the PyPI and improving the logging mechanism. During the discussions with my mentor on the algorithm of the program I’ve combined common arrangements and received guidance on the sketch (because it was easier and faster for me). Unfortunately, this isn’t an UML description, only illustrative graph by Google Docs, but after all, I think for those interested in can be helpful, so I enclose it[1]. As mentioned in my last report, source tarballs are successfully downloaded from PyPI. The next step was to change name of packages to comply with Debian policy[2]. If the name does not comply with Debian policy, it is changed using regular expression (e.g.: “Python Bytecode Verifier” to “python-bytecode-verifier”). Another regular expression tries to translate version numbers to those correctly recognized by dpkg --compare-versions (e.g.: “1alpha” is renamed to “1~alpha”). At this point, I have also decided to repack zip packages to tar.gz and to compress (previously uncompressed) tar archives with gzip. After managing to rename tarballs to desired name (that means in a format that is closest to e.g.: “lxml_2.3.4.orig.tar.gz”), I copy them from “tarballs” to “builds” directory, where I unpack them and finally have access to the setup.py file. Plugins: -------- The next very important step of the program is to run the plugin that converts package. I’ve decided to use a plugin system, so that in future my program can be easily extended. Generally I’ve decided that plugins should know a path to dir with setup.py, name and version of the packages (so that’s what I pass in constructor). I’ve limited the role of conversion plugins to generate debian/ directory. Although typically converters are trying to build source or binary package, it seems to me that it would be better to let plugins create debian directory only. I’ve designed a base class, which includes these methods: * __init__: constructor, accepts on the input: path to dir with setup.py file (usually dir created after extracting tarball), name and version of package; * configure: method which is called before conversion to let plugins initialize, configure, etc. (f.e. stdeb plugin can create config file at this stage) should return True if successfully initialized/configured; * prepare_command: is called by “run” method, returns string containing command used in run method that starts the conversion; * run: main method, converts package. If the conversion was successful, it returns True. This method should return (True/False, log from stdout, log from stderr) tuple. Most plugins should not override this method (and use prepare_command instead); * overrides: method which is called after run, it can be used to fix/improve files generated by the plugin in the debian directory. At this point I stopped for a moment for further planning and to check how plugins will work in practice. Initial implementation of first two converting plugins are ready: stdeb[3] and pkgme[4]. Then I’ve written the manage plugins function, set the bases of BuildPluginBase class and implemented dpkg[5] build plugin. A feeling when for the first time debian/ directory has began to appear serially and binary packages have started to build - priceless :-) Of course, this solution is still underdeveloped and requires a lot of work. At the moment used conversion plugins are working on packages which support Python 2 and I have to make adjustments to converters to support Python3 packaging[6] (which is one of announced goals of my project). Let me mention here, that during implementation of pkgme plugin, I’ve already fixed one bug[7] and received a warm acceptance after reporting new feature request[8]. As I’ve noted above, the option of creating by plugins only the debian/ dir will be a large simplification for me. When it comes to building plugins, I have plans to implement in addition to dpkg-buildpackage, inter alia: pbuilder[9] and sbuild[10]. In its current form, algorithm looks as follows: when conversion by the plugin with the highest priority doesn’t work, the program will try one with lower priority. Also, I'm going to introduce hooks, which will improve efficiency of plugins (f.e. pkgme plugin author can also prepare stdeb hooks that improve build dependency detection using pkgme’s internals). When conversion of a package has been successfully finished (that is, debian/ dir has been created) build plugin (also according to the priority) is started. I’ve started designing database structure to store most important information obtained from plugins (stdout and stderr logs, return code, test results, etc.) Summary: -------- It was a really nice moment, when my tool has started to be able to convert and build a lot of packages, but now I have to focus on improving quality and performance of each component. Plans: ------ In the next few days the most important task is to design detailed structure of database and to write the ORM corresponding to it. Next, based on information from the database, I want to code the appropriate behavior for each possible state of package. Then I'll try to improve many components of my tool (which, for now, have basic functionality only). Implement tests/validate plugins (eg.: lintian[11], lintian4py[12], unit tests, ...) and implement few new commandline options, such as: --skip-existing (to skip packages already present in Debian); --distro (to change the default distribution in generated packages) or --package (to convert only selected packages (or specific version, if given)), --clean (to clean up temporary files). My repository can be followed at: https://gitorious.org/pypi2deb -------- [1] http://img513.imageshack.us/img513/3820/packaging.png [2] http://www.debian.org/doc/debian-policy/ch-binary.html [3] https://github.com/astraw/stdeb [4] https://launchpad.net/pkgme [5] http://www.debian.org/doc/manuals/debian-faq/ch-pkgtools.en.html [6] http://wiki.debian.org/Python/Packaging [7] https://bugs.launchpad.net/pkgme/+bug/1010411 [8] https://bugs.launchpad.net/pkgme/+bug/1010419 [9] http://pbuilder.alioth.debian.org/ [10] http://packages.debian.org/sid/sbuild [11] http://lintian.debian.org/ [12] http://jwilk.net/software/lintian4python -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: http://lists.debian.org/[email protected]
