Re: PEP 453 affects Debian packaging of Python packages
On 09/18/2013 11:41 PM, Piotr Ożarowski wrote:
4) Python modules from dpkg are borderline useless for developers. We
package modules so that apps can use them, not so that people can
develop with them.
nobody forces Python/Ruby/... developers to use libraries prepared by
us... and yet they want to force us to use their .eggs and overwrite our
files.
Kind of funny to read this. Before doing a lot of Python packaging, I
was doing some PHP PEAR packaging. There as well, they forced the users
to use pear install instead of apt / dpkg. There as well, they were
writing stuff in their README about how to use PEAR. I suppose we have
lots of this kind of occurrence all over the place in Debian (Piotr, you
mentioned Ruby, it must be a good example, I guess (I don't know ruby at
all)).
In PHP, having the pear shell tool wasn't a problem, even if you don't
use it (and even if I would recommend against using it). Then Mathieu
Parent wrote pkg-php-tools, and I wrote debpear over it, which
automatically transforms a PEAR package into a Debian package. The same
way, I wrote debpypi, so that I don't waste my time writing again and
again the same stuff. Though my debpypi isn't good enough to be
released, I heard Piotr wrote the same kind of tool.
Shouldn't we go the same way, and encourage our users to use a kind of
wrapper around pip, so that they really get a Debian package installed
instead of a pip install thing which will mess everything? Piotr,
where is the tool you told me about? Could you share it? I've attached
my stupid script, just as a reference (though please don't tell me it's
not good enough, I know that already).
Cheers,
Thomas
#!/bin/sh
set -e
set -x
if [ ${1} = -u ] [ -n ${2} ] ; then
ORIG_URL=${2}
shift
shift
fi
if [ -z ${1} ] ; then
echo Usage: ${0} package-name
exit 1
fi
PKG_NAME=${1}
DEB_PKG_NAME=python-${PKG_NAME}
if [ ! -e ${PKG_NAME}.xml ] ; then
echo === Downloading DOAP XML record
wget -nv https://pypi.python.org/pypi?:action=doapname=${PKG_NAME};
-O ${PKG_NAME}.xml
fi
# Get info from the XML document using xpath.
VERSION_STRING=`xpath -e //release/Version/revision/text() ${PKG_NAME}.xml 2
/dev/null`
SHORT_DESC=`xpath -e //shortdesc/text() ${PKG_NAME}.xml 2 /dev/null`
LONG_DESC=`xpath -e //description/text() ${PKG_NAME}.xml 2 /dev/null`
UP_MAINT_NAME=`xpath -e //maintainer/foaf:Person/foaf:name/text()
${PKG_NAME}.xml 2 /dev/null`
HOMEPAGE=`xpath -e //homepage/@rdf:resource ${PKG_NAME}.xml 2 /dev/null |
cut -d= -f2 | sed 's###g'`
FIRST_LETTER=`echo ${PKG_NAME} | awk '{print substr($0,0,1)}'`
if [ -e ${DEB_PKG_NAME}_${VERSION_STRING}.orig.tar.xz ] ; then
ORIG=${DEB_PKG_NAME}_${VERSION_STRING}.orig.tar.xz
else
ORIG=${DEB_PKG_NAME}_${VERSION_STRING}.orig.tar.gz
fi
if [ ! -e ${ORIG} ] ; then
echo === Downloading ${ORIG} file
if [ -z ${ORIG_URL} ] ; then
ORIG_URL=https://pypi.python.org/packages/source/${FIRST_LETTER}/${PKG_NAME}/${PKG_NAME}-${VERSION_STRING}.tar.gz
fi
wget -nv ${ORIG_URL} -O ${ORIG}
fi
echo === Extracting ${ORIG}
tar -xf ${ORIG}
mv ${PKG_NAME}-${VERSION_STRING} ${DEB_PKG_NAME}-${VERSION_STRING}
echo === Creating debian folder for ${DEB_PKG_NAME}
if [ ! -d ${DEB_PKG_NAME}-${VERSION_STRING}/debian/source ] ; then
mkdir -p ${DEB_PKG_NAME}-${VERSION_STRING}/debian/source
fi
cd ${DEB_PKG_NAME}-${VERSION_STRING}
echo Source: ${DEB_PKG_NAME}
Section: python
Priority: optional
Maintainer: PKG OpenStack openstack-de...@lists.alioth.debian.org
Uploaders: Julien Danjou a...@debian.org,
Thomas Goirand z...@debian.org,
Mehdi Abaakouk sil...@sileht.net
Build-Depends: debhelper (= 9), python-setuptools, python-all (= 2.6.6-3~),
openstack-pkg-tools
Standards-Version: 3.9.4
Vcs-Browser: http://anonscm.debian.org/gitweb/?p=openstack/${DEB_PKG_NAME}.git
Vcs-Git: git://anonscm.debian.org/openstack/${DEB_PKG_NAME}.git
Homepage: ${HOMEPAGE}
Package: ${DEB_PKG_NAME}
Architecture: all
Pre-Depends: dpkg (= 1.15.6~)
Depends: \${python:Depends}, \${misc:Depends}
Recommends: \${python:Recommends}
Description: ${SHORT_DESC}
${LONG_DESC}
debian/control
EDITOR=touch dch --create --package ${DEB_PKG_NAME} --distribution unstable
--urgency low -v ${VERSION_STRING}-1
rm +1
echo 9 debian/compat
echo Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: ${PKG_NAME}
Source: ${HOMEPAGE}
Files: debian/*
Copyright: (c) 2013, Thomas Goirand z...@debian.org
License: Apache-2
Files: *
Copyright: (c) 2013, ${UP_MAINT_NAME}
License: Apache-2
License: Apache-2
Licensed under the Apache License, Version 2.0 (the \License\);
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
.
http://www.apache.org/licenses/LICENSE-2.0
.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on
Re: PEP 453 affects Debian packaging of Python packages
On 09/19/2013 12:55 AM, Thomas Kluyver wrote: On 18 September 2013 08:41, Piotr Ożarowski pi...@debian.org mailto:pi...@debian.org wrote: so instead of reinventing the wheel and trying to make something that works everywhere they should make it easier for others to convert whatever they provide (tarballs?) into .rpm, .deb or .exe. From a developer point of view: this leaves you dependent on other people to get the latest release of your software to users, which can be very frustrating. For instance, I'm a developer for IPython: we made a 1.0 release over a month ago, and there's already been a 1.1 release since then, but Debian unstable still doesn't have either of these. This is not to criticise our packager, who we have a good relationship with, but simply to point out that this system is beyond our control. If we recommend that people use apt/yum/port/whatever to install IPython, they'll get an old package, with bugs that we've already fixed. By contrast, we update the packages on PyPI at release time, so users installing with pip will always get the current version. Thomas Then get involved in the Debian packaging: problem solved! Thomas -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523c031e.3030...@debian.org
Re: Disabling pip for root?
On 09/19/2013 05:26 PM, W. Martin Borgert wrote: On 2013-09-18 09:36, Paul Tagliamonte wrote: 1) pip isn't for global package management, for this is stupid. If we disabled root use of pip, I think we'd all be a bit happier. Very quick and very dirty patch attached. 4) Python modules from dpkg are borderline useless for developers. We package modules so that apps can use them, not so that people can develop with them. That is maybe my problem with pip: Developers tend to use every Python library in every version they like from PyPI. I couldn't agree more with that. In OpenStack, I see upstream writing: package-name==1.2.3 Or even: package-name=1.2.3,=1.2.4 when in fact, there's not even a 1.2.4 released, and the developer just writes this just in case. I've been fighting a lot of these. That's really bullshit dependency management, and fighting with it is exhausting, patching endelessly the requirements.txt file. Especially tiring when dealing with nearly a hundred package. Worse, some upstream developers don't understand that writing a dependency like = 1.2.4 in a requirements.txt file does *not* fix the problem for a distribution, and that if 1.2.5 is available in the distro, we will *not* go back and package an older version. This is the kind of trouble we have with pip, and it's not going away. How can we get rid of this, or at least keep it to a bearable minimum? Well, unfortunately, I think that there's no other way but educating upstream coders. But also, it'd be super nice if the people upstream for Python itself understood it and could see from the point of view of a distribution like Debian. Seeing that they push for even more pip crap shows that they didn't get it. As a project leader I generally have to think about deployment and this means: Use Debian stable and backports! Only for long-term projects the next Debian stable release might be relevant. But if you have a dozen or so libraries installed by pip, there are libraries that will not be packaged for Debian and the deployment is wrecked. It's really not hard to package some Python modules for Python. Having a tool that does it automatically - even not very well -, like debpear does, can solve this kind of trouble. Giving access to this kind of tools to a wide user base can also help solving the oh, but that module isn't available in Debian kind of problem. Thomas -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523c0780.7010...@debian.org
Re: Disabling pip for root?
On Fri Sep 20 2013 07:48:07 PM HKT, Paul R. Tagliamonte paul...@gmail.com wrote: Why not see if upstream pip wants to add an optional dpkg install method when using root on a dpkgey system? Slightly hackish but at least itd reduce suck. Very good idea indeed! Thomas -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1379678437.1838.5.camel@Nokia-N900-42-11
Re: Disabling pip for root?
Why not see if upstream pip wants to add an optional dpkg install method when using root on a dpkgey system? Slightly hackish but at least itd reduce suck. Or disable root pip. Either way is fine T On Sep 20, 2013 4:30 AM, Thomas Goirand z...@debian.org wrote: On 09/19/2013 05:26 PM, W. Martin Borgert wrote: On 2013-09-18 09:36, Paul Tagliamonte wrote: 1) pip isn't for global package management, for this is stupid. If we disabled root use of pip, I think we'd all be a bit happier. Very quick and very dirty patch attached. 4) Python modules from dpkg are borderline useless for developers. We package modules so that apps can use them, not so that people can develop with them. That is maybe my problem with pip: Developers tend to use every Python library in every version they like from PyPI. I couldn't agree more with that. In OpenStack, I see upstream writing: package-name==1.2.3 Or even: package-name=1.2.3,=1.2.4 when in fact, there's not even a 1.2.4 released, and the developer just writes this just in case. I've been fighting a lot of these. That's really bullshit dependency management, and fighting with it is exhausting, patching endelessly the requirements.txt file. Especially tiring when dealing with nearly a hundred package. Worse, some upstream developers don't understand that writing a dependency like = 1.2.4 in a requirements.txt file does *not* fix the problem for a distribution, and that if 1.2.5 is available in the distro, we will *not* go back and package an older version. This is the kind of trouble we have with pip, and it's not going away. How can we get rid of this, or at least keep it to a bearable minimum? Well, unfortunately, I think that there's no other way but educating upstream coders. But also, it'd be super nice if the people upstream for Python itself understood it and could see from the point of view of a distribution like Debian. Seeing that they push for even more pip crap shows that they didn't get it. As a project leader I generally have to think about deployment and this means: Use Debian stable and backports! Only for long-term projects the next Debian stable release might be relevant. But if you have a dozen or so libraries installed by pip, there are libraries that will not be packaged for Debian and the deployment is wrecked. It's really not hard to package some Python modules for Python. Having a tool that does it automatically - even not very well -, like debpear does, can solve this kind of trouble. Giving access to this kind of tools to a wide user base can also help solving the oh, but that module isn't available in Debian kind of problem. Thomas -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523c0780.7010...@debian.org
Re: PEP 453 affects Debian packaging of Python packages
[Thomas Goirand, 2013-09-20] Though my debpypi isn't good enough to be released, I heard Piotr wrote the same kind of tool. Shouldn't we go the same way, and encourage our users to use a kind of wrapper around pip, so that they really get a Debian package installed instead of a pip install thing which will mess everything? Piotr, where is the tool you told me about? Could you share it? I didn't write it, I packaged it for Debian. I have write access to upstream repo and plan to update it to support Python 3.X, though package name: python-stdeb provides: /usr/bin/py2dsc, /usr/bin/pypi-install -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130920121455.gm12...@p1otr.com
Re: PEP 453 affects Debian packaging of Python packages
On Fri, Sep 20, 2013 at 2:14 PM, Piotr Ożarowski pi...@debian.org wrote: [Thomas Goirand, 2013-09-20] Though my debpypi isn't good enough to be released, I heard Piotr wrote the same kind of tool. Shouldn't we go the same way, and encourage our users to use a kind of wrapper around pip, so that they really get a Debian package installed instead of a pip install thing which will mess everything? Piotr, where is the tool you told me about? Could you share it? I didn't write it, I packaged it for Debian. I have write access to upstream repo and plan to update it to support Python 3.X, though package name: python-stdeb provides: /usr/bin/py2dsc, /usr/bin/pypi-install Wow! I didn't know about pypi-install. That it wasn't mentioned before in this thread may indicate many people don't know about it either. -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caa77j2dtwzjggmdct+xle66rijojoyc+b2e6h57rffdtir7...@mail.gmail.com
Re: PEP 453 affects Debian packaging of Python packages
On 20 September 2013 01:11, Thomas Goirand z...@debian.org wrote: From a developer point of view: this leaves you dependent on other people to get the latest release of your software to users, which can be very frustrating. For instance, I'm a developer for IPython: we made a 1.0 release over a month ago, and there's already been a 1.1 release since then, but Debian unstable still doesn't have either of these. This is not to criticise our packager, who we have a good relationship with, but simply to point out that this system is beyond our control. If we recommend that people use apt/yum/port/whatever to install IPython, they'll get an old package, with bugs that we've already fixed. By contrast, we update the packages on PyPI at release time, so users installing with pip will always get the current version. Thomas Then get involved in the Debian packaging: problem solved! I try to get involved with Debian packaging. But, to be blunt, it is a slow, frustrating process, and the effort I put in feels utterly disproportionate to the results. We are not going to get many Python programmers involved with the packaging process as it stands. Take the most recent two packages I have worked on: - python3-sympy: The package is sitting in the team SVN, waiting for someone to review or upload it. I last touched it two months ago to package a new upstream release. - python-xlrd: My changes were rejected, although the package was extremely out of date, because the package had an individual 'Maintainer' who hadn't been seen for three years. It took another four months (a long time in developer terms) before the wheels turned and someone actually got an up to date version packaged. I wish I could say this is atypical. But from my limited experience of DPMT, a slow, tricky process is the norm. There are procedural traps (e.g. to make a package you must first file a bug by e-mail, then mark your new package as closing the bug), social traps (annoying a 'maintainer' overprotective of what is ultimately little bit of metadata to turn a Python package into a Debian package), and you may simply fail to catch the interest of a Debian developer--as I seem to have failed with python3-sympy--in which case you're out of luck. I don't wish to criticise without making suggestions as to how this could be improved: - Write a 'how to keep your distro packager happy' guide for developers. E.g. many Python developers don't know that distros will move data files to /usr/share, but when you do know, it's easy to write code so that the patch to achieve this is trivial. - Have a simple way for developers to submit packaging information without having to join Debian teams, file ITP bugs, and all that cruft. Technically, Debian mentors is already something like that, but maintainers mostly ignore it. Which brings me to: - Put the emphasis on keeping packages up to date and simple, not on 'maintainer rights'. Packaging should not be an art form. If someone uploads a newer version to Debian mentors (or another submission system), the maintainer should get an e-mail. If a package is out of date for more than a few days without a clear reason, people should be prodding the maintainer to ask what's up. If a maintainer is regularly unresponsive, drop the package to team maintainership so that other people can work on it. Put another way, focus on what is best for Debian and for the upstream project being packaged, not what the person who has appointed themself 'maintainer' of that package wants. - Make it really, really easy to accept changes to packaging. One of the reasons for the meteoric rise of Github is that when someone submits a change that clearly improves things, the repository owner literally just has to click a big green button to accept that. I don't mean DPMT should use Github, but, for instance, if upstream makes a bugfix release 1.4.3, and Debian has 1.4.2, it should be as near as possible one click/one command to grab the new version, update the changelog, commit the change, upload the package, and whatever else needs doing. I know this won't go down well with everyone here. I contend that if the situation continues as is, Debian packaging will be seen as ever more irrelevant by Python developers. Already, the default assumption is that distro packages will be out of date. The scientific Python community is unhappy with pip PyPI, but is looking to yet more packaging tools, such as conda, to address this (despite Yaroslav Halchenko's excellent work with NeuroDebian). Thomas
Re: PEP 453 affects Debian packaging of Python packages
On Fri, Sep 20, 2013 at 10:46:14AM -0700, Thomas Kluyver wrote: I try to get involved with Debian packaging. But, to be blunt, it is a slow, Debian works at a slower pace. We make sound technical decisions, over all. It's annoying to people who want results and want them now, but it's something we all have to deal with. frustrating process, and the effort I put in feels utterly disproportionate to the results. We are not going to get many Python programmers involved with the packaging process as it stands. Take the most recent two packages I have worked on: - python3-sympy: The package is sitting in the team SVN, waiting for someone to review or upload it. I last touched it two months ago to package a new upstream release. - python-xlrd: My changes were rejected, although the package was extremely out of date, because the package had an individual 'Maintainer' who hadn't been seen for three years. It took another four months (a long time in developer terms) before the wheels turned and someone actually got an up to date version packaged. I wish I could say this is atypical. But from my limited experience of DPMT, a slow, tricky process is the norm. There are procedural traps (e.g. to make a package you must first file a bug by e-mail, then mark your new package as closing the bug), social traps (annoying a 'maintainer' overprotective of what is ultimately little bit of metadata to turn a Python package into a Debian package), and you may simply fail to catch the interest of a Debian developer--as I seem to have failed with python3-sympy--in which case you're out of luck. I don't wish to criticise without making suggestions as to how this could be improved: - Write a 'how to keep your distro packager happy' guide for developers. E.g. many Python developers don't know that distros will move data files to /usr/ share, but when you do know, it's easy to write code so that the patch to achieve this is trivial. - Have a simple way for developers to submit packaging information without having to join Debian teams, file ITP bugs, and all that cruft. Technically, Debian mentors is already something like that, but maintainers mostly ignore it. Which brings me to: - Put the emphasis on keeping packages up to date and simple, not on Sorry, no :) It's not about keeping the libraries up to date, it's about keeping the applications up to date. If a library breaks API because the maintainer wanted another toy rewrite, we're not going to upload it and break half the archive. That's silly. Hell, we shouldn't even introduce a module unless it has an app using it. We focus on *user stability*, not bleeding edge, yes, even in Unstable. Perhaps we can consider exerpimental uploads? 'maintainer rights'. Packaging should not be an art form. If someone uploads a newer version to Debian mentors (or another submission system), the maintainer should get an e-mail. If a package is out of date for more than a few days without a clear reason, people should be prodding the maintainer to ask what's up. If a maintainer is regularly unresponsive, drop the package to team maintainership so that other people can work on it. Put another way, focus on what is best for Debian and for the upstream project being packaged, not what the person who has appointed themself 'maintainer' of that package wants. While I generally agree, most of the time the maintainer of a package knows the details better than anyone else in the project, and can make better techincal decisions. - Make it really, really easy to accept changes to packaging. One of the reasons for the meteoric rise of Github is that when someone submits a change that clearly improves things, the repository owner literally just has to click a big green button to accept that. I don't mean DPMT should use Github, but, for instance, if upstream makes a bugfix release 1.4.3, and Debian has 1.4.2, it should be as near as possible one click/one command to grab the new version, update the changelog, commit the change, upload the package, and whatever else needs doing. (check the license, review the diff for changes that may cause issues or are unsafe, ensure it doesn't break API, ) I know this won't go down well with everyone here. I contend that if the situation continues as is, Debian packaging will be seen as ever more irrelevant by Python developers. Already, the default assumption is that distro We care more about users than developers. Python developers can use virtualenv and pip on Debian like any other Python development env. packages will be out of date. The scientific Python community is unhappy with pip PyPI, but is looking to yet more packaging tools, such as conda, to address this (despite Yaroslav Halchenko's excellent work with NeuroDebian). Thomas -- .''`. Paul Tagliamonte paul...@debian.org : :' : Proud Debian Developer `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30
Re: PEP 453 affects Debian packaging of Python packages
Hi Paul. Keep in mind that despite my strong opinions I'm doing my best to be constructive with what I say below. W dniu pią, wrz 20, 2013 o 7:52 ,nadawca Paul Tagliamonte paul...@debian.org napisał: On Fri, Sep 20, 2013 at 10:46:14AM -0700, Thomas Kluyver wrote: I try to get involved with Debian packaging. But, to be blunt, it is a slow, Debian works at a slower pace. We make sound technical decisions, over all. It's annoying to people who want results and want them now, but it's something we all have to deal with. Slow might not be bad *but* arcane and crufty interfaces that are more reminiscent of the teletype rather than the web should be improved upon. Obviously this requires manpower but I want to highlight that it does put people off from contributing. frustrating process, and the effort I put in feels utterly disproportionate to the results. We are not going to get many Python programmers involved with the packaging process as it stands. Take the most recent two packages I have worked on: - python3-sympy: The package is sitting in the team SVN, waiting for someone to review or upload it. I last touched it two months ago to package a new upstream release. - python-xlrd: My changes were rejected, although the package was extremely out of date, because the package had an individual 'Maintainer' who hadn't been seen for three years. It took another four months (a long time in developer terms) before the wheels turned and someone actually got an up to date version packaged. I wish I could say this is atypical. But from my limited experience of DPMT, a slow, tricky process is the norm. There are procedural traps (e.g. to make a package you must first file a bug by e-mail, then mark your new package as closing the bug), social traps (annoying a 'maintainer' overprotective of what is ultimately little bit of metadata to turn a Python package into a Debian package), and you may simply fail to catch the interest of a Debian developer--as I seem to have failed with python3-sympy--in which case you're out of luck. I don't wish to criticise without making suggestions as to how this could be improved: - Write a 'how to keep your distro packager happy' guide for developers. E.g. many Python developers don't know that distros will move data files to /usr/ share, but when you do know, it's easy to write code so that the patch to achieve this is trivial. - Have a simple way for developers to submit packaging information without having to join Debian teams, file ITP bugs, and all that cruft. Technically, Debian mentors is already something like that, but maintainers mostly ignore it. Which brings me to: - Put the emphasis on keeping packages up to date and simple, not on Sorry, no :) It's not about keeping the libraries up to date, it's about keeping the applications up to date. Sorry but *YES*. This view might be appropriate for what the world was ten years ago but now it's not the only right way to look at what packages are and who is consuming them. If Debian has only stable packages for users (whatever that means) then people that need packages for other reasons will simply ignore either Debian or our packages - plain and simple. I'm not trying to say that we should throw away everything we have and start over but we *should* recognize that there are other use cases which we are not serving *at all* by enforcing our current policy. If a library breaks API because the maintainer wanted another toy rewrite, we're not going to upload it and break half the archive. That's silly. toy rewrite clouds the picture, let's not be the judge here, of what constitutes a valid reason for breaking the API. Not all projects have stable API but still have real-world users. Should those never be in Debian? Debian should not fit the ideal spherical cow in vaccum model. Debian should support the what-is-really-out-there model. Hell, we shouldn't even introduce a module unless it has an app using it. This is a catch22 problem: My own experience: I wanted to package my application that I wrote for Linaro a while ago and all we did was to abandon the ideas as, at that time, python-celery was not available and the amount of dependencies it has made the project infeasible. I could have tried packaging them but then people could reject that with 'apps first' policy. How many apps out there would be simple to package if we had all of pypi inside Debian? We focus on *user stability*, not bleeding edge, yes, even in Unstable. Perhaps we can consider exerpimental uploads? Focusing on user stability *only* misses being user useful for others. Some users suffer because we have old version of a library and cannot get the upgrade required by another application. Our model assumes that only one version may be installed. This is the spherical model in vacuum model IMHO. We seem to love
Re: PEP 453 affects Debian packaging of Python packages
On 20 September 2013 10:52, Paul Tagliamonte paul...@debian.org wrote:
If a library breaks API because the maintainer wanted another toy
rewrite, we're not going to upload it and break half the archive. That's
silly.
This condescending attitude towards developers ('another toy rewrite')
doesn't help. Work with upstream developers to understand what they're
doing, encourage them to care about API stability and to use conventions
like semantic versioning and deprecation warnings to reduce the impact of
changes. There are plenty of developers who do care about backwards
compatibility.
We could also have topic-specific extra repos, so that a user can add, say,
a Python-science repo to get newer versions of some packages by accepting a
bit of extra risk associated with that. Neurodebian offers something like
that, but in general it's hard to set up. Ubuntu PPAs are much easier to
get set up, but don't build for Debian relases.
Hell, we shouldn't even introduce a module unless it has an app using
it.
- This gives module authors little to no incentive to get involved in
Debian packaging.
- In scientific Python, the expectation is that *users* will import and use
modules directly. Likewise, most code that depends on a package like Django
is not going to be packaged as a Debian app. I don't think Debian has some
special insight into what currently unpackaged things users want.
We care more about users than developers. Python developers can use
virtualenv and pip on Debian like any other Python development env.
Believe it or not, developers care about users as well. That's why we're
writing code and fixing bugs. We want the people using our software to have
the best possible experience. However, we regularly see bug reports for
problems where we have already released a fix, because users are on
outdated versions. So, upstream projects are increasingly inclined to
bypass distros and offer their software to users by a more direct route.
Again, it feels like packagers see developers as the enemy. Yes, developers
will at times do things that you disagree with, but fundamentally we are on
the same team. We both want to deliver great software to users. If you
fight developers, you will lose, by sheer weight of numbers.
Thomas
Re: PEP 453 affects Debian packaging of Python packages
On 2013-09-20 at 10:46:14 -0700, Thomas Kluyver wrote: - Write a 'how to keep your distro packager happy' guide for developers. E.g. many Python developers don't know that distros will move data files to /usr/share, but when you do know, it's easy to write code so that the patch to achieve this is trivial. there is an UpstreamGuide_ on the wiki: it's not python specific, but it also has some informations about python, and most of it should be pretty language indipendent. .. _UpstreamGuide: https://wiki.debian.org/UpstreamGuide - Have a simple way for developers to submit packaging information without having to join Debian teams, file ITP bugs, and all that cruft. ITP bugs aren't cruft, they are a way to prevent duplication of work, which would lead to even more frustration. - Put the emphasis on keeping packages up to date and simple, not on 'maintainer rights'. Packaging should not be an art form. If someone uploads a newer version to Debian mentors (or another submission system), the maintainer should get an e-mail. If a package is out of date for more than a few days without a clear reason, people should be prodding the maintainer to ask what's up. A few days are definitely too little: even on the AUR (for Arch Linux) to orphan a package one needed to wait 2 weeks after the maintainer had been contacted (not just after an upstream release), and that's in a distribution that prides itself on being extremely up-to-date, no matter what. Most people working on packaging are volounteers, some of them maybe can only work on Debian in the weekend, others maybe can usually work every evening, but an upstream release happened to be on a week when their paid job had a deadline, and they had to work long hours. If a maintainer is regularly unresponsive, drop the package to team maintainership so that other people can work on it. That's already what happens (sort of): the MIA_ procedure exists exactly to try to contact an unresponsive maintainer and, if there is no answer, give up the package for adoption. It takes more than the two weeks for AUR, but afaik it should take less than the 4 months you mentioned: was the MIA team contacted? .. _MIA: https://wiki.debian.org/Teams/MIA for instance, if upstream makes a bugfix release 1.4.3, and Debian has 1.4.2, it should be as near as possible one click/one command to grab the new version, update the changelog, commit the change, upload the package, and whatever else needs doing. there is a single command, (I believe it's ``svn-upgrade --uscan``) which downloads the new version of a package (maintained in svn, but I believe there is a similar command for git-buildpackage), updates the changelog etc. The maintainer still has to test the package before uploading, but that's something that can only be automatized up to a point. -- Elena ``of Valhalla'' -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130920192941.gb3...@genesis.home.trueelena.org
Re: PEP 453 affects Debian packaging of Python packages
On 2013-09-20 13:52, Paul Tagliamonte wrote: It's not about keeping the libraries up to date, it's about keeping the applications up to date. ... Hell, we shouldn't even introduce a module unless it has an app using it. I tend to disagree here (slightly). Too me, it is very important, that Debian is a perfect platform for different applications, even if not packaged for Debian. E.g. Debian contains all libraries and Python modules needed for an application my company does. I heard the same from OpenERP developers. They use Ubuntu and Debian and find it very useful, that everything they need is available, be it a Python module or nginx. If there were Python modules missing, it would make Debian a less usable tool for us and probably many others. Using virtualenv and pip to play with a new module is nice, but not necessarily an option for serious application development, e.g. because you have to think how to deploy the application later. That's why I want to see all useful Python modules in Debian, even if no Debian package uses them. If somebody files an ITP or RFP, they know, what it is good for :~) To me, having Debian packages of Python modules does not only mean the package is available via apt-get instead of pip. It does also mean, that at least one Debian maintainer looked at the usability and maybe quality of the code, that DD and FTP masters accepted the license, that I have the same bug tracker available for the Python module, web server and kernel. Debian packages are checked by others, whether one can still build and install them, etc. -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130920223236.ga20...@fama.tangosoft.com
Re: PEP 453 affects Debian packaging of Python packages
On 20 September 2013 12:08, Paul Tagliamonte paul...@debian.org wrote: Don't take this as me trashing on Python or Pythonistas. If you want to talk about this in person, I'm usually at PyCon. I'm also usually in the packaging BOF. Perhaps we can bring some of this up there this year? Unfortunately, I don't think I'll be going to PyCon next year. I'll probably be at next year's SciPy, but I presume you won't be there? I'm glad to hear that you are a developer as well. However, a lot of the tone in this discussion suggests that all developers are idiots who can't keep an API intact. I think there is a much more productive conversation to be had with upstreams. help. Work with upstream developers to understand what they're doing, encourage them to care about API stability and to use conventions like semantic versioning and deprecation warnings to reduce the impact of changes. There are plenty of developers who do care about backwards compatibility. And just as many (if not more) that don't. As a result, we have to design the system to defend against this breakage. Alright, a proposal: create some kind of 'expedited upload' pathway, so upstream developers sign to say that they will use semantic versioning, never break APIs in a bugfix release, issue deprecation warnings for any code relying on something that they plan to remove, and create strong regression tests. We could add other reasonable conditions to this list. The carrot for developers is that, by agreeing to this, they get semi-automated uploads of minor updates (run the package's own tests, run DEP-8 tests of its dependencies, and a brief manual check). If projects that have signed break the requirements, kick them off the pathway for a couple of subsequent releases. At present, there is no carrot - getting stuff packaged is slow whether upstream cares about API stability or not. (I'm sure someone reading is about to point out a reason why this won't work exactly as I have described. Don't bother telling me - I'm not really expecting this will actually happen. But please try to think of how things might change, instead of why nothing must change.) We could also have topic-specific extra repos, so that a user can add, say, a Python-science repo to get newer versions of some packages by accepting a bit of extra risk associated with that. Neurodebian offers something like that, but in general it's hard to set up. Ubuntu PPAs are much easier to get set up, but don't build for Debian relases. PPAMAIN would be nice for special cases like this, aye! OK, I searched that term. It's nice to see that Debian is finally considering setting up a PPA system. Did anything more happen on that after the mailing list thread got sidetracked into security questions? Elena: there is an UpstreamGuide_ on the wiki: So promote it! I'm pretty sure I've never seen that URL before. ITP bugs aren't cruft, they are a way to prevent duplication of work, which would lead to even more frustration. That seems like an unlikely problem in real world cases - how often will two people decide to package the same, currently unpackaged, piece of software, within the couple of days or so before the first one publishes their work. And if it is necessary: make it simple. A web page listing 'things people have said they're working on packaging in the last week', with a text box to add something, would be easy to implement. Instead, I have to install a command line tool, run through several questions, paste it into an e-mail client, and then remember to close the bug in the package's changelog. It's a double-reciprocating sledgehammer with cruise control to crush an ant. ;-) It takes more than the two weeks for AUR, but afaik it should take less than the 4 months you mentioned: was the MIA team contacted? Yes. They never responded to me, but instead filed a bug which I didn't see. Most of the four months was then just waiting for someone to act on that bug. I didn't realise anything had happened until I checked on the package today. Thomas
Re: PEP 453 affects Debian packaging of Python packages
On Fri, Sep 20, 2013 at 03:44:05PM -0700, Thomas Kluyver wrote: That seems like an unlikely problem in real world cases - how often will two people decide to package the same, currently unpackaged, piece of software, within the couple of days or so before the first one publishes their work. Packaging often takes much longer than a couple of days, especially if the packager is not experienced. And when the work is published somewhere, but not yet uploaded, there is no general way to know if it's published and where. -- WBR, wRAR signature.asc Description: Digital signature
Re: PEP 453 affects Debian packaging of Python packages
On 20 September 2013 16:00, Andrey Rahmatullin w...@wrar.name wrote: Packaging often takes much longer than a couple of days, especially if the packager is not experienced. And when the work is published somewhere, but not yet uploaded, there is no general way to know if it's published and where. And how many inexperienced packagers will start by filing an ITP bug? Or indeed by searching for existing ITP bugs? More likely, the bug only gets filed when they've prepared a working package and Lintian is complaining about the lack of a line closing the ITP bug. This is not a problem specific to Debian - people work on all sorts of projects, and post them all over the web. Yes, sometimes work gets duplicated, but the world does not end. For a project like Debian, you can sanely keep a registry of 'stuff people are working on', but if you're going to do that, at least make it easy to use. Thomas
Re: PEP 453 affects Debian packaging of Python packages
On Friday, September 20, 2013 15:44:05 Thomas Kluyver wrote: On 20 September 2013 12:08, Paul Tagliamonte paul...@debian.org wrote: Don't take this as me trashing on Python or Pythonistas. If you want to talk about this in person, I'm usually at PyCon. I'm also usually in the packaging BOF. Perhaps we can bring some of this up there this year? Unfortunately, I don't think I'll be going to PyCon next year. I'll probably be at next year's SciPy, but I presume you won't be there? I'm glad to hear that you are a developer as well. However, a lot of the tone in this discussion suggests that all developers are idiots who can't keep an API intact. I think there is a much more productive conversation to be had with upstreams. help. Work with upstream developers to understand what they're doing, encourage them to care about API stability and to use conventions like semantic versioning and deprecation warnings to reduce the impact of changes. There are plenty of developers who do care about backwards compatibility. And just as many (if not more) that don't. As a result, we have to design the system to defend against this breakage. Alright, a proposal: create some kind of 'expedited upload' pathway, so upstream developers sign to say that they will use semantic versioning, never break APIs in a bugfix release, issue deprecation warnings for any code relying on something that they plan to remove, and create strong regression tests. We could add other reasonable conditions to this list. The carrot for developers is that, by agreeing to this, they get semi-automated uploads of minor updates (run the package's own tests, run DEP-8 tests of its dependencies, and a brief manual check). If projects that have signed break the requirements, kick them off the pathway for a couple of subsequent releases. At present, there is no carrot - getting stuff packaged is slow whether upstream cares about API stability or not. (I'm sure someone reading is about to point out a reason why this won't work exactly as I have described. Don't bother telling me - I'm not really expecting this will actually happen. But please try to think of how things might change, instead of why nothing must change.) In cases where the maintainer and upstream have an active relationship, the maintainer will know which upstreams these are and how quickly it's reasonable to upload. The problem is the maintainer structure, but that do many packages are under maintained. We could also have topic-specific extra repos, so that a user can add, say, a Python-science repo to get newer versions of some packages by accepting a bit of extra risk associated with that. Neurodebian offers something like that, but in general it's hard to set up. Ubuntu PPAs are much easier to get set up, but don't build for Debian relases. PPAMAIN would be nice for special cases like this, aye! OK, I searched that term. It's nice to see that Debian is finally considering setting up a PPA system. Did anything more happen on that after the mailing list thread got sidetracked into security questions? Elena: there is an UpstreamGuide_ on the wiki: So promote it! I'm pretty sure I've never seen that URL before. ITP bugs aren't cruft, they are a way to prevent duplication of work, which would lead to even more frustration. That seems like an unlikely problem in real world cases - how often will two people decide to package the same, currently unpackaged, piece of software, within the couple of days or so before the first one publishes their work. There's one package I maintain in Debian that, when I went to package it, I found an old RFP bug (similar to ITP) and was able to make contact and now we co-maintain the package. The ITP bugs are not just to avoid collisions. They go to debian-devel where many developers can review them and comment. it's not rare to see discussion on them that leads to the packager making some chances in their plans. And if it is necessary: make it simple. A web page listing 'things people have said they're working on packaging in the last week', with a text box to add something, would be easy to implement. Instead, I have to install a command line tool, run through several questions, paste it into an e-mail client, and then remember to close the bug in the package's changelog. It's a double-reciprocating sledgehammer with cruise control to crush an ant. ;-) The questions are useful. I find reportbug pretty easy to use and there's certainly no copy/pasting needed. Scott K -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1903307.AnJu4akPzN@scott-latitude-e6320
Re: PEP 453 affects Debian packaging of Python packages
PLEASE everyone, I'm registered to the list, and adding me as Cc: is breaking my filters... Thomas -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523d1d76.60...@debian.org
Re: PEP 453 affects Debian packaging of Python packages
Le 21/09/13 01:17, Thomas Kluyver a écrit : And how many inexperienced packagers will start by filing an ITP bug? Or indeed by searching for existing ITP bugs? More likely, the bug only gets filed when they've prepared a working package and Lintian is complaining about the lack of a line closing the ITP bug. I keep forgetting filling an ITP. It is usually caught on final review because for some reason lintian is not being run by default by my packaging chain :-( I find the ITP/wnpp annoying as well. I usually end up browsing http://www.debian.org/devel/wnpp/ , opening all links then searching for my package :-] I then bootup an instance and try to remember to report the bug against the 'wnpp' package. But I am a newcomer, so I am easily confusing / forgetting rules :) -- Antoine hashar Musso -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523d2883.9070...@free.fr
