Request for sponsored upload of tinyarray
Hello, I believe tinyarray[1] is ready for a sponsored upload. I have fixed the two remaining issues with the packaging according to policy: - The salsa repository no longer contains upstream history. - pristine-tar is used. I would be grateful if someone could upload the package or let me know if problems persist. Thanks Christoph [1] https://salsa.debian.org/python-team/modules/python-tinyarray signature.asc Description: PGP signature
Re: Salsa repository and first upload of tinyarray
Hello, A long time ago, I opened an ITP [1] for Kwant, a Python library for quantum physics computations that is popular with researchers in the field. Since we, the developers of Kwant, are running our own Debian package repository, there was not much pressure to migrate the packages into Debian proper, but finally as a new big Kwant release is taking shape, I would like to finish the work and phase out our own repository. (In order to be able to do that we will have to provide backports so that people running Debian stable will be able to use up-to-date Kwant, but I suppose that this should be possible.) As a first step I would like to package the library tinyarray [2]. Ondrej Novy was so kind to create a repository on salsa [3] to which I had pushed my packaging efforts. I noticed that since then Ondrej did some small modifications to the packaging. In the following, I would like to take up the discussion that got interrupted in August 2018: Ondrej Novy wrote on 22 Aug 2018: > st 22. 8. 2018 v 14:04 odesílatel Christoph Groth > napsal: > > > trusting PyPI to store the official release tarballs... > > trusting PyPi store without PGP signature is really bad idea. > > > I've also kept the full upstream history in the upstream branch, > > assuming this to be more robust and powerful. For example, this > > allows to directly cherry-pick commits into the patch queue. > > you can cherry-pick commits into pq even without full upstream history > in salsa git, using second remote. > > > Since both these practices are discouraged by the policy, I'm ready > > to give them up, but before I spend time working on it, I would like > > to ask > > I prefer to have git layout according to DPMT policy. There are > reasons for it. So, it seems to me that I should proceed as follows: * Create a new packaging repository that follows DPMT policy in the above two points. * Update it with the packaging modifications that have been done so far. * Push the new repository to salsa. Is there some recommended way to get rid of all the old branches and tags? Perhaps creating a new repository on salsa (and renaming / deleting the old one) would be good solution? Thanks Christoph [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886418 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886628 [3] https://salsa.debian.org/python-team/modules/python-tinyarray
Re: Salsa repository and first upload of tinyarray
Hello, Ondrej, thank you for the quick reaction. I've read the policy. So far (for my unofficial packages) I haven't been using pristine tar tool, trusting PyPI to store the official release tarballs. The package is configured such that the pristine tars can be downloaded using the command 'gbp import-orig --uscan'. I find this approach more elegant compared to the hack that is pristine-tar. I've also kept the full upstream history in the upstream branch, assuming this to be more robust and powerful. For example, this allows to directly cherry-pick commits into the patch queue. Since both these practices are discouraged by the policy, I'm ready to give them up, but before I spend time working on it, I would like to ask whether tinyarray could be exempt from any one of both rules. According with the policy, I would provide a debian/README.source file with the above rationale and instructions. Thanks, Christoph signature.asc Description: PGP signature
Salsa repository and first upload of tinyarray
Hello, I have packaged Tinyarray [1], that is a prerequisite for packaging Kwant [2], a library for simulating nanoelectronic devices that is in wide use by the respective research community. Several people from debian-science have expressed their willingness to sponsor the upload of Kwant. Since Tinyarray is a generic Python package (similar to NumPy), I believe that it belongs to the debian-python team. So far, I have prepared a packaging repository for Tinyarray [3]. The packaging has been reviewed by Steve Robbins (in CC). Would someone be so kind to create a repository on salsa.debian.org for Tinyarray and put it under group maintenance by the debian-python team? I am cwg-guest on salsa, but so far I'm not even a Debian contributor. Still, being the main author of Tinyarray, I commit myself to maintain the packaging. I guess that the repository [3] can be simply cloned at salsa and the current non-official packaging history may be kept. I verified yesterday that the package builds on both testing and stable. Once the package is in unstable/testing, I would also like to create a backport for current stable, so that I can shut down the unofficial debian package repository at https://kwant-project.org/. Thanks, Christoph [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886628 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886418 [3] https://gitlab.kwant-project.org/kwant/tinyarray
Parallelizing a submake of debian/rules
Hello, (Please keep me CCed, I'm not subscribed to debian-python.) I'm working on packaging the libraries Kwant [1] and Tinyarray [2] (a dependency for Kwant) for Debian. The Debian packaging was done in git repositories [3] [4] and seems finished. I'm not satisfied with one aspect of it, however: build parallelization does not work, i.e. running gbp buildpackage -j4 does not result in a parallel build. There are several parts of the build process of Kwant that could be parallelized. The only one that I'd like to consider here is the one that is consuming the most time: the generation of figures for the HTML documentation from patched example scripts. Having cloned [1], one can build the documentation in parallel like this: ./setup.py build cd doc make -j4 html In the Debian packaging [3] this is done in debian/rules like this: # Make documentation only if needed. ifneq "$(shell dh_listpackages | grep -- -doc)" "" PY3VER = $(shell python3 --version | cut -d' ' -f2 | cut -d. -f1,2) override_dh_installdocs: PYTHONPATH=$(CURDIR)/debian/python3-kwant/usr/lib/python$(PY3VER)/dist-packages \ LC_ALL=C.UTF-8 $(MAKE) -C doc html dh_installdocs endif But the submake launchd by "$(MAKE)" is not a parallel one. I have enabled parallel builds by giving the --parallel option to "dh", and I launch gbp buildpackage with "-j4". Moreover, DEB_BUILD_OPTIONS is set to "parallel=4", but all to no avail. The only way to make it work is to add an explicit "-j" to the make invocation in debian/rules, but that's certainly not the way it should be done. I'm grateful for any hints on how to fix the situation. Thanks, Christoph [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886418 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886628 [3] https://gitlab.kwant-project.org/kwant/debian-kwant [4] https://gitlab.kwant-project.org/kwant/debian-tinyarray signature.asc Description: PGP signature
Re: pybuild sphinxdoc and extensions
Piotr Ożarowski wrote:
> binary-indep (or build-indep or override_dh_auto_build-indep) is the
> right target: -indep targets will be ignored when only architecture specific
> files are build
OK, I see. Unfortunately, I didn’t succeed to use any other target than
dh_installdocs for building the docs.
>> • Using the override_dh_auto_install target instead of
>> override_dh_installdocs as Piotr suggested in his first message does
>> not work: the documentation build scripts cannot import kwant. It
>> seems that when dh_auto_install is run the directory to which
>> PYTHONPATH (see below) is set, is empty.
>
> that's because you used versioned path and all files were alredy moved
> to /usr/lib/python3/dist-packages
I tried again. Using (…)/usr/lib/python3/dist-packages doesn’t seem
to change anything.
>> • The suggestion from Piotr’s fourth message was to use the target
>> override_dh_auto_build and set PYTHONPATH to `pybuild --build
>> -i python3 -s custom --build-args 'echo {build_dir}'`. This evaluates
>> to (…)/.pybuild/python3_3.4/build but that directory is empty.
>
> did you run it *after* invoking dh_auto_build?
I did...
>> • Finally, Piotr’s last suggestion (to use pybuild) also doesn’t work:
>> the import of kwant fails.
>
> if you still have this code in another branch, I can take a look
Please do.
I’ve made a “better_rules” branch on
https://gitlab.kwant-project.org/kwant/debian-kwant.git (You can clone
that URL directly.)
You will also need
https://gitlab.kwant-project.org/kwant/debian-tinyarray.git as a
dependency.
Thanks,
Christoph
Re: Sponsor wanted for quantum physics Python package
Hello again, I think I should have asked rather on the Python Science mailing list, which is what I will do know. (I was fooled into thinking that the Debian Python team is handling all the Python packages in Debian because it packages Numpy and Scipy. But I see now that these two are most likely seen as “general infrastructure”.) Cheers, Christoph signature.asc Description: PGP signature
Sponsor wanted for quantum physics Python package
Hello, Together with a few colleagues I maintain the “Kwant” [1] Python package for computational quantum physics. Our library is quite popular: since its release in late 2013 it has been used for the preparation of at least 60 scientifc papers by people all over the world. It’s also used by many more people who do not (yet) publish, for example students. Is this enough to motivate the addition of Kwant to Debian? I maintain Debian packages of Kwant in a dedicated repository, as well as an Ubuntu PPA, but I think it would be cool if Kwant could enter Debian proper. Would anyone volunteer to sponsor/mentor it? The git repositories of the Debian packages for Kwant and Tinyarray (a library that is a dependency) can be found here: [2] [3] Thanks, Christoph [1] http://kwant-project.org/ [2] https://gitlab.kwant-project.org/kwant/debian-kwant [3] https://gitlab.kwant-project.org/kwant/debian-tinyarray signature.asc Description: PGP signature
Re: pybuild sphinxdoc and extensions
Piotr Ożarowski wrote:
> last one, I promise:
>
> override_dh_auto_build:
> dh_auto_build
> pybuild --build -i python3 -s custom --build-args 'make -C {dir}/doc
> html'
Piotr, thanks for all the funny versions…
Unfortunately, most of Piotr’s suggestions do not seem to work (I tried
for several hours). In the end, I’m using the debian/rules that is
pasted at the bottom. Specifically, I encountered the following problems:
• In order to create the doc package only when building for
binary-indep, Piotr’s suggestion of using the clause ifeq (,$(filter
nodoc,$(DEB_BUILD_OPTIONS))) does not work: the doc package gets built
always.
• Using the override_dh_auto_install target instead of
override_dh_installdocs as Piotr suggested in his first message does
not work: the documentation build scripts cannot import kwant. It
seems that when dh_auto_install is run the directory to which
PYTHONPATH (see below) is set, is empty.
• The suggestion from Piotr’s fourth message was to use the target
override_dh_auto_build and set PYTHONPATH to `pybuild --build
-i python3 -s custom --build-args 'echo {build_dir}'`. This evaluates
to (…)/.pybuild/python3_3.4/build but that directory is empty.
• Finally, Piotr’s last suggestion (to use pybuild) also doesn’t work:
the import of kwant fails.
I’m grateful for any suggestions. (Mostly out of curiosity, since the
below debian/rules seems to work well.)
If anyone would like to play with these things, the packaging git
repository of my project can be found here:
https://gitlab.kwant-project.org/kwant/debian-kwant
It also needs:
https://gitlab.kwant-project.org/kwant/debian-tinyarray
Cheers,
Christoph
#!/usr/bin/make -f
# export DH_VERBOSE = 1
export PYBUILD_NAME = kwant
%:
dh $@ --with python3,sphinxdoc --buildsystem=pybuild
override_dh_auto_clean:
dh_auto_clean
cd doc && $(MAKE) realclean
rm -rf tutorial build.conf
override_dh_compress-indep:
dh_compress -X.py
# The default locale with pbuilder is C. This makes Sphinx fail wit
# UFT-8-encoded input files. As a workaround, we set LC_ALL manually. If
# this is a bug in Sphinx, we may be able to remove this one day.
# Make documentation only if needed.
ifneq "$(shell dh_listpackages | grep -- -doc)" ""
PY3VER = $(shell python3 --version | cut -d' ' -f2 | cut -d. -f1,2)
override_dh_installdocs:
PYTHONPATH=$(CURDIR)/debian/python3-kwant/usr/lib/python$(PY3VER)/dist-packages
\
LC_ALL=C.UTF-8 $(MAKE) -C doc html
dh_installdocs
endif
Re: OT: To CC or not to CC?
Scott Kitterman wrote: > Debian has a policy for this: Thanks, I didn't know.
OT: To CC or not to CC? (was: Sorry for posting the same message twice)
Piotr Ożarowski wrote: > your message went through, I simply decide to ignore you for a while > because you CCed me on a mailing list that I subscribe :P But how can I know that you, or someone else, is subscribed? In this particular case it's of course quite obvious, but I believe that instead of asking oneself this question each time when replying to some message on some mailing list, CCing the author by default is the only reasonable policy. People who are annoyed by receiving the same message twice can quite easily configure any good MUA to hide messages that also go to lists that they read anyway.
Re: RFS: numpydoc 0.5-1
Denis Laxalde wrote: I'd like to request sponsorship for numpydoc, which had a new upstream release, the first supporting Python 3. It's available in python-modules SVN repository (I can upload it on mentors if that helps). Could you please reconsider sponsoring this upload? It would close bug 801947. The packaging is available in http://anonscm.debian.org/cgit/python-modules/packages/numpydoc.git. Thanks, Christoph signature.asc Description: PGP signature
Re: pybuild sphinxdoc and extensions
I found this thread while trying to update the Debian packaging
for a Python library that I maintain. The library has Sphinx
documentation that is quite complex to build: First, the library
needs to be built (it contains C extensions), then figures are
generated by scripts, and finally Sphinx is run. This is all done
in a Makefile.
Piotr Ożarowski wrote:
[PICCA Frederic-Emmanuel, 2014-04-06]
> so I would like to know how to change this snipset to use the
> {build_dir} for the PYTHONPATH.
you cannot really use pybuild for this, it will invoke your
command
after or before each interpreter/version call. If you hardcode
pybuild's
internal paths on the other hand (.pybuild/something in case of
distutils build plugin), it will break once the internal path
changes...
I suggest to move building docs to install target and add
debian/foo/usr/lib/python3/dist-packages) to PYTHONPATH. I also
build
directly into debian/foo/usr/share/doc/foo/html so I feel a
little less
guilty for doing it in install step ;)
With “install target”, do you mean dh_installdocs? In our
debian/rules, we have now
# Make documentation only if needed.
ifneq "$(shell dh_listpackages | grep -- -doc)" ""
override_dh_installdocs:
cd doc && PYTHONPATH=$(abspath
debian/python-kwant/usr/lib/python2.7/dist-packages)
$(MAKE) html
dh_installdocs
endif
This seems to work, but is this what you meant? Also, I have two
questions:
• How to get rid of the explicit mention of python2.7?
• Is the “ifneq”-clause the proper way to ensure that
documentation is only built for binary-indep?
I have one further more general question with regard to Debian
packaging with pybuild. Pybuild deletes the *.egg-info directory
but that directory is included in the tarball as made by “setup.py
sdist”. (I am aware that some people consider that directory to
be generated data, but this is debatable [1], and it’s standard
practice to include it in the tarball anyway.) This has the
consequence that after running “gbp buildpackage”, some files are
missing and thus “gbp buildpackage” cannot be re-run without “git
checkout .”. This situation is unsatisfactory. Is there a
recommended solution?
The complete Debian packaging of our library is available in a git
repository [2]. I hope that one day (rather soon) it could be
added to Debian proper.
Thanks for any hints,
Christoph
[1] https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2092#comment:16
[2] http://git.kwant-project.org/debian-kwant/
signature.asc
Description: PGP signature
Sorry for posting the same message twice
Hello again, Sorry for posting the same message twice. I believed that it got silently dropped as I did not see it show up on gmane (this was because I replied to an old thread). So I subscribed to whitel...@lists.debian.org and resent it. Christoph signature.asc Description: PGP signature
Re: pybuild sphinxdoc and extensions
Cześć Piotr,
I found this thread while trying to update the Debian packaging
for a Python library that I maintain. The library has Sphinx
documentation that is quite complex to build: First, the library
needs to be built (it contains C extensions), then figures are
generated by scripts, and finally Sphinx is run. This is all done
in a Makefile.
Piotr Ożarowski wrote:
[PICCA Frederic-Emmanuel, 2014-04-06]
> so I would like to know how to change this snipset to use the
> {build_dir} for the PYTHONPATH.
you cannot really use pybuild for this, it will invoke your
command
after or before each interpreter/version call. If you hardcode
pybuild's
internal paths on the other hand (.pybuild/something in case of
distutils build plugin), it will break once the internal path
changes...
I suggest to move building docs to install target and add
debian/foo/usr/lib/python3/dist-packages) to PYTHONPATH. I also
build
directly into debian/foo/usr/share/doc/foo/html so I feel a
little less
guilty for doing it in install step ;)
With “install target”, do you mean dh_installdocs? In our
debian/rules, we have now
# Make documentation only if needed.
ifneq "$(shell dh_listpackages | grep -- -doc)" ""
override_dh_installdocs:
cd doc && PYTHONPATH=$(abspath
debian/python-kwant/usr/lib/python2.7/dist-packages)
$(MAKE) html
dh_installdocs
endif
This seems to work, but is this what you meant? Also, I have two
questions:
• How to get rid of the explicit mention of python2.7?
• Is the “ifneq”-clause the proper way to ensure that
documentation is only built for binary-indep?
I have one further more general question with regard to Debian
packaging with pybuild. Pybuild deletes the *.egg-info directory
but that directory is included in the tarball as made by “setup.py
sdist”. (I am aware that some people consider that directory to
be generated data, but this is debatable [1], and it’s standard
practice to include it in the tarball anyway.) This has the
consequence that after running “gbp buildpackage”, some files are
missing and thus “gbp buildpackage” cannot be re-run without “git
checkout .”. This situation is unsatisfactory. Is there a
recommended solution?
The complete Debian packaging of our library is available in a git
repository [2].
Thanks for any hints,
Christoph
[1] https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2092#comment:16
[2] http://git.kwant-project.org/debian-kwant/
signature.asc
Description: PGP signature
