Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Piotr Ożarowski]

[Ben Finney, 2015-11-26]
> * Patch upstream's ‘setup.py’ to rename the command file to something
>   like ‘execute-lorem’, so that it can be installed alongside the
>   ‘lorem’ package directory.

instead of patching I usually let it install into /usr/bin and mv it to
/usr/share/foo/ under a different name (and then symlink) - see f.e.
gaupol's debian/rules

> * Create a symlink in the application-private directory, making an
>   application namespace from which to import the private modules.

in gaupol I use --install-lib=/usr/share/gaupol/
so that in the end I have:

 /usr/share/gaupol/gaupol - the library
 /usr/share/gaupol/run - the script
 /usr/bin/gaupol→/usr/share/gaupol/run - symlink

-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Ben Finney]

Piotr Ożarowski  writes:

> As I already suggested in my previous mail, installing scripts into
> private directory and symlinking them in /usr/bin is the cleanest
> solution IMHO

Thank you. I have come to a compromise solution on this which builds on
that suggestion.


Ben Finney  writes:

> * The ‘console_scripts’ entry point is used to specify command line
>   programs, ‘lorem’ and ‘ipsum’, using that Distutils feature. Those are
>   the correct names for the commands, so I don't want to mess with that.

Main point of compromise: Change the installed names of the commands.

Second point of compromise: Make an application namespace package.


Distutils (and Python generally) makes it highly awkward to have an
importable module *also* be an executable program.

Python's import mechanism makes it infeasible to have a module file
named without a suffix; this makes it infeasible to name a command file
by Unix conventions of having no suffix for the implementation language.

There are many approaches to address this, which I won't detail here,
but suffice it to say that none of them work smoothly.

So the compromise is:

* Patch upstream's ‘setup.py’ to rename the command file to something
  like ‘execute-lorem’, so that it can be installed alongside the
  ‘lorem’ package directory.

* Create a symlink in the application-private directory, making an
  application namespace from which to import the private modules.

* Create a public symlink, ‘/usr/bin/lorem’, to the private
  ‘execute-lorem’ command file.

> * The ‘--install-lib’ option is used in ‘PYBUILD_INSTALL_ARGS’ to place
>   the Python packages in an application-private directory,
>   ‘/usr/share/foo-app/’. The point of that is so they won't be on the
>   Python module search path.

This remains the same; the ‘--install-lib’ option is used to place the
application-private packages in a private directory. That's the whole
point of this endeavour, and everything else is a workaround for
Python's awkwardness to deal with this.

The programs will still need to be importable, but we want to protect
against a conflict with some other publicly-installed Python library.

So we create a namespace ‘FooApp’ specifically for the application, with
a symlink internal to the application's own library directory.

> * The ‘--install-scripts’ option is *not* used, because Distutils places
>   the constructed scripts in the correct directory (‘/usr/bin/’) by
>   default.

We must now install the program command files to the same location, in
order that the application-private packages can be imported by those
programs.

So the ‘--install-scripts’ option is used, naming the same
application-private directory as for the ‘--install-lib’ option.

> * The script names are correct, but are identical to names of
>   directories in the ‘/usr/share/foo-app/’ top level. That's
>   another good reason not to use the ‘--install-scripts’ option. It is
>   also normal and expected, and shouldn't be a problem because the
>   scripts will not exist there; they belong in ‘/usr/bin/’.

Because the command files cannot keep their upstream name and live in
the same location as package directories with the same names, we need to
patch upstream's ‘setup.py’.

Patch upstream's ‘setup.py’ so the ‘console_scripts’ names differ from
the package directory names. For clarity of association and purpose, I
chose to name them ‘execute-lorem’ and ‘execute-ipsum’.

Patch the target of the ‘console_scripts’ entry points, so that those
entry points are preceded by the ‘FooApp’ namespace package.

>   According to Robert's earlier message, that means the Distutils
>   metadata file needs to be not in the application's private
>   directory, but in a directory on the Python module search path. That
>   seems odd to me, since this is not amodule import being done.

Thanks to everyone for explaining this to me; it's a messy hack, and I
do wish the Python Packaging Authority patience and strength in solving
this better than the mess we have now.

> Here is the output from the example. I can provide the files if anyone
> wants to experiment.

Here is an updated session to show how this works:

=
$ pwd
/home/bignose/Projects/debian/foo-app-1.2.3

$ find .
.
./setup.py
./lorem
./lorem/amet.py
./lorem/dolor.py
./lorem/sit.py
./lorem/__init__.py
./ipsum
./ipsum/elit.py
./ipsum/adipiscing.py
./ipsum/consecteur.py
./ipsum/__init__.py
./debian
./debian/compat
./debian/rules
./debian/foo-app.links
./debian/changelog
./debian/control

$ cat ./setup.py
from setuptools import (setup, find_packages)

setup(
name="FooApp",
version="1.2.3",
packages=find_packages(),

entry_points={
'console_scripts': [
"execute-lorem = FooApp.lorem.dolor:main",
"execute-ipsum = FooApp.ipsum.consecteur:main",
],
},  
)   


debhelper (>= 9)

$ cat ./debian/rules 
#! /usr/bin/make -f

PACKAGE_NAME = foo-app

package_share_d

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

[Vincent Cheng, 2015-11-26]
> > that's why I don't believe you built it in jessie. python-sphinx is not 
> > backported
> > and you didn't mention any patches
> >
> > solutions:
> > 1) backport python-sphinx, or
> > 2) use patch from Ubuntu: 
> > https://patches.ubuntu.com/r/routes/routes_2.2-1ubuntu1.patch
> 
> Backports packages are allowed to build-dep on other backported
> packages of the same suite (i.e. you can build packages destined for

that's right, same as in any other suite. That's why I mentioned
backporting sphinx as one of solutions

> jessie-backports in a chroot with packages from both jessie and
> jessie-backports to satisfy build dependencies). Given that Thomas has
> already mentioned in the very beginning of this thread that he intends
> on backporting sphinx to jessie-backports as well (which satisfies the
> unmet build-dep above), I think it's safe to assume that he will be
> uploading packages to backports in the necessary order to satisfy
> build dependencies.

right, he listed sphinx among those he wants to backport (I missed that
fact). I checked jessie-backports and it didn't contain it (it still
doesn't). Building and uploading in the right order was always the plan,
right?

> FWIW, I see no *technical* reason to reject Thomas' backported
> packages. It looks to me that he's done everything he needs to do
> according to the backports rules, and that you're imposing additional
> requirements that (strictly speaking) he doesn't have to meet (i.e.
> providing debdiffs for every backports upload).

I still stand by this requirement - I'm the one who gets all the emails
and bug reports and I do not welcome NMUs that do not fix RC bugs.
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

[Vincent Cheng, 2015-11-26]
> >> solution: pass -sa to dpkg-source ("--force-orig-source" in sbuild or 
> >> "--debbuildopts -sa" in pbuilder)
> 
> No, that's not necessary as long as the orig tarball is already
> present in the archive in another suite (which would be true for
> backports, since the package is supposed to be derived from the
> current version in testing).

any pointers to docs/announcements? That's news to me...
Does that mean backports are just another suite now and I actually can
tell zigo to not hijack my packages?

> > the second one is missing changelog items and your packages have this one 
> > too.
> >
> > solution: pass -vFOO to dpkg-source (where FOO is version in stable)
> > ("--force-orig-source --debbuildopt -vFOO" in sbuild and
> > "--debbuildopts '-sa -vFOO'" in pbuilder)
> 
> No, the -v flag is no longer required as of April of this year [1]. It
> is merely "recommended" now [2].

ACK
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: Pushing a bunch of packages to jessie-backports

[Vincent Cheng]

Hi Piotr,

On Thu, Nov 26, 2015 at 7:02 AM, Piotr Ożarowski  wrote:
>> I've uploaded all what I prepared to this URL:
>> http://sid.gplhost.com/bpo/
>
> THANK YOU!
>
[build log snipped]
> that's why I don't believe you built it in jessie. python-sphinx is not 
> backported
> and you didn't mention any patches
>
> solutions:
> 1) backport python-sphinx, or
> 2) use patch from Ubuntu: 
> https://patches.ubuntu.com/r/routes/routes_2.2-1ubuntu1.patch

Backports packages are allowed to build-dep on other backported
packages of the same suite (i.e. you can build packages destined for
jessie-backports in a chroot with packages from both jessie and
jessie-backports to satisfy build dependencies). Given that Thomas has
already mentioned in the very beginning of this thread that he intends
on backporting sphinx to jessie-backports as well (which satisfies the
unmet build-dep above), I think it's safe to assume that he will be
uploading packages to backports in the necessary order to satisfy
build dependencies.

FWIW, I see no *technical* reason to reject Thomas' backported
packages. It looks to me that he's done everything he needs to do
according to the backports rules, and that you're imposing additional
requirements that (strictly speaking) he doesn't have to meet (i.e.
providing debdiffs for every backports upload).

Regards,
Vincent

Re: Pushing a bunch of packages to jessie-backports

[Vincent Cheng]

Hi Piotr,

On Thu, Nov 26, 2015 at 7:09 AM, Piotr Ożarowski  wrote:
> [Piotr Ożarowski, 2015-11-26]
>> that's the first mistake most backporters do, they forgot to include orig 
>> tarball
>> (ftp masters would detect that and reject your package, additional work for 
>> them)
>>
>> solution: pass -sa to dpkg-source ("--force-orig-source" in sbuild or 
>> "--debbuildopts -sa" in pbuilder)

No, that's not necessary as long as the orig tarball is already
present in the archive in another suite (which would be true for
backports, since the package is supposed to be derived from the
current version in testing).

> the second one is missing changelog items and your packages have this one too.
>
> solution: pass -vFOO to dpkg-source (where FOO is version in stable)
> ("--force-orig-source --debbuildopt -vFOO" in sbuild and
> "--debbuildopts '-sa -vFOO'" in pbuilder)

No, the -v flag is no longer required as of April of this year [1]. It
is merely "recommended" now [2].

Regards,
Vincent

[1] https://lists.debian.org/debian-backports-announce/2015/04/msg0.html
[2] http://backports.debian.org/Contribute/#index7h3

Re: How to manually install python scripts and needed modules (Was: How to tweak homebrew install target for Gubbins)

[Piotr Ożarowski]

[Andreas Tille, 2015-11-26]
> since nobody seems to have any experience with brew in my last
> commit to the gubbins packaging git[1] I tried to install the
> included python files manually.  Unfortunately for the python
> module I installed to
> 
>/usr/share/pyshared/gubbins

pyshared is an implementation detail of some Python related tools in
Debian. Please do not use it.

> the processing to /usr/lib/python3.4/gubbins does not work.  Any
> hint how to do this correctly (or is my attempt a bad idea in
> general)?

if it's a public Python module, install into
/usr/lib/python3/dist-packages/, if not (and name of your binary package
suggests that), use anything but that dir, f.e. /usr/share/gubbins/
(remember to install there also scripts or your will have to patch them
to update sys.path)

-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

How to manually install python scripts and needed modules (Was: How to tweak homebrew install target for Gubbins)

[Andreas Tille]

Hi,

since nobody seems to have any experience with brew in my last
commit to the gubbins packaging git[1] I tried to install the
included python files manually.  Unfortunately for the python
module I installed to

   /usr/share/pyshared/gubbins

the processing to /usr/lib/python3.4/gubbins does not work.  Any
hint how to do this correctly (or is my attempt a bad idea in
general)?

(Any hints to my question below are welcome as well - I just
want to install the python files ...).

Thanks for any help

  Andreas.

[1] git://anonscm.debian.org/debian-med/gubbins.git

On Wed, Nov 25, 2015 at 01:04:31PM +0100, Andreas Tille wrote:
> Hi,
> 
> I'm working on packaging Gubbins for the Debian Med team.  The build
> currently ends up in
> 
> ...
> Making install in python
> make[2]: Entering directory '/build/gubbins-1.4.3/python'
> /usr/bin/python3 setup.py build
> running build
> running build_py
> running build_scripts
> make[3]: Entering directory '/build/gubbins-1.4.3/python'
> /usr/bin/python3 setup.py install
> running install
> Checking .pth file support in /usr/local/lib/python3.4/dist-packages/
> error: can't create or remove files in install directory
> 
> The following error occurred while trying to add or remove files in the
> installation directory:
> 
> [Errno 13] Permission denied: 
> '/usr/local/lib/python3.4/dist-packages/test-easy-install-14282.pth'
> 
> The installation directory you specified (via --install-dir, --prefix, or
> the distutils default setting) was:
> 
> /usr/local/lib/python3.4/dist-packages/
> 
> Perhaps your account does not have write access to this directory?  If the
> installation directory is a system-owned directory, you may need to sign in
> as the administrator or "root" account.  If you do not have administrative
> access to this machine, you may wish to choose a different installation
> directory, preferably one that is listed in your PYTHONPATH environment
> variable.
> 
> For information on other options, you may wish to consult the
> documentation at:
> 
>   https://pythonhosted.org/setuptools/easy_install.html
> 
> Please make the appropriate changes for your system and try again.
> 
> Makefile:462: recipe for target 'install-exec-local' failed
> ...
> 
> 
> 
> After contacting the author he suspects that's due to the usage of
> brew:
> 
> For homebrew I don’t let it make install the python dependancies
> (line 67), and instead invoke it later in the standard python manner.
> I don’t know if this is the best way to do it, but it makes it work
> in this instance.
> https://github.com/Homebrew/homebrew-science/blob/master/gubbins.rb
> Any advise on how to improve things would be welcome
> 
> Since I also never stumbled upon brew I wonder what might be the best
> way to get a clean install target for the Debian packaging and may be as
> a hint for upstream who is very responsive.
> 
> Kind regards
> 
>   Andreas.
> 
> 
> [1] git://anonscm.debian.org/debian-med/gubbins.git
> 
> -- 
> http://fam-tille.de
> 
> 

-- 
http://fam-tille.de

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

[Piotr Ożarowski, 2015-11-26]
> that's the first mistake most backporters do, they forgot to include orig 
> tarball
> (ftp masters would detect that and reject your package, additional work for 
> them)
> 
> solution: pass -sa to dpkg-source ("--force-orig-source" in sbuild or 
> "--debbuildopts -sa" in pbuilder)

the second one is missing changelog items and your packages have this one too.

solution: pass -vFOO to dpkg-source (where FOO is version in stable)
("--force-orig-source --debbuildopt -vFOO" in sbuild and 
"--debbuildopts '-sa -vFOO'" in pbuilder)

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

┘
| 
| Build Architecture: amd64
| Build-Space: 0
| Build-Time: 0
| Distribution: jessie
| Fail-Stage: install-deps
| Host Architecture: amd64
| Install-Time: 0
| Job: /tmp/build/routes_2.2-1~bpo8+1.dsc
| Machine Architecture: amd64
| Package: routes
| Package-Time: 0
| Source-Version: 2.2-1~bpo8+1
| Space: 0
| Status: failed
| Version: 2.2-1~bpo8+1
| 

| Finished at 20151126-1550
| Build needed 00:00:00, 0k disc space
| E: Package build dependencies not satisfied; skipping

that's why I don't believe you built it in jessie. python-sphinx is not 
backported
and you didn't mention any patches

solutions:
1) backport python-sphinx, or
2) use patch from Ubuntu: 
https://patches.ubuntu.com/r/routes/routes_2.2-1ubuntu1.patch


now that you have working package, you can test it in jessie. After
upload, please subscribe routes' bugs.

Re: Pushing a bunch of packages to jessie-backports

[Thomas Goirand]

On 11/25/2015 12:07 PM, Piotr Ożarowski wrote:
>>> debdiff of what you want to upload + time for me to check it (a week, I
>>> will respond most probably in 24h, though)
>>
>> I'll have to repeat myself then: there's no change but an entry in
> 
> I will not repeat myself. I already wrote why I think you should do
> that.

I've uploaded all what I prepared to this URL:
http://sid.gplhost.com/bpo/

Re: Pushing a bunch of packages to jessie-backports

[Thomas Goirand]

On 11/26/2015 09:32 AM, Piotr Ożarowski wrote:
> I hope he
> will realize one package will FTBFS in jessie

Instead of playing such a game, would you mind telling which package
you're talking about?

Or is it more fun that you just keep the issue for yourself, and then
use it as a mockery, just to prove yourself right and show how bad of a
package maintainer I am?

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

> Note that regarding bug reports, the backports website states:
> 
> "Please report bugs that you found in the packages to the backports
> mailinglist  and *NOT* to the
> Debian BTS!"

so why do I get them? DAK sends emails to me too and it doesn't mention
backports AT ALL! How can I distinct hostile uploads (hijacks) and
uploads from broken tools (backports.debian.org)?

anyway, I admitted in 20151123205458.gb16...@sar0.p1otr.com that it's
only a "best practice" so I cannot force zigo to do the right thing.
If he decides to upload broken package, I cannot do anything.
That doesn't change the fact that I can try to convince him that evil
Piotr sometimes has good intentions, and even if he's able to generate
debdiff (out of broken packages) then there's something wrong with his
CI/sbuild setup and we'll figure that out. He will also subscribe bugs
from all these packages just to shut up this asshole (and that's a good
outcome, even if I had to be rude; being nice doesn't work with him, I
tried that for few years, he just ignores you)
-- 
evil general Piotr

Re: Pushing a bunch of packages to jessie-backports

[Tristan Seligmann]

On Thu, 26 Nov 2015 at 10:32 Piotr Ożarowski  wrote:

> > This is not the policy for backports. Amend the policy if you
> > disagree. It's not fair to kick Zigo because he didn't follow the rules
> > and consider oneself above them when he respects them.
>
> fix it yourself, I'm getting all the bugs and emails so there's nothing
> that suggest I should use backport's policy rather then Debian's.
>

Note that regarding bug reports, the backports website states:

"Please report bugs that you found in the packages to the backports
mailinglist  and *NOT* to the
Debian BTS!"

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

[IOhannes m zmölnig (Debian/GNU), 2015-11-26]
> i think the kick in question is not your disapproval of the current
> affair, but referring to past action <560c3acd.9000...@debian.org>.

then I'm glad I did that. There's a change already (we had a mail! a
mail BEFORE an upload!). I'm still not happy that it simply announces
actions with all the old problems ("I will do something that I need, you
deal with consequences")
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: Pushing a bunch of packages to jessie-backports

[Debian/GNU]

On 2015-11-26 09:32, Piotr Ożarowski wrote:
> PS I didn't "kick" him, 

i think the kick in question is not your disapproval of the current
affair, but referring to past action <560c3acd.9000...@debian.org>.

fgmasdr
IOhannes

Re: Pushing a bunch of packages to jessie-backports

[Piotr Ożarowski]

> This is not the policy for backports. Amend the policy if you
> disagree. It's not fair to kick Zigo because he didn't follow the rules
> and consider oneself above them when he respects them.

fix it yourself, I'm getting all the bugs and emails so there's nothing
that suggest I should use backport's policy rather then Debian's.

PS I didn't "kick" him, I told him to send me debdiff because I hope he
will realize one package will FTBFS in jessie. He's sure everything
works now so he will not fix it and I will get bug report (I'm sure as
hell he will not subscribe bugs from all these packages he plans to
upload)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: Pushing a bunch of packages to jessie-backports

[Vincent Bernat]

 ❦ 25 novembre 2015 00:00 +0100, Piotr Ożarowski  :

>> > PS I'm getting all the emails from backports even if I wasn't the one
>> > who uploaded so this is official after all.
>> 
>> This isn't new, and it's been discussed at large. Basically, we would
>> all like the BTS to have the feature to select which is the maintainer
>> for a given suite. It is supposed to be fixed for the PPA^WBikeshed
>> support. In the mean time, we have to deal with it.
>
> until this is fixed, I consider it official Debian archive and thus
> any upload of *my* package that I do not approve: a hijack

This is not the policy for backports. Amend the policy if you
disagree. It's not fair to kick Zigo because he didn't follow the rules
and consider oneself above them when he respects them.
-- 
Don't over-comment.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature