Re: Packaging pythonpy

[Tiago Ilieve]

Dmitry,

On 31 March 2016 at 15:54, Dmitry Shachnev  wrote:
> Cool, one little change and it's much better!

Actually this has one big implication: as it uses the same interpreter
to evaluate its input, it becomes compatible with Python 3 expressions
only. Noticed that after trying to access "string.letters" which was
renamed to "string.ascii_letters".

Added a note to the manpage[1].

> Now that you've changed to Python 3, you need to change the shebang from
> python2 to python3 anyway…

That's true.

> This command works for me (as a DEP-8 test):
>
>   python3 ./test/test_pythonpy.py -v
>
> Unfortunately dh runs tests before install/link phase, so we cannot run it
> during build (without ugly hacks). Though if upstream adds an option to
> specify path to executable used by tests, it will help a lot.

I'll leave this upstream contribution as a future improvement, OK?
This way we can get the package going in its current state.

> Ok, if you add the DEP-8 test (just one 3-lines file), I'll sponsor it.
> (If you prefer to wait for a new upstream release, I don't mind of course.)

Awesome news! I've imported[2] the new upstream version and added
DEP-8 tests[3]. The updated package is on mentors.d.n[4] (with no
warnings at all) and its source was pushed to collab-maint[5].

> One small suggestion: if you are using pybuild, you can simplify the install
> command a bit:
>
>   dh_auto_install -- --install-args "--install-lib=/usr/share/pythonpy 
> --install-scripts=/usr/share/pythonpy"
>
> It's a bit shorter because you don't need to specify --install-layout and
> --root.

This is way simpler. Thanks for the tip. Done[6].

> Also, don't you think that /usr/bin/py is too generic? Maybe use
> /usr/bin/pythonpy or something similar instead? (It's easy to change that,
> as the linking is done by packaging rather than upstream buildsystem).

It is indeed pretty short and generic, but is the way upstream have
been distributing it since the beginning (the project is nearly two
years old), so I guess it would be a hassle if we change it right now.
I've triple-checked if it wasn't colliding with another binary name
and it isn't.

Also, if it is called "pythonpy" or something like that, would be
somewhat annoying to use completion, as every time someone hits
"pyt" it would end with "python" first.

On 31 March 2016 at 16:31, Florent Rougon  wrote:
> Especially considering that 'py' is the name chosen for the Python
> Launcher for Windows by upstream Python developers :

I wasn't aware of that. Thanks for point this out.

Regards,
Tiago.

[1]: https://github.com/myhro/deb-pythonpy/commit/bdb7aeb
[2]: https://github.com/myhro/deb-pythonpy/commit/093b63a
[3]: https://github.com/myhro/deb-pythonpy/commit/8f74394
[4]: http://mentors.debian.net/package/pythonpy
[5]: https://anonscm.debian.org/git/collab-maint/pythonpy.git
[6]: https://github.com/myhro/deb-pythonpy/commit/c6f0d79

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Florent Rougon]

Hello,

Dmitry Shachnev  wrote:

> Also, don't you think that /usr/bin/py is too generic? Maybe use
> /usr/bin/pythonpy or something similar instead? (It's easy to change that,
> as the linking is done by packaging rather than upstream buildsystem).

Especially considering that 'py' is the name chosen for the Python
Launcher for Windows by upstream Python developers :

  https://docs.python.org/3/using/windows.html#launcher

-- 
Florent

Re: Packaging pythonpy

[Dmitry Shachnev]

Hi Tiago,

On Wed, Mar 30, 2016 at 09:03:07PM -0300, Tiago Ilieve wrote:
>> Major nit about your package: did you read the Python policy, in particular
>> the part that tells that all new packages should use Python 3 [3]?
>
> Actually I had consulted it, but not read it entirely. I don't know
> why but I thought that the Python 3 requirement was a *nice to have*,
> not a *should have*. Anyway, I've updated[3] the build system to use
> Python 3.

Cool, one little change and it's much better!

>> Re Lintian error, this looks like a false positive. See [2].
>
> You mean that maybe it's better if we add an override instead of a workaround?

Now that you've changed to Python 3, you need to change the shebang from
python2 to python3 anyway…

> I noticed that the test suite wasn't being properly executed and sent
> a patch to upstream[4]. As soon as a new release is made with this
> changed integrated, I'll be adding support for DEP-8 (as suggested by
> Barry Warsaw[5] in the last week), as they are functional tests that
> depends on the package being installed.
>
> I've uploaded the updated package to mentors.d.n[6], but I guess its
> better to wait for a new release integrating the test suite fixes. The
> upstream is pretty fast and responsive.

This command works for me (as a DEP-8 test):

  python3 ./test/test_pythonpy.py -v

Unfortunately dh runs tests before install/link phase, so we cannot run it
during build (without ugly hacks). Though if upstream adds an option to
specify path to executable used by tests, it will help a lot.

> Are you able to sponsor the upload when we finish taking care of those
> details? I've filled an RFS (#819289[7]), but forgot to add the
> "debian-python" mailing list in "X-Debbugs-CC".

Ok, if you add the DEP-8 test (just one 3-lines file), I'll sponsor it.
(If you prefer to wait for a new upstream release, I don't mind of course.)

One small suggestion: if you are using pybuild, you can simplify the install
command a bit:

  dh_auto_install -- --install-args "--install-lib=/usr/share/pythonpy 
--install-scripts=/usr/share/pythonpy"

It's a bit shorter because you don't need to specify --install-layout and
--root.

Also, don't you think that /usr/bin/py is too generic? Maybe use
/usr/bin/pythonpy or something similar instead? (It's easy to change that,
as the linking is done by packaging rather than upstream buildsystem).

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Re: Packaging pythonpy

[Tiago Ilieve]

Dmitry,

On 30 March 2016 at 16:48, Dmitry Shachnev  wrote:
> Looks like I was a bit mistaken — dh_python2 will not replace shebangs for
> files in /usr/share. But then you can do this manually using a sed call [1],
> that is still easier than a patch.

This is indeed way clever than an entire patch to fix something so simple. :-)

Done[1].

> Re Lintian error, this looks like a false positive. See [2].

You mean that maybe it's better if we add an override instead of a workaround?

> Minor nit about your package: please build-depend on dh-python to get a modern
> version of dh_python2.

Done[2].

> Major nit about your package: did you read the Python policy, in particular
> the part that tells that all new packages should use Python 3 [3]?

Actually I had consulted it, but not read it entirely. I don't know
why but I thought that the Python 3 requirement was a *nice to have*,
not a *should have*. Anyway, I've updated[3] the build system to use
Python 3.

I noticed that the test suite wasn't being properly executed and sent
a patch to upstream[4]. As soon as a new release is made with this
changed integrated, I'll be adding support for DEP-8 (as suggested by
Barry Warsaw[5] in the last week), as they are functional tests that
depends on the package being installed.

I've uploaded the updated package to mentors.d.n[6], but I guess its
better to wait for a new release integrating the test suite fixes. The
upstream is pretty fast and responsive.

Are you able to sponsor the upload when we finish taking care of those
details? I've filled an RFS (#819289[7]), but forgot to add the
"debian-python" mailing list in "X-Debbugs-CC".

Regards,
Tiago.

[1]: https://github.com/myhro/deb-pythonpy/commit/f4ce711
[2]: https://github.com/myhro/deb-pythonpy/commit/868667b
[3]: https://github.com/myhro/deb-pythonpy/commit/3c2f4bd
[4]: https://github.com/Russell91/pythonpy/pull/79
[5]: https://lists.debian.org/debian-python/2016/03/msg00101.html
[6]: http://mentors.debian.net/package/pythonpy
[7]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819289

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Dmitry Shachnev]

Hi Tiago,

On Wed, Mar 30, 2016 at 05:26:05AM -0300, Tiago Ilieve wrote:
> On 29 March 2016 at 18:40, Dmitry Shachnev  wrote:
> > You do not need a patch for this kind of thing. Just pass
> > --shebang=/usr/bin/whatever to dh_python2 call in your debian/rules.
> >
> > Also, for debian/packages, /usr/bin/pythonX is preferred over /usr/bin/env
> > pythonX.
>
> Thanks for the tip. I've tried to remove the referred patch and added
> "--shebang=/usr/bin/python" to the already existing
> "override_dh_python2". The problem is that with this modification the
> error "python-script-but-no-python-dep" comes up again. :-(

Looks like I was a bit mistaken — dh_python2 will not replace shebangs for
files in /usr/share. But then you can do this manually using a sed call [1],
that is still easier than a patch.

Re Lintian error, this looks like a false positive. See [2].

Minor nit about your package: please build-depend on dh-python to get a modern
version of dh_python2.

Major nit about your package: did you read the Python policy, in particular
the part that tells that all new packages should use Python 3 [3]?

[1] sed -i 's,^#!/usr/bin/.*,#!/usr/bin/python,' 
debian/pythonpy/usr/share/pythonpy/pythonpy/*.py
[2] https://bugs.debian.org/743599
[3] https://www.debian.org/doc/packaging-manuals/python-policy/ch-python3.html

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Re: Packaging pythonpy

[Tiago Ilieve]

Hi Dmitry,

On 29 March 2016 at 18:40, Dmitry Shachnev  wrote:
> You do not need a patch for this kind of thing. Just pass
> --shebang=/usr/bin/whatever to dh_python2 call in your debian/rules.
>
> Also, for debian/packages, /usr/bin/pythonX is preferred over /usr/bin/env
> pythonX.

Thanks for the tip. I've tried to remove the referred patch and added
"--shebang=/usr/bin/python" to the already existing
"override_dh_python2". The problem is that with this modification the
error "python-script-but-no-python-dep" comes up again. :-(

Regards,
Tiago.

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Dmitry Shachnev]

Hi,

On Fri, Mar 25, 2016 at 09:00:46PM -0300, Tiago Ilieve wrote:
> Tried one or two new approaches and it didn't worked. In the I've
> created a patch[1] changing "#!/usr/bin/env python2" to
> "#!/usr/bin/env python".

You do not need a patch for this kind of thing. Just pass
--shebang=/usr/bin/whatever to dh_python2 call in your debian/rules.

Also, for debian/packages, /usr/bin/pythonX is preferred over /usr/bin/env
pythonX.

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Re: Packaging pythonpy

[Tiago Ilieve]

Gianfranco,

On 25 March 2016 at 19:07, Gianfranco Costamagna
 wrote:
> up to your sponsor :)

Tried one or two new approaches and it didn't worked. In the I've
created a patch[1] changing "#!/usr/bin/env python2" to
"#!/usr/bin/env python". This should work as long as Python 2 is the
default interpreter, something which may change in the next years, but
isn't a problem at least for Stretch.

I'm all in for another options if someone doesn't like this patch.

> swap Files: debian/*
> and Files: *
>
> first the more comprehensive and later the less.
> (lintian might be more specific)

This is awesome. I would never figure out by myself that it was so
simple to fix. :-)

> I did fix the python apps in blah section with section "utils", and uploaded 
> on debomatic again.
> Now that lintian warning is not there anymore.

Yup. I've did that as well[2].

> (I won't download the package, I think I already answered the points)

No problem. You answers were very helpful, as always.

I've uploaded a new version of the package to mentors.d.n[3]. There
are the following lintian messages now:

* "newer-standards-version": which can be ignored, as mentors.d.n
doesn't consider 3.9.7 as the current standard.
* "debian-watch-file-is-missing" and "no-upstream-changelog": which
will be fixed in the near future with upstream help regarding tagged
releases.
* "binary-without-manpage": which I'll be fixing, adding a manpage
before submitting an RFS.

Thank you very much, Gianfranco!

Regards,
Tiago.

[1]: https://github.com/myhro/deb-pythonpy/commit/5450656
[2]: https://github.com/myhro/deb-pythonpy/commit/0e2d987
[3]: http://mentors.debian.net/package/pythonpy

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Gianfranco Costamagna]

Hi,
>I really appreciate your effort in trying to package it yourself, but

>you didn't solved the main problem, which is the
>"python-script-but-no-python-dep".


indeed, control file line two.

it is an application, so choose some Section from there
https://qa.debian.org/developer.php?login=python-apps-team%40lists.alioth.debian.org

Section: python is for libraries (e.g. called python-foo or python3-foo)


>The "dh_auto_install" override is used to place it in>"/usr/share/pythonpy" 
>which is the proper place for Python
>*applications*[1]. Without it, it goes to the place where *libraries*
>should be located.


ok
>The "remove_entry_points_scripts.patch" avoids the creation of
>py{2,2.7} binaries that aren't needed. Without it and removing the
>override for "dh_fixperms", the package becomes useless. There's no
>way to call the "py" command, as its main script doesn't have
>execution permissions.


ok, but why some of them have the +x set?
http://debomatic-amd64.debian.net/distribution#unstable/pythonpy/0.4.4-1/contents

I wouls suggest patching the setup file to perform correctly instead of 
overriding stuff.

>Looks like it would be way easier to fix the entry point scripts to
>created a binary named "py", avoiding just the other ones. We can also
>ignore the override that changes the target folder, but doing this
>feels wrong, is like we are ignoring the best practices for packaging
>Python applications. That's why I'm wrecking my head with this issue,
>removing every file that would be useless, instead of following the
>easiest path.


up to your sponsor :)

>About the lintian output:



>* "unused-file-paragraph-in-dep5-copyright": this info doesn't appear
>even when I run lintian with the same arguments on my machine. This is
>strange, as I'm running "v2.5.42.1" from sid and debomatic-amd64.d.n
>is using "v2.5.42.1~bpo8+1", which should be the same version. Do you
>know how can I do this?


swap Files: debian/*
and Files: *

first the more comprehensive and later the less.
(lintian might be more specific)


>* "debian-watch-file-is-missing": this is right. I've asked[2]>upstream to tag 
>every release on GitHub, so we can fetch information
>about new versions from there.
>* "application-in-library-section": fixed[3].
>* "no-upstream-changelog": the upstream added a changelog file in the
>last version (0.4.9, which I have packaged this afternoon), but it
>doesn't comes with the tarball available in PyPI. This will be solved
>when the releases are tagged and we grab them from GitHub.
>* "package-installs-into-obsolete-dir": fixed using dh_bash-completion[4].
>
>I've uploaded the last (0.4.9-1) package version to mentors.d.n[5].


I did fix the python apps in blah section with section "utils", and uploaded on 
debomatic again.

Now that lintian warning is not there anymore.


(I won't download the package, I think I already answered the points)

cheers,

G.

Re: Packaging pythonpy

[Tiago Ilieve]

Hi Gianfranco,

On 25 March 2016 at 16:21, Gianfranco Costamagna
 wrote:
> http://debomatic-amd64.debian.net/distribution#unstable/pythonpy/0.4.4-1/lintian
> please dget it from there and start again :)
>
> I fixed a lot of issues, and many more are there now!

I really appreciate your effort in trying to package it yourself, but
you didn't solved the main problem, which is the
"python-script-but-no-python-dep".

The "dh_auto_install" override is used to place it in
"/usr/share/pythonpy" which is the proper place for Python
*applications*[1]. Without it, it goes to the place where *libraries*
should be located.

The "remove_entry_points_scripts.patch" avoids the creation of
py{2,2.7} binaries that aren't needed. Without it and removing the
override for "dh_fixperms", the package becomes useless. There's no
way to call the "py" command, as its main script doesn't have
execution permissions.

Looks like it would be way easier to fix the entry point scripts to
created a binary named "py", avoiding just the other ones. We can also
ignore the override that changes the target folder, but doing this
feels wrong, is like we are ignoring the best practices for packaging
Python applications. That's why I'm wrecking my head with this issue,
removing every file that would be useless, instead of following the
easiest path.

About the lintian output:

* "unused-file-paragraph-in-dep5-copyright": this info doesn't appear
even when I run lintian with the same arguments on my machine. This is
strange, as I'm running "v2.5.42.1" from sid and debomatic-amd64.d.n
is using "v2.5.42.1~bpo8+1", which should be the same version. Do you
know how can I do this?
* "debian-watch-file-is-missing": this is right. I've asked[2]
upstream to tag every release on GitHub, so we can fetch information
about new versions from there.
* "application-in-library-section": fixed[3].
* "no-upstream-changelog": the upstream added a changelog file in the
last version (0.4.9, which I have packaged this afternoon), but it
doesn't comes with the tarball available in PyPI. This will be solved
when the releases are tagged and we grab them from GitHub.
* "package-installs-into-obsolete-dir": fixed using dh_bash-completion[4].

I've uploaded the last (0.4.9-1) package version to mentors.d.n[5].

Thanks,
Tiago.

[1]: https://wiki.debian.org/Python/Packaging#Example_2:_Python_application
[2]: https://github.com/Russell91/pythonpy/issues/76
[3]: https://github.com/myhro/deb-pythonpy/commit/0e2d987
[4]: https://github.com/myhro/deb-pythonpy/commit/954e424
[5]: http://mentors.debian.net/package/pythonpy

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Tiago Ilieve]

Hi Gianfranco,

On 25 March 2016 at 06:25, Gianfranco Costamagna
 wrote:
>
> adding python to the dependencies?
> do you have python to build dependencies, dh-python and python:Depends on the
> binary package?

This is what is strange. I've made the following changes:

-
diff --git a/debian/control b/debian/control
index f0c1b3f..5205298 100644
--- a/debian/control
+++ b/debian/control
@@ -3,6 +3,7 @@ Section: python
 Priority: optional
 Maintainer: Tiago Ilieve 
 Build-Depends: debhelper (>= 9),
+   dh-python,
python (>= 2.7.3),
python-setuptools (>= 0.6.24)
 Standards-Version: 3.9.7
@@ -13,7 +14,9 @@ Vcs-Browser: https://github.com/myhro/deb-pythonpy

 Package: pythonpy
 Architecture: all
-Depends: ${misc:Depends}, ${python:Depends}
+Depends: python (>= 2.7.3),
+ ${misc:Depends},
+ ${python:Depends}
 Description: 'python -c', with tab completion and shorthand
  pythonpy is an utility that facilitates the usage of Python one-liners. The
  command 'py' evaluates a string consisting of any Python expression. It can do
-

But this didn't helped at all. The lintian.d.o page for
"python-script-but-no-python-dep"[1] says:

"Packages with Python scripts should depend on the package python.
Those with scripts that specify a specific version of Python must
depend on that version of Python (exactly).

For example, if a script in the package uses #!/usr/bin/python, the
package needs a dependency on python. If a script uses
#!/usr/bin/python2.6, the package needs a dependency on python2.6. A
dependency on python (>= 2.6) is not correct, since later versions of
Python may not provide the /usr/bin/python2.6 binary."

What is the "exactly" version of Python for a script which has
"#!/usr/bin/env python2" as its shebang?

> it is generated *during/after* the build, so the clean target won't work.
>
> a "package.pyremove" file might help, not sure
>
> codesearch.debian.net might help
> egg path:pyremove$
>
> https://codesearch.debian.net/results/egg%20path%3Apyremove%24/page_0

"pyremove" didn't worked, but in the same page that I found references
to it[2], there's a suggestion to override "dh_python", which is what
I did[3]. Thanks for the suggestion. :-)

Regards,
Tiago.

[1]: https://lintian.debian.org/tags/python-script-but-no-python-dep.html
[2]: 
https://wiki.debian.org/Python/LibraryStyleGuide#Python_3.3.2F3.4_unittest_fixers_for_2to3
[3]: https://github.com/myhro/deb-pythonpy/commit/68db18e

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Gianfranco Costamagna]

Hi Tiago,



>* How to fix the "python-script-but-no-python-dep" lintian error? I've
>tried with and without pybuild and the error still persists.


adding python to the dependencies?
do you have python to build dependencies, dh-python and python:Depends on the 

binary package?


>* How to get rid of the ".egg-info/" folder that is being packaged?>Looks like 
>"debian/clean" rules aren't working.


it is generated *during/after* the build, so the clean target won't work.

a "package.pyremove" file might help, not sure

codesearch.debian.net might help
egg path:pyremove$

https://codesearch.debian.net/results/egg%20path%3Apyremove%24/page_0


cheers,

G.

There's a GitHub repository for the package[1]. I have intention on
asking for a repository on collab-maint when the package is ready (I
have write permissions to it).

Regards,
Tiago.

[1]: https://github.com/myhro/deb-pythonpy


-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

Re: Packaging pythonpy

[Tiago Ilieve]

Hi,

Can someone please help me on this one? I'm pretty close to finish the
initial packaging work. After fixing the following issues, it will be
a matter of adding a manpage and filling a RFS.

* How to fix the "python-script-but-no-python-dep" lintian error? I've
tried with and without pybuild and the error still persists.
* How to get rid of the ".egg-info/" folder that is being packaged?
Looks like "debian/clean" rules aren't working.

There's a GitHub repository for the package[1]. I have intention on
asking for a repository on collab-maint when the package is ready (I
have write permissions to it).

Regards,
Tiago.

[1]: https://github.com/myhro/deb-pythonpy

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil