AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
> Von: Piotr Ożarowski [mailto:pi...@debian.org] > > [Fiedler Roman, 2016-06-09] > > > is AMiner and AMinerRemoteControl symlinked in /usr/bin/? > > > (you can use debian/logdata-anomaly-miner.links to do that) > > > > Currently they are not symlinked. I took the same approach as with > > e.g. /usr/lib/apt/methods/http or /usr/lib/eject/dmcrypt-get-device > > where those binaries are usually not intended to be called by user > > directly. Meanwhile this has changed, e.g. AMinerRemoteControl is more > > or less now OK for cmd-line use. So I would add the symlinks. Or would > > moving of the files be more Debianic? > > dh_python2 doesn't add interpreter to Depends for scripts that are not > in $PATH. That's why lintian complains. You can: > * symlink them in /usr/bin/ (or any other appropriate path) I did that, modification: $ cat debian/logdata-anomaly-miner.links /usr/lib/logdata-anomaly-miner/AMiner /usr/bin/AMiner /usr/lib/logdata-anomaly-miner/AMinerRemoteControl /usr/bin/AMinerRemoteControl > * drop executable bit from these scripts, if they're not meant to be > execute (IIRC lintian ignores shebangs in non-executable files) not an option > * or if they're meant to be executed, but you don't want them to be > available out of the box, add "python" to Suggests or Recommends There are some usecases for users to call them, so let's have them available also. Still lintian will complain about this file, current header: #!/usr/bin/python2 -BEsSt But as patching that to 2.7 will make the warnings go away, from my point of view, that workaround is sufficient for me. New package is uploaded. Are there any other open points I shall address? smime.p7s Description: S/MIME cryptographic signature
Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
[Fiedler Roman, 2016-06-09] > > is AMiner and AMinerRemoteControl symlinked in /usr/bin/? > > (you can use debian/logdata-anomaly-miner.links to do that) > > Currently they are not symlinked. I took the same approach as with > e.g. /usr/lib/apt/methods/http or /usr/lib/eject/dmcrypt-get-device > where those binaries are usually not intended to be called by user > directly. Meanwhile this has changed, e.g. AMinerRemoteControl is more > or less now OK for cmd-line use. So I would add the symlinks. Or would > moving of the files be more Debianic? dh_python2 doesn't add interpreter to Depends for scripts that are not in $PATH. That's why lintian complains. You can: * symlink them in /usr/bin/ (or any other appropriate path) * drop executable bit from these scripts, if they're not meant to be executed... (IIRC lintian ignores shebangs in non-executable files) * or if they're meant to be executed, but you don't want them to be available out of the box, add "python" to Suggests or Recommends -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
> Von: Piotr Ożarowski [mailto:pi...@debian.org] > > [Fiedler Roman, 2016-06-09] > > > * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick > > > it up without any overrides, additional options, etc. > > > > So I guess, before that (where dh_python2 refused to operate at all), > > using /usr/lib/aminer as a shorthand - as the package name is quite > > long - was a bad idea. > > NAME in /usr/lib/NAME is not used in Python¹. It's not module name, you > cannot import it so Python module naming rules do not apply here. > > [¹] at least in /usr/lib/NAME/foo scripts OK, I understand. So for simplicity it seems to be best when NAME==[full-package-name], already changed in newest mentors upload. > > As import names cannot contain a dash, is use of > > "usr/lib/logdata-anomaly-miner/aminer" as module storage directory and > > "aminer" as shorthand module name in imports OK? > > if /usr/lib/logdata-anomaly-miner is the storage and > /usr/lib/logdata-anomaly-miner/aminer is the module dir (with > __init__.py file), then yes Confirmed, that's exactly how it works now. > > Still when building, lintian is unhappy with the output package: > > > > E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata- > anomaly-miner/AMiner > > E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata- > anomaly-miner/AMinerRemoteControl > > is AMiner and AMinerRemoteControl symlinked in /usr/bin/? > (you can use debian/logdata-anomaly-miner.links to do that) Currently they are not symlinked. I took the same approach as with e.g. /usr/lib/apt/methods/http or /usr/lib/eject/dmcrypt-get-device where those binaries are usually not intended to be called by user directly. Meanwhile this has changed, e.g. AMinerRemoteControl is more or less now OK for cmd-line use. So I would add the symlinks. Or would moving of the files be more Debianic? > PS please don't CC me, I read debian-python@lists.debian.org Sorry about that. smime.p7s Description: S/MIME cryptographic signature
Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
[Fiedler Roman, 2016-06-09] > > * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick > > it up without any overrides, additional options, etc. > > So I guess, before that (where dh_python2 refused to operate at all), > using /usr/lib/aminer as a shorthand - as the package name is quite > long - was a bad idea. NAME in /usr/lib/NAME is not used in Python¹. It's not module name, you cannot import it so Python module naming rules do not apply here. [¹] at least in /usr/lib/NAME/foo scripts > As import names cannot contain a dash, is use of > "usr/lib/logdata-anomaly-miner/aminer" as module storage directory and > "aminer" as shorthand module name in imports OK? if /usr/lib/logdata-anomaly-miner is the storage and /usr/lib/logdata-anomaly-miner/aminer is the module dir (with __init__.py file), then yes > Still when building, lintian is unhappy with the output package: > > E: logdata-anomaly-miner: python-script-but-no-python-dep > usr/lib/logdata-anomaly-miner/AMiner > E: logdata-anomaly-miner: python-script-but-no-python-dep > usr/lib/logdata-anomaly-miner/AMinerRemoteControl is AMiner and AMinerRemoteControl symlinked in /usr/bin/? (you can use debian/logdata-anomaly-miner.links to do that) PS please don't CC me, I read debian-python@lists.debian.org -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
> Von: Piotr Ożarowski [mailto:pi...@debian.org]
>
> just a quick reply:
>
> * private dir is the right call, do not install into dist-packages
> (only "python-*" binary packages should install there)
OK, done.
> * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
> it up without any overrides, additional options, etc.
So I guess, before that (where dh_python2 refused to operate at all), using
/usr/lib/aminer as a shorthand - as the package name is quite long - was a bad
idea.
As import names cannot contain a dash, is use of
"usr/lib/logdata-anomaly-miner/aminer" as module storage directory and "aminer"
as shorthand module name in imports OK?
> * use dh --with python2 or dh_python2 (/usr/bin/dh_python2) but never
> ever depend on internal implementation detail. I will change
> /usr/share/dh-python/dh_python2 file name just to mess with you ;-P
OK. Understood. Using the "usr/lib/logdata-anomaly-miner" layout for python
stuff as recommended above, dh_python2 is more happy with the situation: It
would complain about "python:Versions" substitution in
./debian/control:XB-Python-Version: ${python:Versions}
but XB-.. should be deprecated anyway. I dropped also " XS-Python-Version"
(which was hardcoded to >=2.6), as substitution would not work here also.
Still when building, lintian is unhappy with the output package:
E: logdata-anomaly-miner: python-script-but-no-python-dep
usr/lib/logdata-anomaly-miner/AMiner
E: logdata-anomaly-miner: python-script-but-no-python-dep
usr/lib/logdata-anomaly-miner/AMinerRemoteControl
Since the situation became a lot of clearer using your suggestions, I will
again try to tackle the lintian issue.
Roman
> --
> Piotr Ożarowski Debian GNU/Linux Developer
> www.ozarowski.pl www.griffith.cc www.debian.org
> GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
smime.p7s
Description: S/MIME cryptographic signature
AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
> Von: Gianfranco Costamagna [mailto:locutusofb...@debian.org] > Hi, > > (answering where I can!) >> Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows >> dh_python2 to extract the version information: >> >> Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz, >> init-system-helpers (>= 1.18~) >> >> (Remark: Is there any reason to restrict the versions to >=2.7.5? The tools >> should have compatibility with >=2.6 and I would expect the "Depends" >> section >> to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and >> DEBPYTHON_DEFAULT intended to be used to fix that?) > > that stuff is built for unstable/testing, so the variables are filled > with the current python situation > (2.6 is only on old-stable, and Stretch has 2.7.11 already). > > Probably if you try to build it with a jessie chroot, you get different > values, and this is correct. > (you can't in general install deb built against stretch into wheezy/jessie, > without > breaking stuff, this is why some dependencies are not too relaxed, to avoid > people > doing that, but instead upgrading the minimal set of packages to make sure > things will work after the apt-pinning). > > I don't foresee any issue here, because even in case of a backport, the > package will need to be rebuilt on top of that. OK, so I will not attempt to fiddle with that and stick to the stricter versions. > >But doing the move, lintian will not like the produced package any more: > > > >E: logdata-anomaly-miner: python-script-but-no-python-dep > >usr/lib/aminer/AMiner > > I can't say, I don't see the package installing stuff in usr/lib/python* on > mentors Maybe the hints from Piotr Ożarowski's followup mail will fix that, using his hints. > >The rationale behind not putting aminer into dist-packages (and removing > >dist-packages from python-path) was: >> [Snip] > > >b) prepare against possible future risks due to accidental loading (call to > >__init__) of other packages residing in dist-packages, that may give rise to > >privilege escalation (like the GNU-TLS CVE from this/last week) > > mmm you want to avoid people importing your library? > > ok Not directly avoid, but make them think more about it before adding. I also remember having issues with some svg-python library, that starts to misbehave as soon as other packages are installed because of "try: import xxx; ..." blocks. Just want to reduce the risks. > >Of course, it should be possible to move the code to > >/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, > but > >this could make it more "sexy" for an admin to include whole dist-packages > in > >python path again. > >In that light, should the code be moved? > > leaving to other folks the answer > >Apart from that, this will also make it harder to use the same codebase for > >both python2.6/python2.7, but that should be fixable by providing more > >patches. > > you can't use python2.6 on Stretch, so no issue here. Understood, dropping the 2.6/2.7 compatibility stuff > >> >-#!/usr/bin/python2 -BEsSt > >> >+#!/usr/bin/python2.7 -BEsSt > >> > >> this should be also handled by dh_python2 AFAIK > > > >Even with move dh_python2 does not touch the files. The only difference is, > >that lintian does not like the files any more. > > can you please double check with the documentation? > https://wiki.debian.org/Python/LibraryStyleGuide The "python2" selection in the binaries was effect of reading it, but perhaps I got something wrong: Intro: "It (the LibraryStyleGuide) is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it" And from " Debian Python Policy" https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html#s-interpreter_name "Python scripts that require the default Python 2 version should specify python2 as the interpreter name." So should be OK in my opinion. > [Snip] smime.p7s Description: S/MIME cryptographic signature
Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
just a quick reply: * private dir is the right call, do not install into dist-packages (only "python-*" binary packages should install there) * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick it up without any overrides, additional options, etc. * use dh --with python2 or dh_python2 (/usr/bin/dh_python2) but never ever depend on internal implementation detail. I will change /usr/share/dh-python/dh_python2 file name just to mess with you ;-P -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
Hi, (answering where I can!) >Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows >dh_python2 to extract the version information: > >Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz, >init-system-helpers (>= 1.18~) > >(Remark: Is there any reason to restrict the versions to >=2.7.5? The tools >should have compatibility with >=2.6 and I would expect the "Depends" section >to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and DEBPYTHON_DEFAULT >intended to be used to fix that?) that stuff is built for unstable/testing, so the variables are filled with the current python situation (2.6 is only on old-stable, and Stretch has 2.7.11 already). Probably if you try to build it with a jessie chroot, you get different values, and this is correct. (you can't in general install deb built against stretch into wheezy/jessie, without breaking stuff, this is why some dependencies are not too relaxed, to avoid people doing that, but instead upgrading the minimal set of packages to make sure things will work after the apt-pinning). I don't foresee any issue here, because even in case of a backport, the package will need to be rebuilt on top of that. >But doing the move, lintian will not like the produced package any more: > >E: logdata-anomaly-miner: python-script-but-no-python-dep >usr/lib/aminer/AMiner I can't say, I don't see the package installing stuff in usr/lib/python* on mentors >The rationale behind not putting aminer into dist-packages (and removing >dist-packages from python-path) was: > >a) require the aminer-admin to "link" in each dist-packages module separately, >thus reminding it, that it is no good idea to include large amounts of third >party libraries in security-critical code. Code volume is also bugs and >attack-surface. I didn't get completely this, but I leave other people to answer >b) prepare against possible future risks due to accidental loading (call to >__init__) of other packages residing in dist-packages, that may give rise to >privilege escalation (like the GNU-TLS CVE from this/last week) mmm you want to avoid people importing your library? ok >Of course, it should be possible to move the code to >/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, but >this could make it more "sexy" for an admin to include whole dist-packages in >python path again. >In that light, should the code be moved? leaving to other folks the answer >Apart from that, this will also make it harder to use the same codebase for >both python2.6/python2.7, but that should be fixable by providing more >patches. you can't use python2.6 on Stretch, so no issue here. >> >-#!/usr/bin/python2 -BEsSt >> >+#!/usr/bin/python2.7 -BEsSt >> >> this should be also handled by dh_python2 AFAIK > >Even with move dh_python2 does not touch the files. The only difference is, >that lintian does not like the files any more. can you please double check with the documentation? https://wiki.debian.org/Python/LibraryStyleGuide >So it would be OK to place the files in usr/lib/python2.7/dist-packages and >add a symlink to usr/lib/python2.6/dist-packages? no, python2.6 is dead, no need to add it here. >I would not like to push for that. Let's take the time and learn. Also this >package might be the template for other security-tool packages, so better have >it clean before cloning. thanks for understanding! >Understood. It might be, that this is all not a big issue for the python-dev >>pros, might be sorted out quickly. I hope so, I understand the package is doing some non-standard things, so trying to make it python-library style might be even a bad idea in general, this is why I would prefer to have some double/triple checks here ;) thanks for the nice email! G.
Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis
Hi Gianfranco, hello Python devs,
Introduction for debian-python members: Gianfranco is giving me great
assistance in the mentoring process to get the logdata-anomaly-miner package
included to Debian. There were some issues, we are not completely sure, how to
sort them out, any help on that would be appreciated!
Sorry for drawing your attention to that in the middle of the running
discussion. You can find the whole discussion in the ITP bug report
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813096 . Perhaps reading the
whole issue would not justify the time spent, so from my opinion most
important 2 topics and related questions are:
* Rationale not using "usr/lib/python*/": See this mail, first inline reply
section. Q: Are the arguments relevant? Move code or not?
* dh_python2 woes: Beginning of it, see Bugreport " Fri, 3 Jun 2016 18:54:16
+". Idea was to share as much code as possible between python2.6/python2.7
systems, but somehow this does not work out with packaging/lintian. Q: is
building a 2.6/2.7 package even a good idea or should it be avoided? Instead
upstream code and Debian packaging rules/patches could be optimized for
generating different packages for those versions.
The package in question can be found at
https://mentors.debian.net/package/logdata-anomaly-miner
> Von: Gianfranco Costamagna [mailto:locutusofb...@debian.org]
> Hi
>
> >I had already added it already to
> >* control
> >Depends: ${python:Depends}, python-tz, ${misc:Depends}
>
> >which was mangled (with warning) to:
>
> maybe you are installing the files in usr/lib/foo, instead of
> usr/lib/python*/
> and then dh_python2 is not acting correctly?
Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows
dh_python2 to extract the version information:
Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz,
init-system-helpers (>= 1.18~)
(Remark: Is there any reason to restrict the versions to >=2.7.5? The tools
should have compatibility with >=2.6 and I would expect the "Depends" section
to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and DEBPYTHON_DEFAULT
intended to be used to fix that?)
But doing the move, lintian will not like the produced package any more:
E: logdata-anomaly-miner: python-script-but-no-python-dep
usr/lib/aminer/AMiner
> I don't think installing python stuff outside that directory is a good
> idea...
The rationale behind not putting aminer into dist-packages (and removing
dist-packages from python-path) was:
a) require the aminer-admin to "link" in each dist-packages module separately,
thus reminding it, that it is no good idea to include large amounts of third
party libraries in security-critical code. Code volume is also bugs and
attack-surface.
b) prepare against possible future risks due to accidental loading (call to
__init__) of other packages residing in dist-packages, that may give rise to
privilege escalation (like the GNU-TLS CVE from this/last week)
Of course, it should be possible to move the code to
/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, but
this could make it more "sexy" for an admin to include whole dist-packages in
python path again.
In that light, should the code be moved?
Apart from that, this will also make it harder to use the same codebase for
both python2.6/python2.7, but that should be fixable by providing more
patches.
> >@@ -1,4 +1,4 @@
> >-#!/usr/bin/python2 -BEsSt
> >+#!/usr/bin/python2.7 -BEsSt
>
> this should be also handled by dh_python2 AFAIK
Even with move dh_python2 does not touch the files. The only difference is,
that lintian does not like the files any more.
> >Everything seems to work without it also and the same package could have
> been
> >used for python2.6 and python2.7 systems but lintian does not understand
> that.
>
> well, installing into usr/lib/python* should fix that issue
So it would be OK to place the files in usr/lib/python2.7/dist-packages and
add a symlink to usr/lib/python2.6/dist-packages?
> I would prefer you to ask on debian-python about this issue, because I'm
> wondering about something bad that is just hidden and will be spot when the
> package
> enters the
> archive.
>
> having an ack for a more python-savvy person would be great
> (I could even sponsor the package as-is, but I'm not too confident with
> that)
I would not like to push for that. Let's take the time and learn. Also this
package might be the template for other security-tool packages, so better have
it clean before cloning.
> >I hope this fixed all the review comments for regarding the packaging. I
> would
> >like to keep the deeper restructuring of upstream code + Debian packaging
> >scripts for upstream/Debian release V0.0.3. Is that OK from Debian side?
>
> it is ok to have new uploads, don't worry about that, I just would like to
> have a
> feedback from another DD before uploading/finish the review.
>
> is it possible for you?
>
