AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Fiedler Roman
> Von: Piotr Ożarowski [mailto:pi...@debian.org]
> 
> [Fiedler Roman, 2016-06-09]
> > > is AMiner and AMinerRemoteControl symlinked in /usr/bin/?
> > > (you can use debian/logdata-anomaly-miner.links to do that)
> >
> > Currently they are not symlinked. I took the same approach as with
> > e.g. /usr/lib/apt/methods/http or /usr/lib/eject/dmcrypt-get-device
> > where those binaries are usually not intended to be called by user
> > directly. Meanwhile this has changed, e.g. AMinerRemoteControl is more
> > or less now OK for cmd-line use. So I would add the symlinks. Or would
> > moving of the files be more Debianic?
> 
> dh_python2 doesn't add interpreter to Depends for scripts that are not
> in $PATH. That's why lintian complains. You can:
> * symlink them in /usr/bin/ (or any other appropriate path)

I did that, modification:

$ cat debian/logdata-anomaly-miner.links 
/usr/lib/logdata-anomaly-miner/AMiner /usr/bin/AMiner
/usr/lib/logdata-anomaly-miner/AMinerRemoteControl /usr/bin/AMinerRemoteControl

> * drop executable bit from these scripts, if they're not meant to be
>   execute  (IIRC lintian ignores shebangs in non-executable files)

not an option

> * or if they're meant to be executed, but you don't want them to be
>   available out of the box, add "python" to Suggests or Recommends

There are some usecases for users to call them, so let's have them available 
also.

Still lintian will complain about this file, current header:

#!/usr/bin/python2 -BEsSt

But as patching that to 2.7 will make the warnings go away, from my point of 
view, that workaround is sufficient for me.

New package is uploaded.

Are there any other open points I shall address?


smime.p7s
Description: S/MIME cryptographic signature


Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Piotr Ożarowski
[Fiedler Roman, 2016-06-09]
> > is AMiner and AMinerRemoteControl symlinked in /usr/bin/?
> > (you can use debian/logdata-anomaly-miner.links to do that)
> 
> Currently they are not symlinked. I took the same approach as with
> e.g. /usr/lib/apt/methods/http or /usr/lib/eject/dmcrypt-get-device
> where those binaries are usually not intended to be called by user
> directly. Meanwhile this has changed, e.g. AMinerRemoteControl is more
> or less now OK for cmd-line use. So I would add the symlinks. Or would
> moving of the files be more Debianic?

dh_python2 doesn't add interpreter to Depends for scripts that are not
in $PATH. That's why lintian complains. You can:
* symlink them in /usr/bin/ (or any other appropriate path)
* drop executable bit from these scripts, if they're not meant to be
  executed...
  (IIRC lintian ignores shebangs in non-executable files)
* or if they're meant to be executed, but you don't want them to be
  available out of the box, add "python" to Suggests or Recommends
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Fiedler Roman
> Von: Piotr Ożarowski [mailto:pi...@debian.org]
> 
> [Fiedler Roman, 2016-06-09]
> > > * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
> > >   it up without any overrides, additional options, etc.
> >
> > So I guess, before that (where dh_python2 refused to operate at all),
> > using /usr/lib/aminer as a shorthand - as the package name is quite
> > long - was a bad idea.
> 
> NAME in /usr/lib/NAME is not used in Python¹. It's not module name, you
> cannot import it so Python module naming rules do not apply here.
> 
> [¹] at least in /usr/lib/NAME/foo scripts

OK, I understand. So for simplicity it seems to be best when 
NAME==[full-package-name], already changed in newest mentors upload.

> > As import names cannot contain a dash, is use of
> > "usr/lib/logdata-anomaly-miner/aminer" as module storage directory and
> > "aminer" as shorthand module name in imports OK?
> 
> if /usr/lib/logdata-anomaly-miner is the storage and
> /usr/lib/logdata-anomaly-miner/aminer is the module dir (with
> __init__.py file), then yes

Confirmed, that's exactly how it works now.

> > Still when building, lintian is unhappy with the output package:
> >
> > E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata-
> anomaly-miner/AMiner
> > E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata-
> anomaly-miner/AMinerRemoteControl
> 
> is AMiner and AMinerRemoteControl symlinked in /usr/bin/?
> (you can use debian/logdata-anomaly-miner.links to do that)

Currently they are not symlinked. I took the same approach as with e.g. 
/usr/lib/apt/methods/http or /usr/lib/eject/dmcrypt-get-device where those 
binaries are usually not intended to be called by user directly. Meanwhile this 
has changed, e.g. AMinerRemoteControl is more or less now OK for cmd-line use. 
So I would add the symlinks. Or would moving of the files be more Debianic?

> PS please don't CC me, I read debian-python@lists.debian.org

Sorry about that.


smime.p7s
Description: S/MIME cryptographic signature


Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Piotr Ożarowski
[Fiedler Roman, 2016-06-09]
> > * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
> >   it up without any overrides, additional options, etc.
> 
> So I guess, before that (where dh_python2 refused to operate at all),
> using /usr/lib/aminer as a shorthand - as the package name is quite
> long - was a bad idea.

NAME in /usr/lib/NAME is not used in Python¹. It's not module name, you
cannot import it so Python module naming rules do not apply here.

[¹] at least in /usr/lib/NAME/foo scripts

> As import names cannot contain a dash, is use of
> "usr/lib/logdata-anomaly-miner/aminer" as module storage directory and
> "aminer" as shorthand module name in imports OK?

if /usr/lib/logdata-anomaly-miner is the storage and
/usr/lib/logdata-anomaly-miner/aminer is the module dir (with
__init__.py file), then yes

> Still when building, lintian is unhappy with the output package:
> 
> E: logdata-anomaly-miner: python-script-but-no-python-dep 
> usr/lib/logdata-anomaly-miner/AMiner
> E: logdata-anomaly-miner: python-script-but-no-python-dep 
> usr/lib/logdata-anomaly-miner/AMinerRemoteControl

is AMiner and AMinerRemoteControl symlinked in /usr/bin/?
(you can use debian/logdata-anomaly-miner.links to do that)


PS please don't CC me, I read debian-python@lists.debian.org
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Fiedler Roman
> Von: Piotr Ożarowski [mailto:pi...@debian.org]
> 
> just a quick reply:
> 
> * private dir is the right call, do not install into dist-packages
>   (only "python-*" binary packages should install there)

OK, done.

> * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
>   it up without any overrides, additional options, etc.

So I guess, before that (where dh_python2 refused to operate at all), using 
/usr/lib/aminer as a shorthand - as the package name is quite long - was a bad 
idea.

As import names cannot contain a dash, is use of 
"usr/lib/logdata-anomaly-miner/aminer" as module storage directory and "aminer" 
as shorthand module name in imports OK?

> * use dh --with python2 or dh_python2 (/usr/bin/dh_python2) but never
>   ever depend on internal implementation detail. I will change
>   /usr/share/dh-python/dh_python2 file name just to mess with you ;-P

OK. Understood. Using the "usr/lib/logdata-anomaly-miner" layout for python 
stuff as recommended above, dh_python2 is more happy with the situation: It 
would complain about "python:Versions" substitution in

./debian/control:XB-Python-Version: ${python:Versions}

but XB-.. should be deprecated anyway. I dropped also " XS-Python-Version" 
(which was hardcoded to >=2.6), as substitution would not work here also.

Still when building, lintian is unhappy with the output package:

E: logdata-anomaly-miner: python-script-but-no-python-dep 
usr/lib/logdata-anomaly-miner/AMiner
E: logdata-anomaly-miner: python-script-but-no-python-dep 
usr/lib/logdata-anomaly-miner/AMinerRemoteControl


Since the situation became a lot of clearer using your suggestions, I will 
again try to tackle the lintian issue.

Roman

> --
> Piotr Ożarowski Debian GNU/Linux Developer
> www.ozarowski.pl  www.griffith.cc   www.debian.org
> GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



smime.p7s
Description: S/MIME cryptographic signature


Re: Maintenance of pydap

2016-06-09 Thread Sandro Tosi
On Thu, Jun 2, 2016 at 3:49 PM, Ghislain Vaillant  wrote:
> Hi Sandro,
>
> I engaged with the pydap community and I can confirm that the project
> has moved on from the old 2.x API. Since pydap 3.x is now using a new
> namespace (pydap instead of dap), then both package versions should be
> co-installable.
>
> Since I need the pydap 3.x API for hdf-compass, I propose to introduce
> a new source package for it (which I am happy to maintain) and keep the
> old source package alive until all reverse dependencies relying on 2.x
> eventually move on.
>
> Hoe does that sound?

sounds good to me

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
G+: https://plus.google.com/u/0/+SandroTosi



AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Fiedler Roman
> Von: Gianfranco Costamagna [mailto:locutusofb...@debian.org]
> Hi,
> 
> (answering where I can!)
>> Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows
>> dh_python2 to extract the version information:
>>
>> Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz,
>> init-system-helpers (>= 1.18~)
>>
>> (Remark: Is there any reason to restrict the versions to >=2.7.5? The tools
>> should have compatibility with >=2.6 and I would expect the "Depends"
>> section
>> to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and
>> DEBPYTHON_DEFAULT intended to be used to fix that?)
> 
> that stuff is built for unstable/testing, so the variables are filled
> with the current python situation
> (2.6 is only on old-stable, and Stretch has 2.7.11 already).
>
> Probably if you try to build it with a jessie chroot, you get different 
> values, and this is correct.
> (you can't in general install deb built against stretch into wheezy/jessie, 
> without
> breaking stuff, this is why some dependencies are not too relaxed, to avoid 
> people
> doing that, but instead upgrading the minimal set of packages to make sure
> things will work after the apt-pinning).
> 
> I don't foresee any issue here, because even in case of a backport, the
> package will need to be rebuilt on top of that.

OK, so I will not attempt to fiddle with that and stick to the stricter 
versions.

> >But doing the move, lintian will not like the produced package any more:
> >
> >E: logdata-anomaly-miner: python-script-but-no-python-dep 
> >usr/lib/aminer/AMiner
> 
> I can't say, I don't see the package installing stuff in usr/lib/python* on
> mentors

Maybe the hints from Piotr Ożarowski's followup mail will fix that, using his 
hints.

> >The rationale behind not putting aminer into dist-packages (and removing
> >dist-packages from python-path) was:
>> [Snip]
> 
> >b) prepare against possible future risks due to accidental loading (call to
> >__init__) of other packages residing in dist-packages, that may give rise to
> >privilege escalation (like the GNU-TLS CVE from this/last week)
> 
> mmm you want to avoid people importing your library?
>
> ok

Not directly avoid, but make them think more about it before adding. I also 
remember having issues with some svg-python library, that starts to misbehave 
as soon as other packages are installed because of "try: import xxx; ..." 
blocks. Just want to reduce the risks.
 
> >Of course, it should be possible to move the code to
> >/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation,
> but
> >this could make it more "sexy" for an admin to include whole dist-packages
> in
> >python path again.
> >In that light, should the code be moved?
> 
> leaving to other folks the answer
> >Apart from that, this will also make it harder to use the same codebase for
> >both python2.6/python2.7, but that should be fixable by providing more
> >patches.
> 
> you can't use python2.6 on Stretch, so no issue here.

Understood, dropping the 2.6/2.7 compatibility stuff
 
> >> >-#!/usr/bin/python2 -BEsSt
> >> >+#!/usr/bin/python2.7 -BEsSt
> >>
> >> this should be also handled by dh_python2 AFAIK
> >
> >Even with move dh_python2 does not touch the files. The only difference is,
> >that lintian does not like the files any more.
> 
> can you please double check with the documentation?
> https://wiki.debian.org/Python/LibraryStyleGuide

The "python2" selection in the binaries was effect of reading it, but perhaps I 
got something wrong:

Intro: "It (the LibraryStyleGuide) is not intended to supplant the Debian 
Python Policy and in fact, if you have not read that document, stop now and go 
read it"

And from " Debian Python Policy"
https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html#s-interpreter_name

"Python scripts that require the default Python 2 version should specify 
python2 as the interpreter name."

So should be OK in my opinion.

> [Snip]


smime.p7s
Description: S/MIME cryptographic signature


Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Piotr Ożarowski
just a quick reply:

* private dir is the right call, do not install into dist-packages
  (only "python-*" binary packages should install there)
* install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
  it up without any overrides, additional options, etc.
* use dh --with python2 or dh_python2 (/usr/bin/dh_python2) but never
  ever depend on internal implementation detail. I will change
  /usr/share/dh-python/dh_python2 file name just to mess with you ;-P
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Gianfranco Costamagna
Hi,

(answering where I can!)
>Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows 

>dh_python2 to extract the version information:
>
>Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz, 
>init-system-helpers (>= 1.18~)
>
>(Remark: Is there any reason to restrict the versions to >=2.7.5? The tools 
>should have compatibility with >=2.6 and I would expect the "Depends" section 
>to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and DEBPYTHON_DEFAULT 
>intended to be used to fix that?)


that stuff is built for unstable/testing, so the variables are filled
with the current python situation
(2.6 is only on old-stable, and Stretch has 2.7.11 already).

Probably if you try to build it with a jessie chroot, you get different values,
and this is correct.
(you can't in general install deb built against stretch into wheezy/jessie, 
without
breaking stuff, this is why some dependencies are not too relaxed, to avoid 
people
doing that, but instead upgrading the minimal set of packages to make sure 
things will
work after the apt-pinning).

I don't foresee any issue here, because even in case of a backport, the package 
will need to
be rebuilt on top of that.

>But doing the move, lintian will not like the produced package any more:
>
>E: logdata-anomaly-miner: python-script-but-no-python-dep 
>usr/lib/aminer/AMiner


I can't say, I don't see the package installing stuff in usr/lib/python* on 
mentors

>The rationale behind not putting aminer into dist-packages (and removing 
>dist-packages from python-path) was:
>
>a) require the aminer-admin to "link" in each dist-packages module separately, 
>thus reminding it, that it is no good idea to include large amounts of third 
>party libraries in security-critical code. Code volume is also bugs and 
>attack-surface.


I didn't get completely this, but I leave other people to answer

>b) prepare against possible future risks due to accidental loading (call to 
>__init__) of other packages residing in dist-packages, that may give rise to 
>privilege escalation (like the GNU-TLS CVE from this/last week)


mmm you want to avoid people importing your library?
ok

>Of course, it should be possible to move the code to 
>/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, but 
>this could make it more "sexy" for an admin to include whole dist-packages in 
>python path again.
>In that light, should the code be moved?


leaving to other folks the answer
>Apart from that, this will also make it harder to use the same codebase for 
>both python2.6/python2.7, but that should be fixable by providing more 
>patches.


you can't use python2.6 on Stretch, so no issue here.

>> >-#!/usr/bin/python2 -BEsSt
>> >+#!/usr/bin/python2.7 -BEsSt
>>
>> this should be also handled by dh_python2 AFAIK
>
>Even with move dh_python2 does not touch the files. The only difference is, 
>that lintian does not like the files any more.


can you please double check with the documentation?
https://wiki.debian.org/Python/LibraryStyleGuide
>So it would be OK to place the files in usr/lib/python2.7/dist-packages and 

>add a symlink to usr/lib/python2.6/dist-packages?

no, python2.6 is dead, no need to add it here.

>I would not like to push for that. Let's take the time and learn. Also this 
>package might be the template for other security-tool packages, so better have 
>it clean before cloning.


thanks for understanding!

>Understood. It might be, that this is all not a big issue for the python-dev 
>>pros, might be sorted out quickly.


I hope so, I understand the package is doing some non-standard things, so trying
to make it python-library style might be even a bad idea in general, this is why
I would prefer to have some double/triple checks here ;)

thanks for the nice email!

G.



Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

2016-06-09 Thread Fiedler Roman
Hi Gianfranco, hello Python devs,


Introduction for debian-python members: Gianfranco is giving me great 
assistance in the mentoring process to get the logdata-anomaly-miner package 
included to Debian. There were some issues, we are not completely sure, how to 
sort them out, any help on that would be appreciated!

Sorry for drawing your attention to that in the middle of the running 
discussion. You can find the whole discussion in the ITP bug report 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813096 . Perhaps reading the 
whole issue would not justify the time spent, so from my opinion most 
important 2 topics and related questions are:

* Rationale not using "usr/lib/python*/": See this mail, first inline reply 
section. Q: Are the arguments relevant? Move code or not?

* dh_python2 woes: Beginning of it, see Bugreport " Fri, 3 Jun 2016 18:54:16 
+". Idea was to share as much code as possible between python2.6/python2.7 
systems, but somehow this does not work out with packaging/lintian. Q: is 
building a 2.6/2.7 package even a good idea or should it be avoided? Instead 
upstream code and Debian packaging rules/patches could be optimized for 
generating different packages for those versions.

The package in question can be found at 
https://mentors.debian.net/package/logdata-anomaly-miner


> Von: Gianfranco Costamagna [mailto:locutusofb...@debian.org]
> Hi
>
> >I had already added it already to
> >* control
> >Depends: ${python:Depends}, python-tz, ${misc:Depends}
>
> >which was mangled (with warning) to:
>
> maybe you are installing the files in usr/lib/foo, instead of 
> usr/lib/python*/
> and then dh_python2 is not acting correctly?

Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows 
dh_python2 to extract the version information:

Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz, 
init-system-helpers (>= 1.18~)

(Remark: Is there any reason to restrict the versions to >=2.7.5? The tools 
should have compatibility with >=2.6 and I would expect the "Depends" section 
to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and DEBPYTHON_DEFAULT 
intended to be used to fix that?)


But doing the move, lintian will not like the produced package any more:

E: logdata-anomaly-miner: python-script-but-no-python-dep 
usr/lib/aminer/AMiner


> I don't think installing python stuff outside that directory is a good 
> idea...

The rationale behind not putting aminer into dist-packages (and removing 
dist-packages from python-path) was:

a) require the aminer-admin to "link" in each dist-packages module separately, 
thus reminding it, that it is no good idea to include large amounts of third 
party libraries in security-critical code. Code volume is also bugs and 
attack-surface.

b) prepare against possible future risks due to accidental loading (call to 
__init__) of other packages residing in dist-packages, that may give rise to 
privilege escalation (like the GNU-TLS CVE from this/last week)

Of course, it should be possible to move the code to 
/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, but 
this could make it more "sexy" for an admin to include whole dist-packages in 
python path again.


In that light, should the code be moved?

Apart from that, this will also make it harder to use the same codebase for 
both python2.6/python2.7, but that should be fixable by providing more 
patches.


> >@@ -1,4 +1,4 @@
> >-#!/usr/bin/python2 -BEsSt
> >+#!/usr/bin/python2.7 -BEsSt
>
> this should be also handled by dh_python2 AFAIK

Even with move dh_python2 does not touch the files. The only difference is, 
that lintian does not like the files any more.

> >Everything seems to work without it also and the same package could have
> been
> >used for python2.6 and python2.7 systems but lintian does not understand
> that.
>
> well, installing into usr/lib/python* should fix that issue

So it would be OK to place the files in usr/lib/python2.7/dist-packages and 
add a symlink to usr/lib/python2.6/dist-packages?

> I would prefer you to ask on debian-python about this issue, because I'm
> wondering about something bad that is just hidden and will be spot when the 
> package
> enters the
> archive.
>
> having an ack for a more python-savvy person would be great
> (I could even sponsor the package as-is, but I'm not too confident with 
> that)

I would not like to push for that. Let's take the time and learn. Also this 
package might be the template for other security-tool packages, so better have 
it clean before cloning.

> >I hope this fixed all the review comments for regarding the packaging. I
> would
> >like to keep the deeper restructuring of upstream code + Debian packaging
> >scripts for upstream/Debian release V0.0.3. Is that OK from Debian side?
>
> it is ok to have new uploads, don't worry about that, I just would like to 
> have a
> feedback from another DD before uploading/finish the review.
>
> is it possible for you?
> 

Re: Request to join the team

2016-06-09 Thread Piotr Ożarowski
[picca, 2016-06-09]
> my name is Picca Frédéric-Emmanuel (picca), I would like to join the
> python-modules team in order to package python-qtconsole for the
> ipython transition. Which I hope could be done for strech.

welcome :)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Request to join the team

2016-06-09 Thread picca
Hello,

my name is Picca Frédéric-Emmanuel (picca), I would like to join the 
python-modules team in order to package python-qtconsole for the ipython 
transition. Which I hope could be done for strech.

so I plan now to add all my new python module packages into python-modules.

my allioth username is:
picca

On a long term I think that I will migrate all my python packages from 
debian-science into the python-modules

guiqwt
guidata
pytango
taurus
sardana
pyfai
pymca
python-fabio
spyder

I read and agree with https://python-modules.alioth.debian.org/policy.html

thanks


Frederic


signature.asc
Description: PGP signature