Re: Bug#877849: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

Christoph Biedl wrote...

> Adam Hupp wrote...

> > - disable the deprecation warning. this wouldn't actually show
> > anything by default unless you run python with the warning enabled
> > (-Wd), but until there's a clear plan for the future of this package
> > it seems premature to declare deprecation.
> 
> Um, I cannot see that one.

Got it, so never mind.

Christoph


signature.asc
Description: Digital signature

Re: Bug#877849: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

Adam Hupp wrote...

> Whew, I was worried for a moment :)  I've pushed an update to the
> branch

Great to see.

> Changes:
> - disable the deprecation warning. this wouldn't actually show
> anything by default unless you run python with the warning enabled
> (-Wd), but until there's a clear plan for the future of this package
> it seems premature to declare deprecation.

Um, I cannot see that one. However, there is a version bump, but in the
libmagic-compat branch only while I'd expect that one in master. Care to
check?

> - fix the libmagic test under python3.  this is actually a bug in the
> libmagic python bindings but I'd rather not make changes there.

ACK

Christoph


signature.asc
Description: Digital signature

Re: Bug#877849: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

Mathias Behrle wrote...

> I found
>
> $ apt-cache rdepends python-magic
(...)

Yeah, I already had checked a few of them where possible with little
efforts. These were the test I had been talking of (actually, including
python3-magic rdeps as well).

> May be we should signal them explicitely to test the new package in
> experimental? What do you think?

My plan is to send a call for testing to debian-devel once python-magic
has been accepted. It shouldn't hurt to add the dd-list output for these
packages. That would be 27 addresses. I might be convinced to send out
individual notices - but I think that's exaggerting. Odds for bugs are
fairly low, and there's still a lot of time until the buster freeze to
detect and fix them.

Christoph, it's called "unstable" for a reason


signature.asc
Description: Digital signature

Re: Bug#877849: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

Christoph Biedl wrote...

> Hence, here is a preliminary packaging of python-magic for Debian:
>
>   
> https://www.in-ulm.de/~cbiedl/debian/python-magic/python-magic_0.4.15-1~exp1.dsc
>
> Debianites, please give it a try.

No reaction of any kind, so just uploaded to experimental. Let the games
begin.

Christoph


signature.asc
Description: Digital signature

Re: Bug#877849: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

From the popular series "Leaving fatal mistakes in spite of
proof-reading three times":

Christoph Biedl wrote...

> Doing a first round of tests showed
   no
> regressions so far.

You may stop panicking now :)

Christoph


signature.asc
Description: Digital signature

Re: Bug#877849: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

Adam Hupp wrote...

> I've pushed an update here:
>
> https://github.com/ahupp/python-magic/tree/libmagic-compat
>
> It includes a copy of libmagic's bindings, wrapped in deprecation
> warnings.  So apps should work regardless of which they depend on.
> Could you take a look and see if this works for your case?

There are two glitches, you got mails in private about these.
Doing a first round of tests showed regressions so far.

Hence, here is a preliminary packaging of python-magic for Debian:

  
https://www.in-ulm.de/~cbiedl/debian/python-magic/python-magic_0.4.15-1~exp1.dsc

Debianites, please give it a try.

About the deprecation warning: I figured out right now (at least for
Debian) it would mostly scare users. Once Christos acknowledges your
work (hopefully) and deprecates the [file] python-magic implementation,
it's about time to alert authors of programs that use python-magic but
no earlier.

Therefore I created a switch that mutes that warning unless a
PYTHONMAGIC_WARN_DEPRECATED environment variable is set, patch below.
Using this switch, users can already test whether a particular
application needs an adjustment for the [pypi] API.

At least in Debian, I expect this transition to take two years anyway.

Cheers,

Christoph
Subject: Make the deprecation warning switchable
Author: Christoph Biedl <debian.a...@manchmal.in-ulm.de>
Date: 2018-01-08
Forwarded: soon

--- a/magic/__init__.py
+++ b/magic/__init__.py
@@ -19,6 +19,7 @@
 
 import sys
 import glob
+import os
 import os.path
 import ctypes
 import ctypes.util
@@ -325,9 +326,10 @@
 
 def deprecation_wrapper(compat, fn, alternate):
 def _(*args, **kwargs):
-warnings.warn(
-"Using compatability mode with libmagic's python binding",
-DeprecationWarning)
+if "PYTHONMAGIC_WARN_DEPRECATED" in os.environ:
+warnings.warn(
+"Using compatibility mode with libmagic's python binding",
+DeprecationWarning)
 
 return compat[fn](*args, **kwargs)
 return _


signature.asc
Description: Digital signature

Re: [tryton-debian] Namespace conflict for python-magic

[Christoph Biedl]

[
It's getting rather Debian specific, so trimming recipicient list.
I'd prefer to discuss this on debian-python but keep me in Cc:,
thanks.
]

Mathias Behrle wrote...

> while approaching the test for relatorio I got aware, that the result won't be
> very significant, because relatorio depends anyway on python-magic[pypi]. To
> get meaningful results it would be much better to get the compatibility layer
> tested with packages depending on python-magic[file-magic].
> What do you think will be the best way? I think best could be to get a 
> prepared
> package with the compatibility layer on experimental and to request the
> dependencies to give it a try.

Sorry for not getting back earlier. My plan is indeed to package [pypi]
python-magic and upload to experimental. Then testers could use that
version and look for changed behaviour.

I'll try to get something done ASAP.

Christoph



signature.asc
Description: Digital signature

Re: Namespace conflict for python-magic

[Christoph Biedl]

[ I'm not lost, just way too occupied with other issues, sorry ]

Mathias Behrle wrote...

> * Christoph Biedl: " Re: Namespace conflict for python-magic" (Fri, 6 Oct 2017
>   08:34:04 +0200):
> 
> > In order to control the transition, I've filed in ITP (intent to
> > package) for Adam's python-magic as https://bugs.debian.org/877849
> 
> Did I misunderstand
> | Ideally the upstream file package would take it over.
> ?
> 
> Perhaps a little bit early, because we didn't see Adams solution, but I would
> like to understand the goals. Am I right that the intentions are
> 
> - Add a compatibility layer to python-magic[file] to support the feature set 
> of
>   python-magic[pypi]
> - Provide in some way on Pypi the same featureset for file-magic and
>   python-magic

As I understand, Adam will provide a [file] compatibility layer to
python-magic[pypi]. Then there'll be no longer a need to ship
python-magic[file], and eventually [file] upstream will drop it anyway.

> If this would be the case, in my understanding there wouldn't be any need for 
> an
> ITP or transition, but just the possibility for (Debian) packages needing
> python-magic[pypi) to use the then compatible python-magic[file].

My goal is to have just one python-magic package that is usable for
applications written for both, [file] and [pypi]. Assuming this will
come from Adam[pypi], I filed the ITP. But it's all about the goal, not
about how it's technically achieved.

Hope this made things a bit more clear.

Christoph


signature.asc
Description: Digital signature

Re: Namespace conflict for python-magic

[Christoph Biedl]

Mathias Behrle wrote...

> That's good news, Adam, thanks for it! Looking forward to get your diff.

+1

In order to control the transition, I've filed in ITP (intent to
package) for Adam's python-magic as https://bugs.debian.org/877849

Christoph


signature.asc
Description: Digital signature

Bug#877849: ITP: python-magic -- A python wrapper for libmagic

[Christoph Biedl]

Package: wnpp
Severity: wishlist
Owner: Christoph Biedl <debian.a...@manchmal.in-ulm.de>

* Package name: python-magic
  Version : 0.4.13
  Upstream Author : Adam Hupp <a...@hupp.org>
* URL : https://github.com/ahupp/python-magic/
* License : MIT
  Programming Lang: Python
  Description : A python wrapper for libmagic

Tentative long description:

python-magic is a python interface to the libmagic file type
identification library. libmagic identifies file types by checking
their headers according to a predefined list of file types.


Some background:

For quite some time, there have been two but incompatible
implementations for python-magic, providing python bindings for
libmagic (src:file).

The one provided by file ("[file]") itself might be a bit older but
appearently Adam Hupp's implementation ("[pypi]") provides a better
feature set and might even have a broader user base.

Some discussion on this took place on the debian-python list, starting
at [1]: Debian currently provides [file] so at least three packages
that require [pypi] ship their own code copy, more to come. In
upstream[2], there has been a basic understanding between Christos
Zoulas ([file]) and Adam Hupp ([pypi]) to resolve that conflict, and
Adam has announced[3] to work on a merge.

The tentative transition plan is as follows, and I'm open for
suggestions:

* Wait until [pypi] provides the [file] API.
* Ship the library using a different binary package name like
  "Package: python-magic-pypi, Provides: python-magic". Same for
  python3, here and in all the following.
* Ask users of python-magic to install this new package and check
  for regression.
* Big switch when it seems feasible: Then src:file will no longer
  provide python-magic, this package here will do insted.
  (For the records, Debian maintainer for src:file is yours truly.)
* Ask maintainers who ship a [pypi] code copy to use the package
  instead. Would be a MBF if there weren't just a few.

As my Python knowledge is limited, I'm not particulary keen on
maintaining this for Debian and I might even ITA it after the
transition. If somebody else wishes to do the job instead, please step
forward. For the filexit time however, good coordination is required.

Christoph

[1] https://lists.debian.org/debian-python/2017/09/msg8.html
[2] The file mailing list archive is down, relevant parts are
quoted in
https://lists.debian.org/debian-python/2017/09/msg00015.html
[3] https://lists.debian.org/debian-python/2017/10/msg00021.html



signature.asc
Description: Digital signature

Re: Namespace conflict for python-magic

[Christoph Biedl]

Mathias Behrle wrote...

> So finally I am still quite undetermined what to do to get the actual release
> of relatorio packaged[5] (it will be needed for the next release of Tryton). 
> For
> now there is only one usage of python-magic[6], so probably best to patch
> relatorio for now to use file-magic...

Since I doubt it's super-urgent, just wait a few weeks to see what's
going to happen.

In the best outcome, this issue will resolve itself within the next
weeks. Another idea was to add a compability layer in Debian, probably
by switching to PyPi and providing glue code for the users of the
file(1) version. Since to me it seems PyPi is the saner implementation,
CMIIW.

Worst approach: Ship a code copy. Appearently kopanocore, peframe and
sqlmap already do that. You'll have to clean up afterwards, though.

Chri- "The delivery of good medica^W packaging is to do as much nothing as 
possible" stoph


signature.asc
Description: Digital signature

Re: Namespace conflict for python-magic

[Christoph Biedl]

Mathias Behrle wrote...

> * Christoph Biedl: " Re: Namespace conflict for python-magic" (Mon, 4 Sep 2017
>   19:38:56 +0200):

> > The cleanest solution indeed was to bring both upstreams together and
> > ask them to reconcile the APIs and eventually make one of the both
> > implementations obsolete. As things happen such an attempt was started
> > two years ago but appearently never came to a result.[1]
>
> Agreed, that this would be the cleanest solution, but as you say there is
> little probability, that the two upstreams will work together to merge their
> implementations.

Still this should be tried first. Also, I'm not that pessimistic, see
below. So let's bring the parties involved into the loop:

Hello

* Christos Zoulas for file(1), and
* Adam Hupp for python-magic.

A while ago, almost two years by now, there was a short discussion on
the file(1) mailing list concerning a merge of the libmagic python
bindings. In summary, Adam proposed to end the long-standing
confusion[2], Christos agreed and suggested a possible solution[3].

After that however, nothing visible happened.

But now I'd like to give that idea a push. Mostly since on the Debian
Python mailing list (also in Cc:) a proposal was made to exchange
python-magic in Debian, from the file(1) to the PyPi one. Wearing the
maintainer's hat for file(1) in Debian, I opposed this for the sole
reason this will break applications that rely on the file(1) version.

Resolving this issue upstream will bring benefit for everybody, even at
the risk of a bumpy but one-time only transition. What is the status of
your project? Do you need some input, suggestions or other incentive
to finish this project?

Cheers,
Christoph

> > [1] The file mailing list server is currently down, so I cannot provide
> > URLs. The Message-IDs are
> > <CAJTao09xATQYZ3qR-4CR+oOrrqB_W=kyujv8esoe4b3bda5...@mail.gmail.com>
> > <20151020133008.9b79517f...@rebar.astron.com>
>
> Still down:(, If you could provide some content for me that would be nice.

Certainly:

[2]
[ Adam: ]
(...)
| Sadly, this has led to regular confusion for our users.  I'd be happy
| to shutdown my version to avoid more confusion but it seems to be
| fairly widely used (200k downloads from pypi last month).  It also has
| a number of fixes for various issues:  libmagic bugs, thread safety,
| support for other platforms, etc.
|
| I'd like to gauge interest in merging these bindings into a single
| codebase.  Specifically, this would mean:
|
| - merge magic.py from each package to produce something reasonably
| close to API-compatible to both, with some methods marked as
| deprecated where it makes sense.
| - the merged package would live under the python-magic name in pypi
| and in the `file` source distribution.
|
| I think this would be ideal for everyone involved, but wanted to gauge
| your interest before I went to the trouble.
(...)

[3]
[ Christos: ]
| Sure, it would be nice for the users if there was a definitive
| python package that had the best features from both. Someone recently
| filed a bug report about it: http://bugs.gw.com/view.php?id=477.
| Why don't you go ahead and merge them and I will copy the result
| over mine, and put a link to your package in the file sources (or
| even delete mine, and put a link to your package).


signature.asc
Description: Digital signature

Re: Namespace conflict for python-magic

[Christoph Biedl]

Mathias Behrle wrote...

> Current python(3)-magic in Debian is built from source package 'file'[0].
(...)
> OTOH the package providing python-magic on PyPi[3] is provided by another
> Upstream[4].

... and I assume the APIs are not identical?

> The cleanest solution for me would look like
> - package file in Debian should provide python(3)-file-magic
> - python-magic should be the package name corresponding to the PyPi package
>   python-magic[4]

This would result in users of the current python-magic (from file) would
be forced into the other one. Unless we (as in Debian) can guarantee
this will work in each and every use case, I fail to see why this should
be considered a clean solution.

The cleanest solution indeed was to bring both upstreams together and
ask them to reconcile the APIs and eventually make one of the both
implementations obsolete. As things happen such an attempt was started
two years ago but appearently never came to a result.[1]

Trying to address this conflict in Debian is always only second best. If
this is the only feasible way, it still should leave a choice to users
so they can install the implementation of their own preference. Co-
installability of both package was certainly nice-to-have but will
probably impossible for technical reasons.

Christoph

[1] The file mailing list server is currently down, so I cannot provide
URLs. The Message-IDs are 

<20151020133008.9b79517f...@rebar.astron.com>


signature.asc
Description: Digital signature