Re: [RFC] RPM's Python dependency generator

[Nick Coghlan]

On 3 December 2016 at 05:40, Toshio Kuratomi  wrote:
> On Dec 1, 2016 6:58 AM, "Petr Viktorin"  wrote:
>> Problems with upstreams getting setup.py wrong should be treated as
>> upstream bugs and treated accordingly: reported as pull requests, or, as the
>> last resort, by a Fedora-specific patch to setup.py.
>
> Note that fedora will have to carry more patches to setup.py's (but
> hopefully of limited durations and pretty trivial).  Upstream versioned
> requirements often don't match with distro packages because we backport to
> work around bugs in compatible manners while upstream simply requires newer
> versions that have the bugs fixed.
>
> Not a showstopper by any means; just something to be aware of.

Depending on how deep people want to dive into that aspect, it may
even make sense to start using the "Local version identifier" feature
in PEP 440 to expose RPM release numbers to Python level tooling:
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Charalampos Stratakis]

Seems like a solid plan. Smaller details can be refined as well, but overall 
seems like a good direction.

Unfortunately at this time there aren't free cycles for the python-maint team 
members, so I doubt we would be able to push this for F26 as a system wide 
change on behalf of our team at least.

So if noone is up for chasing this for the current development cycle, I would 
suggest pushing it for F27.

Regards,

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat


- Original Message -
From: "Igor Gnatenko" <ignate...@redhat.com>
To: "Fedora Python SIG" <python-devel@lists.fedoraproject.org>
Sent: Thursday, December 1, 2016 2:36:48 PM
Subject: Re: [RFC] RPM's Python dependency generator

On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava <tors...@redhat.com> wrote:
> On 11/30/2016 02:44 PM, Neal Gompa wrote:
>>
>> On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava <tors...@redhat.com> wrote:
>>>
>>> I don't think the depgen should be enabled by default, at least not in
>>> the
>>> foreseeable future. IIRC it's not that well implemented—e.g. I believe it
>>> doesn't read requirements.txt for example (but I might be wrong).
>>> There will be a lot of cases where the generated requirements are
>>> incomplete, or contain unnecessary entries, etc. I think it should remain
>>> an
>>> opt-in.
>>>
>> According to various Python people, we're not actually supposed to
>> read requirements.txt. That file is explicitly designed for people to
>> individualized deployments. The proper place to get this information
>> is from the egg-info/dist-info data, which is what we read. The fact
>> that some people abuse requirements.txt and have it read in by their
>> setup.py is beside the point. Whatever the setup.py (thus
>> pip/easy_install/etc.) says it needs, the generator will dutifully
>> report.
>
>
> The fact remains in too many cases it will need to be manually adjusted, it
> won't be foolproof.
> Therefore I argue it would be better for it to be an opt-in so that new
> packagers don't immediately have to jump in into debugging a depgen they
> have no clue how really works.
We'll see how it will go. we have depgen for pkgconfig, libraries,
etc. for many years and people don't go and debug it immediately, but
for many of packages it will help a lot. Anyhow, we'll see after
couple of releases.

Neal suggested to have:
%__python_requires
%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
--requires
in python.attr inside RPM.

I tested it and it just works once I specify `%global
pythondistdeps_enable 1` in spec. Can you help me to get this
included? With RPM part it's clear how to get this, but updating
guidelines and other stuff...
>
> ___
> python-devel mailing list -- python-devel@lists.fedoraproject.org
> To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org



-- 
-Igor Gnatenko
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Petr Viktorin]

On 12/01/2016 02:42 PM, Neal Gompa wrote:

On Thu, Dec 1, 2016 at 8:36 AM, Igor Gnatenko  wrote:

On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava  wrote:

On 11/30/2016 02:44 PM, Neal Gompa wrote:


On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:


I don't think the depgen should be enabled by default, at least not in
the
foreseeable future. IIRC it's not that well implemented—e.g. I believe it
doesn't read requirements.txt for example (but I might be wrong).
There will be a lot of cases where the generated requirements are
incomplete, or contain unnecessary entries, etc. I think it should remain
an
opt-in.


According to various Python people, we're not actually supposed to
read requirements.txt. That file is explicitly designed for people to
individualized deployments. The proper place to get this information
is from the egg-info/dist-info data, which is what we read. The fact
that some people abuse requirements.txt and have it read in by their
setup.py is beside the point. Whatever the setup.py (thus
pip/easy_install/etc.) says it needs, the generator will dutifully
report.



The fact remains in too many cases it will need to be manually adjusted, it
won't be foolproof.
Therefore I argue it would be better for it to be an opt-in so that new
packagers don't immediately have to jump in into debugging a depgen they
have no clue how really works.

We'll see how it will go. we have depgen for pkgconfig, libraries,
etc. for many years and people don't go and debug it immediately, but
for many of packages it will help a lot. Anyhow, we'll see after
couple of releases.

Neal suggested to have:
%__python_requires
%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
--requires
in python.attr inside RPM.

I tested it and it just works once I specify `%global
pythondistdeps_enable 1` in spec. Can you help me to get this
included? With RPM part it's clear how to get this, but updating
guidelines and other stuff...


This will also drastically simplify the work of tools like pyp2rpm,
since instead of having to do crazy processing of module names, it can
just use the appropriate pypi provides/requires. If the macro template
enables the requires generator, it won't even need to specify requires
at all, as they'll be generated from the wheel data anyway.


+1, ideally that'll leave us with only one automatic dependency generator.


Problems with upstreams getting setup.py wrong should be treated as 
upstream bugs and treated accordingly: reported as pull requests, or, as 
the last resort, by a Fedora-specific patch to setup.py.



CCing Michal from the pyp2rpm project, so that everyone knows he's aware 
of the discussion here.


--
Petr Viktorin
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Tomas Orsava]

On 12/01/2016 02:36 PM, Igor Gnatenko wrote:

We'll see how it will go. we have depgen for pkgconfig, libraries,
etc. for many years and people don't go and debug it immediately, but
for many of packages it will help a lot. Anyhow, we'll see after
couple of releases.


Yeah, absolutely. When it's battle-tested and solid, it will make 
packaging that much easier.



Neal suggested to have:
%__python_requires
%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
--requires
in python.attr inside RPM.


Oh that's clever, I was wondering how things like that are done!


I tested it and it just works once I specify `%global
pythondistdeps_enable 1` in spec. Can you help me to get this
included? With RPM part it's clear how to get this, but updating
guidelines and other stuff...


AFAIK the best way to get the guidelines updated is to create an 
accompanying Fedora Change. [0]
I sadly don't have free cycles to take that on, as I'm currently 
involved in 2 upcoming Fedora Changes, nevertheless I will gladly 
provide any help and guidance you might need!


[0] https://fedoraproject.org/wiki/Changes/Policy
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Neal Gompa]

On Thu, Dec 1, 2016 at 8:36 AM, Igor Gnatenko  wrote:
> On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava  wrote:
>> On 11/30/2016 02:44 PM, Neal Gompa wrote:
>>>
>>> On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:

 I don't think the depgen should be enabled by default, at least not in
 the
 foreseeable future. IIRC it's not that well implemented—e.g. I believe it
 doesn't read requirements.txt for example (but I might be wrong).
 There will be a lot of cases where the generated requirements are
 incomplete, or contain unnecessary entries, etc. I think it should remain
 an
 opt-in.

>>> According to various Python people, we're not actually supposed to
>>> read requirements.txt. That file is explicitly designed for people to
>>> individualized deployments. The proper place to get this information
>>> is from the egg-info/dist-info data, which is what we read. The fact
>>> that some people abuse requirements.txt and have it read in by their
>>> setup.py is beside the point. Whatever the setup.py (thus
>>> pip/easy_install/etc.) says it needs, the generator will dutifully
>>> report.
>>
>>
>> The fact remains in too many cases it will need to be manually adjusted, it
>> won't be foolproof.
>> Therefore I argue it would be better for it to be an opt-in so that new
>> packagers don't immediately have to jump in into debugging a depgen they
>> have no clue how really works.
> We'll see how it will go. we have depgen for pkgconfig, libraries,
> etc. for many years and people don't go and debug it immediately, but
> for many of packages it will help a lot. Anyhow, we'll see after
> couple of releases.
>
> Neal suggested to have:
> %__python_requires
> %{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
> --requires
> in python.attr inside RPM.
>
> I tested it and it just works once I specify `%global
> pythondistdeps_enable 1` in spec. Can you help me to get this
> included? With RPM part it's clear how to get this, but updating
> guidelines and other stuff...

This will also drastically simplify the work of tools like pyp2rpm,
since instead of having to do crazy processing of module names, it can
just use the appropriate pypi provides/requires. If the macro template
enables the requires generator, it won't even need to specify requires
at all, as they'll be generated from the wheel data anyway.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Tomas Orsava]

On 11/30/2016 02:44 PM, Neal Gompa wrote:

On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:

I don't think the depgen should be enabled by default, at least not in the
foreseeable future. IIRC it's not that well implemented—e.g. I believe it
doesn't read requirements.txt for example (but I might be wrong).
There will be a lot of cases where the generated requirements are
incomplete, or contain unnecessary entries, etc. I think it should remain an
opt-in.


According to various Python people, we're not actually supposed to
read requirements.txt. That file is explicitly designed for people to
individualized deployments. The proper place to get this information
is from the egg-info/dist-info data, which is what we read. The fact
that some people abuse requirements.txt and have it read in by their
setup.py is beside the point. Whatever the setup.py (thus
pip/easy_install/etc.) says it needs, the generator will dutifully
report.


The fact remains in too many cases it will need to be manually adjusted, 
it won't be foolproof.
Therefore I argue it would be better for it to be an opt-in so that new 
packagers don't immediately have to jump in into debugging a depgen they 
have no clue how really works.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Neal Gompa]

On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
>
> I don't think the depgen should be enabled by default, at least not in the
> foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> doesn't read requirements.txt for example (but I might be wrong).
> There will be a lot of cases where the generated requirements are
> incomplete, or contain unnecessary entries, etc. I think it should remain an
> opt-in.
>

According to various Python people, we're not actually supposed to
read requirements.txt. That file is explicitly designed for people to
individualized deployments. The proper place to get this information
is from the egg-info/dist-info data, which is what we read. The fact
that some people abuse requirements.txt and have it read in by their
setup.py is beside the point. Whatever the setup.py (thus
pip/easy_install/etc.) says it needs, the generator will dutifully
report.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

[RFC] RPM's Python dependency generator

[Igor Gnatenko]

Hi,

in short, it reads egg metadata and can generate Provides (which we
already do now), Requires (which I want to talk about) and Recommends
(which I don't care atm).

Let's take simple package -- aiohttp.
https://bugzilla.redhat.com/show_bug.cgi?id=1381750

As you can see, since some version multidict requirement was bumped to
>= 2.0 and async_timeout requirement was added. Currently we specify
all requirements during initial package and usually without versions
which is breaking after some time.

So, let's try it (I will skip unrelated parts).
Before:
python(abi) = 3.5
python3-chardet
python3-multidict
After:
python(abi) = 3.5
python3.5dist(async-timeout)
python3.5dist(chardet)
python3.5dist(multidict) >= 2.0

Without more complicated packages (MNE, nipy, nilearn, moss) it's
getting much more harder since I have there 10+ deps.

We can't really track all changes in upstream code, so if we will
enable dependency generator, it will do this work for us. Note that we
can't just enable it in RPM, because it will break a lot of packages
due to:
* missing requires in egg metadata
* extra requires in egg metadata (e.g. windows-modules)

I would propose plan:
1. Create macro which will enable/disable depgen (e.g %python_enable_depgen)
2. Start enabling depgen and porting things (somehow reuse
portingdb.xyz probably?) and submitting patches upstream
3. In 1-2 releases I hope we can use it for major amount of packages
4. Enable depgen by default in RPM, add disabling depgen for remaining packages

Neal, you can share how Mageia did that as well and feel free to comment this ;)

Thoughts?
-- 
-Igor Gnatenko
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org