subject:"Re\: F35 Change\: Python Packaging Guidelines overhaul \(System\-Wide Change proposal\)"


On 15. 06. 21 13:48, Neal Gompa wrote:

On Tue, Jun 15, 2021 at 6:50 AM Petr Viktorin  wrote:


Hi Neal,
We had this conversation in the past (and you can see it in the change).
I don't think I can convince you, but I'll reiterate since it's new for
devel@.

Unlike the "mandatory tests" issue elsewhere in this thread, using the
PyPI namespace is the main point of the change. I can't take it out.


On 15. 06. 21 2:11, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:


On 6/14/21 1:53 PM, Neal Gompa wrote:

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.



I don't think we can reasonably separate the two.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

(I believe this is the "namespace" issue mentioned in the proposed change.)


You can. When it comes to distribution packages, we can maintain
consistency of the package names referenced within the distribution.
The point of this was to be able to use the names Python packages call
themselves for dependencies. PyPI is merely a distribution center for
such things. Fedora is another distribution center for such things.


You can only separate the namespaces if you sacrifice interoberability,
i.e. allowing pip-installable packages to depend on Fedora packages.
It turns out you can already do that (venv --system-site-packages), and
users expect that dist-info project names use the PyPI namespace. This
is a current problem that we need to solve.

There are other (harder) solutions for the issue than reusing the PyPI
namespace in Fedora, but why should we bother with them? There already
is an ecosystem-wide namespace; why should we add a Fedora-specific one?
(Or a Linux-distro-specific one, if we're interested in cross-distro
collaboration.)

You seem to assume that registering/blocking a name on PyPI is hard.
I say it is not and I am willing to do it for packagers who need it.
(And to mitigate the bus factor, I'll make it a responsibility of Red
Hat's python-maint team.)


It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.


I agree, in fact, I think Fedora's problems here are a subset of the
problems the private organizations have: if issues of proprietary corps
are solved, we can use the solution as well. (However, it'll need more
work than is necessary for Fedora/FOSS needs, so I don't want to drive
the effort.)
BUT, if the issues are solved, it'll likely be through namespacing: we'd
need to prefix our names with `fedora-` or `fedora:`. I still think it
is better for Fedora to reuse the public PyPI namespace rather than
start its own.


As long as you don't make packagers deal with it, I'm fine with it.
I'm okay with the PyPI contract as long as you don't make me or any
other Fedora packager specifically deal with it. And I strongly
suspect you're going to want automation in the future, but that choice
is up to you.


When packaging something that isn't on PyPI, you can let me know (on 
python-de...@lists.fedoraproject.org ) and I'll get the name blocked. I 
expect this to be rare enough to not need automation.


If the name *conflicts* with something on PyPI, then also let me know. 
Dealing with that issue might be more difficult, and it might involve 
you more. Still, this situation is confusing for users, so I think it's 
fair to want to solve it.

(And again, I'm not expecting existing packages to do this *right* now.)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Miro Hrončok


On 14. 06. 21 21:00, Miro Hrončok wrote:


I tentatively agree with the idea of requiring an “import foo.bar” smoke test 
in cases where the upstream tests cannot be used, especially since 
pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to 
add runtime dependencies as BR’s. (It may not always be practical to 
explicitly import all subpackages/modules in a complicated package, but even 
importing the top-level package is a good start). It would catch a large 
portion of the FTI bugs that appear in practice.


We could very well add a macro helper that imports modules from the 
%{buildroot} and:


  - when given positional arguments, import the given names


Proof of concept:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Miro Hrončok


On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:

If that is not possible with reasonable effort,
at least a basic smoke test (such as importing the packaged module)
*MUST* be run in `+%check+`.


A simple scriplet should be introduced I think:

%check
%do_import_test


Already on it:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Neal Gompa

On Tue, Jun 15, 2021 at 6:50 AM Petr Viktorin  wrote:
>
> Hi Neal,
> We had this conversation in the past (and you can see it in the change).
> I don't think I can convince you, but I'll reiterate since it's new for
> devel@.
>
> Unlike the "mandatory tests" issue elsewhere in this thread, using the
> PyPI namespace is the main point of the change. I can't take it out.
>
>
> On 15. 06. 21 2:11, Neal Gompa wrote:
> > On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  
> > wrote:
> >>
> >> On 6/14/21 1:53 PM, Neal Gompa wrote:
> >>> This is completely unreasonable. The dist-info/egg-info data of a
> >>> Python module is for generating dependencies, not for forcing people
> >>> to deal with PyPI.
> >>
> >>
> >> I don't think we can reasonably separate the two.
> >>
> >> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
> >>
> >> (I believe this is the "namespace" issue mentioned in the proposed change.)
> >
> > You can. When it comes to distribution packages, we can maintain
> > consistency of the package names referenced within the distribution.
> > The point of this was to be able to use the names Python packages call
> > themselves for dependencies. PyPI is merely a distribution center for
> > such things. Fedora is another distribution center for such things.
>
> You can only separate the namespaces if you sacrifice interoberability,
> i.e. allowing pip-installable packages to depend on Fedora packages.
> It turns out you can already do that (venv --system-site-packages), and
> users expect that dist-info project names use the PyPI namespace. This
> is a current problem that we need to solve.
>
> There are other (harder) solutions for the issue than reusing the PyPI
> namespace in Fedora, but why should we bother with them? There already
> is an ecosystem-wide namespace; why should we add a Fedora-specific one?
> (Or a Linux-distro-specific one, if we're interested in cross-distro
> collaboration.)
>
> You seem to assume that registering/blocking a name on PyPI is hard.
> I say it is not and I am willing to do it for packagers who need it.
> (And to mitigate the bus factor, I'll make it a responsibility of Red
> Hat's python-maint team.)
>
> > It's not terribly different from how organizations may have private
> > Python package indexes that may use whatever names they want for
> > Python software they build and release.
>
> I agree, in fact, I think Fedora's problems here are a subset of the
> problems the private organizations have: if issues of proprietary corps
> are solved, we can use the solution as well. (However, it'll need more
> work than is necessary for Fedora/FOSS needs, so I don't want to drive
> the effort.)
> BUT, if the issues are solved, it'll likely be through namespacing: we'd
> need to prefix our names with `fedora-` or `fedora:`. I still think it
> is better for Fedora to reuse the public PyPI namespace rather than
> start its own.

As long as you don't make packagers deal with it, I'm fine with it.
I'm okay with the PyPI contract as long as you don't make me or any
other Fedora packager specifically deal with it. And I strongly
suspect you're going to want automation in the future, but that choice
is up to you.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Vitaly Zaitsev via devel


On 15.06.2021 13:33, Petr Viktorin wrote:

If a test suite exists upstream,
it *SHOULD* be run in the `+%check+` section.


LGTM now. Many thanks.


If that is not possible with reasonable effort,
at least a basic smoke test (such as importing the packaged module)
*MUST* be run in `+%check+`.


A simple scriplet should be introduced I think:

%check
%do_import_test

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 15. 06. 21 13:32, Alexander Bokovoy wrote:

On ti, 15 kesä 2021, Petr Viktorin wrote:

On 14. 06. 21 20:09, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Ben Cotton wrote:

[...]

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat 
namespace,
and PyPI is ''the'' place where open-source Python packages are 
generally
published, so users and tools assume a package called 
requests

is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes 
sense

for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python 
packaging

metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be 
registered

on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved 
on PyPI]

to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


Then I'd recommend that the project name in dist-info metadata would 
be, for example "samba-bindings".


Samba Python bindings have no dist-info metadata. Does it need to? The
bindings are consumed by few projects around Samba but none of them has
anything related on PyPI. What exists on PyPI is either an outdated
stuff without upstream sources or a reference to a similar outdated
stuff, none of which is using Samba Python bindings.


It doesn't need to have dist-info metadata, but I believe it would be 
better if it did, so that it's *possible* to encode "requires Samba" in 
Python packaging metadata.
(*Should* the bindings be consumed by more than a few projects around 
Samba? If yes, it dist-info would make it easier.)


There should also be a good way to add dist-info metadata to projects 
*like* Samba, even if Samba packagers decide they don't want it.


This matters in cases like a pip-installable package declaring a 
dependency on Samba bindings: it now can't use `samba` because that 
means something else on PyPI.


There are already unrelated projects with Samba name on PyPI. This
change in Fedora guidelines cannot affect those. It cannot also affect
actual use of Samba Python bindings. It looks like there is no real use
of PyPI for our use case either, so why the fuss for this particular
package?

I suspect at best we'd submit a request for exception, if pressed.


No pressure; this is exactly why the old guidelines aren't going anywhere.


I fully agree about not installing Samba itself via pip.

(Also, this is one of the cases that might take a decade -- I'm not 
putting pressure on you to implement this, but pointing out a 
direction for the future.)


Frankly, I do not see a real use case for what you might see in other
projects. We had a similar issue with FreeIPA and what we provide on
PyPI is a crippled subset of FreeIPA Python modules that mostly does not
use real FreeIPA features. To date, I am not sure it is really bringing
any benefit to us (upstream).

This does not mean the whole effort is useless, not at all. It means, in
my opinion, that proposed guidelines are failing to address real world
use cases present in Fedora. Invalidating these use cases is as bad as
dropping the guidelines' changes in one go. ;)


Don't put Samba or FreeIPA up on PyPI. But provide a name in dist-info, 
so packages on PyPI can depend on Samba/FreeIPA (which need to be 
installed separately). And reserve that name, so  no one else can put it 
on PyPI either.


Again, no pressure, just a direction for the future.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:

On 14.06.2021 15:32, Ben Cotton wrote:

Running upstream tests is mandatory.

What about tests that require network access?

Thanks for this and all the other concerns about mandatory tests!
I updated the proposal to mane them not mandatory, but require a smoke
test if they're not there.
I also removed the mention of Fedora CI: it was meant as an escape hatch
for tests that need network, but that's now unnecessary.

Diff:
https://pagure.io/fork/pviktori/packaging-committee/c/ec9643873c989e0ff264128891665f6ad78f3e4a?branch=new-py-guidelines
History including other minor changes:
https://pagure.io/fork/pviktori/packaging-committee/commits/new-py-guidelines

New wording:

If a test suite exists upstream,
it *SHOULD* be run in the `+%check+` section.
If that is not possible with reasonable effort,
at least a basic smoke test (such as importing the packaged module)
*MUST* be run in `+%check+`.

You *MAY* exclude specific failing tests.
You *MUST NOT* disable the entire testsuite
or ignore its result to solve a build failure.

As an exception,
you *MAY* disable tests with an appropriate `+%if+` conditional
(e.g. http://rpm.org/user_doc/conditional_builds.html[bcond])
when xref:index.adoc#bootstrapping[bootstrapping].

Most errors in Python happen at run-time,
so tests are extremely important to root out issues,
especially when mass rebuilds are required.

Common reasons for skipping tests in `+%check+` include requiring
network access,
dependencies not packaged in Fedora,
and/or specialized hardware or resources.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Alexander Bokovoy


On ti, 15 kesä 2021, Petr Viktorin wrote:

On 14. 06. 21 20:09, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Ben Cotton wrote:

[...]

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a 
flat namespace,

and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called 
requests

is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python 
packaging

metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be 
registered

on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been 
reserved on PyPI]

to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


Then I'd recommend that the project name in dist-info metadata would 
be, for example "samba-bindings".


Samba Python bindings have no dist-info metadata. Does it need to? The
bindings are consumed by few projects around Samba but none of them has
anything related on PyPI. What exists on PyPI is either an outdated
stuff without upstream sources or a reference to a similar outdated
stuff, none of which is using Samba Python bindings.

This matters in cases like a pip-installable package declaring a 
dependency on Samba bindings: it now can't use `samba` because that 
means something else on PyPI.


There are already unrelated projects with Samba name on PyPI. This
change in Fedora guidelines cannot affect those. It cannot also affect
actual use of Samba Python bindings. It looks like there is no real use
of PyPI for our use case either, so why the fuss for this particular
package?

I suspect at best we'd submit a request for exception, if pressed.


I fully agree about not installing Samba itself via pip.

(Also, this is one of the cases that might take a decade -- I'm not 
putting pressure on you to implement this, but pointing out a 
direction for the future.)


Frankly, I do not see a real use case for what you might see in other
projects. We had a similar issue with FreeIPA and what we provide on
PyPI is a crippled subset of FreeIPA Python modules that mostly does not
use real FreeIPA features. To date, I am not sure it is really bringing
any benefit to us (upstream).

This does not mean the whole effort is useless, not at all. It means, in
my opinion, that proposed guidelines are failing to address real world
use cases present in Fedora. Invalidating these use cases is as bad as
dropping the guidelines' changes in one go. ;)


--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Vitaly Zaitsev via devel


On 14.06.2021 22:33, Dan Čermák wrote:

I would then suggest to change the wording from "Running upstream tests
is mandatory." to "Upstream tests SHOULD be run unless there are
compelling reasons. In that case basic smoke tests MUST be added to
%check".


+1 for this.

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Vitaly Zaitsev via devel


On 15.06.2021 10:14, Vít Ondruch wrote:
I think that "use your bests judgement" still applies. So whatever is in 
guidelines should be respected, but sometimes there needs to be exceptions.


The word "MUST" should be replaced by "SHOULD" then.

If upstream tests works fine, I always enable them, but if I need to do 
a lot of work by patching/rewriting them, I would rather skip them 
altogether, because I'm not getting paid for this unnecessary work.


Also, some upstream tests are designed for their own CI only. You will 
need to do a lot of patching to reproduce their build/tests environment.


--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

I'll address the larger "mandatory tests" issue later; thanks for all 
your concerns!

This point deserves a reply on its own:

On 14. 06. 21 19:35, Benjamin Beasley wrote:
[...]

It’s also not clear to me why the Python guidelines should be so much stricter 
than the overall Fedora guidelines 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in 
this area.


Two reasons:
- Python is a dynamic language; most errors only happen at runtime. In 
C, just compiling and linking might work as a reasonable smoke test.
- I'd like the Python guidelines to be a bit "ahead" of the Fedora-wide 
ones. The proposal contains a few ideas that might be better 
Fedora-wide, but I'd rather introduce and test them in the ecosystem I know.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 20:09, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Ben Cotton wrote:

[...]

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat 
namespace,

and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called 
requests

is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python 
packaging

metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be 
registered

on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on 
PyPI]

to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


Then I'd recommend that the project name in dist-info metadata would be, 
for example "samba-bindings".
This matters in cases like a pip-installable package declaring a 
dependency on Samba bindings: it now can't use `samba` because that 
means something else on PyPI.


I fully agree about not installing Samba itself via pip.

(Also, this is one of the cases that might take a decade -- I'm not 
putting pressure on you to implement this, but pointing out a direction 
for the future.)



=== Burdening packagers ===

[https://lists.fedoraproject.org/archives/list/python-de...@lists.fedoraproject.org/message/IYBF5GX6HVQYPC5EJ6HVK7GJHAIKHVBK/ 


Neal]
does
“not necessarily disagree that PyPI and Fedora pythonXdist() names
should match, but [he disagrees] on burdening packagers with this.”

However, actually pushing a package to PyPI (and maintaining it there)
is not necessary.
The minimal-effort solution is to ''reserve'' the name on PyPI
so that a conflicting package can not be created there.

The author of this change is willing to do this work for all
Python packages whose upstream is not on PyPI;
packagers need to send a request to the Python SIG list.
If something happens to me, Red Hat's python-maint team is also ready
to do this, and in the event of catastrophic management change,
PyPI maintainers can be e-mailed directly.

This is all described in the new guidelines (see the ''PyPI parity'' 
section).


The new guidelines discourage the minimal-effort solution because it is
suboptimal ''for users'' (and for the wider Python ecosystem).


In case of Samba I would argue that going with anything but
minimal-effort is actually suboptimal to users.


I agree; Samba is a special case.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


Hi Neal,
We had this conversation in the past (and you can see it in the change). 
I don't think I can convince you, but I'll reiterate since it's new for 
devel@.


Unlike the "mandatory tests" issue elsewhere in this thread, using the 
PyPI namespace is the main point of the change. I can't take it out.



On 15. 06. 21 2:11, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:


On 6/14/21 1:53 PM, Neal Gompa wrote:

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.



I don't think we can reasonably separate the two.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

(I believe this is the "namespace" issue mentioned in the proposed change.)


You can. When it comes to distribution packages, we can maintain
consistency of the package names referenced within the distribution.
The point of this was to be able to use the names Python packages call
themselves for dependencies. PyPI is merely a distribution center for
such things. Fedora is another distribution center for such things.


You can only separate the namespaces if you sacrifice interoberability, 
i.e. allowing pip-installable packages to depend on Fedora packages.
It turns out you can already do that (venv --system-site-packages), and 
users expect that dist-info project names use the PyPI namespace. This 
is a current problem that we need to solve.


There are other (harder) solutions for the issue than reusing the PyPI 
namespace in Fedora, but why should we bother with them? There already 
is an ecosystem-wide namespace; why should we add a Fedora-specific one? 
(Or a Linux-distro-specific one, if we're interested in cross-distro 
collaboration.)


You seem to assume that registering/blocking a name on PyPI is hard.
I say it is not and I am willing to do it for packagers who need it. 
(And to mitigate the bus factor, I'll make it a responsibility of Red 
Hat's python-maint team.)



It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.


I agree, in fact, I think Fedora's problems here are a subset of the 
problems the private organizations have: if issues of proprietary corps 
are solved, we can use the solution as well. (However, it'll need more 
work than is necessary for Fedora/FOSS needs, so I don't want to drive 
the effort.)
BUT, if the issues are solved, it'll likely be through namespacing: we'd 
need to prefix our names with `fedora-` or `fedora:`. I still think it 
is better for Fedora to reuse the public PyPI namespace rather than 
start its own.



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 17:39, Richard W.M. Jones wrote:

Questions, questions ...

These new guidelines seem to be fine for pure Python packages, but I'm
maintaining a couple of packages where Python bindings are built as
subpackages of existing C libraries:

https://src.fedoraproject.org/rpms/libnbd
https://src.fedoraproject.org/rpms/libguestfs

[Yes there are good reasons for this.  No we are not going to decouple
them and package the Python stuff separately.]

Running %pyproject_install etc is not really an option for us.
However I guess some of these macros could make sense
(%pyproject_save_files?  %pyproject_files?).  Also I don't think these
projects generate the extra *.egg-info files or how to create them.


Note that the macros are helpers/implementation details. They're not 
necessary; you can use other tools to be compatible with the guidelines.

Some of those tools are still to be written.
That's the main reason why adopting the guidelines might take a decade. 
We do intend to add helpers for more use cases, but it's not a priority 
right now.


Like the change says, there's no rush; the old guidelines will be 
retired "some time after the vast majority of Python packages follow the 
new guidelines and there are no known blockers for the remaining ones".
I hope it's clear that I do not expect individual packagers to solve 
this problem. (Though if y'all want to help, I won't object.)



=== Not removing older guidelines ===
There's no rush; it might well take a decade.


I guess we've got a while ...

While I've got your attention, one other package is interesting:

https://src.fedoraproject.org/rpms/nbdkit

This has Python 3 bindings but they work in completely the opposite
way to normal bindings, namely a Python interpreter is embedded in a
C program.  As far as I know none of the Python guidelines in Fedora
address this scenario directly, and indeed we don't currently use any
of the special %python* macros.


Thanks for the examples!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-15 Thread Vít Ondruch



Dne 14. 06. 21 v 19:30 Vitaly Zaitsev via devel napsal(a):

On 14.06.2021 19:16, Miro Hrončok wrote:
That is exactly the thing we need to avoid. Python packages without 
tests always cause trouble when we measure impact of our changes. 
E.g. when we continuously rebuild packages with the next Python 
release, pure Python packages with incompatibilities but without 
%check always succeed the build, but they break other packages.


OK, I will orphan all my Python packages than. I don't want to deal 
with tests at all.




I think that "use your bests judgement" still applies. So whatever is in 
guidelines should be respected, but sometimes there needs to be exceptions.


Nevertheless, I definitely support Python team in making %check section 
mandatory and should if disabled, it should be well reasoned. This is 
what Ruby guidelines says on this topic:



~~~

If there is test suite available for the package (even separately, for 
example not included in the gem but available in the upstream 
repository), it SHOULD be run in |%check|. The test suite is the only 
automated tool which can assure basic functionality of the package. 
Running it is especially helpful when mass rebuilds are required. You 
MAY skip test suite execution when not all build dependencies are met 
but this MUST be documented in the specfile. The missing build 
dependencies to enable the test suite SHOULD be packaged for Fedora as 
soon as possible and the test suite re-enabled.


~~~


Vít

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Gordon Messmer


On 6/14/21 6:07 PM, Neal Gompa wrote:

That only worked because public was default, not private.
That's not how it works for Fedora packages at all.



You're much more experienced in this area than I am, so I'm inclined to 
think that you're right and I'm missing something. The guidelines do 
make sense to me though, as a packager, because if a package in Fedora 
provides python3dist(samba), then I infer that other packages record a 
dependency on a python package named "samba".  And as long as those 
packages are *only* built as RPMs and installed in Fedora, everything is 
fine.  But if one of those packages is also published on PyPI, and 
records a dependency on "samba", then anyone who installs that package 
through "pip" is going to pull in a bad dependency, and we should be 
trying to avoid that situation, typically by either publishing packages 
in PyPI or at least reserving the names.


It's unfortunate that there is a "samba" module on PyPI.  I don't know 
if we could negotiate with its owner to transfer that name. The 
"homepage" links to an absent github project, and the module does not 
appear to be actively developed.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 8:35 PM Gordon Messmer  wrote:
>
> On 6/14/21 5:11 PM, Neal Gompa wrote:
> > On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  
> > wrote:
> >> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
> > It's not terribly different from how organizations may have private
> > Python package indexes that may use whatever names they want for
> > Python software they build and release.
>
>
> Yes, that was my point.  That's exactly how Alex Birsan was able to
> infiltrate and exploit "dozens" of tech companies.

That only worked because public was default, not private.
That's not how it works for Fedora packages at all.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Gordon Messmer


On 6/14/21 5:11 PM, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.



Yes, that was my point.  That's exactly how Alex Birsan was able to 
infiltrate and exploit "dozens" of tech companies.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:
>
> On 6/14/21 1:53 PM, Neal Gompa wrote:
> > This is completely unreasonable. The dist-info/egg-info data of a
> > Python module is for generating dependencies, not for forcing people
> > to deal with PyPI.
>
>
> I don't think we can reasonably separate the two.
>
> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
>
> (I believe this is the "namespace" issue mentioned in the proposed change.)

You can. When it comes to distribution packages, we can maintain
consistency of the package names referenced within the distribution.
The point of this was to be able to use the names Python packages call
themselves for dependencies. PyPI is merely a distribution center for
such things. Fedora is another distribution center for such things.

It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.

-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Gordon Messmer


On 6/14/21 1:53 PM, Neal Gompa wrote:

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.



I don't think we can reasonably separate the two.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

(I believe this is the "namespace" issue mentioned in the proposed change.)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 3:05 PM Miro Hrončok  wrote:
>
> On 14. 06. 21 20:09, Alexander Bokovoy wrote:
> >>  PyPI Parity 
> >>
> >> Machine-readable metadata (''distribution'' names in
> >> dist-info directories on disk and the corresponding
> >> python3.Xdist(foo) RPM provides) will match the Python
> >> Package Index (PyPI).
> >>
> >> This solves a ''namespace'' issue. Python packaging tools use a flat 
> >> namespace,
> >> and PyPI is ''the'' place where open-source Python packages are generally
> >> published, so users and tools assume a package called requests
> >> is whatever requests means on PyPI.
> >> While this is not ideal (especially for private packages), it makes sense
> >> for Fedora to align with the PyPI namespace.
> >>
> >> Note that Fedora package names are not affected – just the Python packaging
> >> metadata on disk and virtual RPM Provides generated from it.
> >>
> >> The new guidelines cover what to do for packages that cannot be registered
> >> on PyPI. The Change owner is prepared to help with PyPI registration.
> >>
> >> Note that names found in Fedora but not on PyPI
> >> [https://github.com/pypa/pypi-support/issues/355 have been reserved on 
> >> PyPI]
> >> to avoid being taken by unrelated projects.
> >
> > samba has extensive Python C bindings but does not use PyPI at all. We
> > don't want to, we don't need to, it is technically not possible without
> > building Samba from scratch and it would not make it usable for PIP
> > install without a stricter coordination of the non-Python dependencies
> > -- that's what Linux distributions do.
> >
> > In addition, 'samba' name is taken by an unrelated package on PyPI which
> > was not updated since 2019. For us this namespacing enforcement would
> > only be a problem.
>
> As long as the RPM package doesn't contain dist-info/egg-info that says "this
> is a Python package called samba" (and hence doesn't provide e.g.
> python3dist(samba)), this rule does not apply to samba's Python bindings.

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.

I reiterate that I did not *ever* envision the Python SIG using the
dependency generator I introduced as a means to act as a gatekeeper
for Python software packaging in Fedora.



--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Dan Čermák

Miro Hrončok  writes:

> On 14. 06. 21 19:35, Benjamin Beasley wrote:
>> I’m still in favor of running every test that is even vaguely practical in 
>> %check, but upstream Python packaging practices are wildly diverse 
>> (arguably, a mess) and it seems like a strongly worded SHOULD with a 
>> fallback of “trust the packager” would be a better approach than forcing 
>> packagers to build complicated CI configurations and go to great lengths to 
>> implement and debug network-connected tests they cannot reproduce locally.
>
> I don't disagree with you.
>
> However I think we should at least strictly require a smoke test (such as 
> %python3 -c "import foo, foo.bar") in such cases, for reasons described 
> below...

I would then suggest to change the wording from "Running upstream tests
is mandatory." to "Upstream tests SHOULD be run unless there are
compelling reasons. In that case basic smoke tests MUST be added to
%check".

We could consider suggesting Fedora CI for tests that require network
access, but given the voiced concerns its unlikely to be a viable
alternative for both package maintainers and the Python SIG conducting
Python version bump rebuilds.


Cheers,

Dan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 20:09, Alexander Bokovoy wrote:

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat namespace,
and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called requests
is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python packaging
metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be registered
on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI]
to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


As long as the RPM package doesn't contain dist-info/egg-info that says "this 
is a Python package called samba" (and hence doesn't provide e.g. 
python3dist(samba)), this rule does not apply to samba's Python bindings.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 20:28, Ben Beasley wrote:



On Mon, Jun 14, 2021, at 2:07 PM, Miro Hrončok wrote:

On 14. 06. 21 19:35, Benjamin Beasley wrote:
However I think we should at least strictly require a smoke test (such as
%python3 -c "import foo, foo.bar") in such cases, for reasons described below...
[…]

It’s also not clear to me why the Python guidelines should be so much stricter 
than the overall Fedora guidelines 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in 
this area.


As written elsewhere in this thread, because we regularly got bitten by pure
Python packages without tests.
[…]


I tentatively agree with the idea of requiring an “import foo.bar” smoke test 
in cases where the upstream tests cannot be used, especially since 
pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add 
runtime dependencies as BR’s. (It may not always be practical to explicitly 
import all subpackages/modules in a complicated package, but even importing the 
top-level package is a good start). It would catch a large portion of the FTI 
bugs that appear in practice.


We could very well add a macro helper that imports modules from the 
%{buildroot} and:


 - when given positional arguments, import the given names
 - when no arguments given, imports all modules saved by %pyproject_save_files


The NodeJS packaging guidelines have a similar requirement 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_build_testing_in_check).


Awesome requirement. Will discuss this with Petr.


Even just mandating that the runtime requirements must be BR’s 
(%pyproject_buildrequires -r or “better”) would catch most such issues at build 
time. I did this in pipx for exactly that reason.


Recently I was considering if -r shouldn't be the default (with -R or similar 
for opt-out). That should catch most of the fails to install issues.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Ben Beasley

On Mon, Jun 14, 2021, at 2:07 PM, Miro Hrončok wrote:
> On 14. 06. 21 19:35, Benjamin Beasley wrote:
> However I think we should at least strictly require a smoke test (such as 
> %python3 -c "import foo, foo.bar") in such cases, for reasons described 
> below...
> […]
> > It’s also not clear to me why the Python guidelines should be so much 
> > stricter than the overall Fedora guidelines 
> > (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) 
> > in this area.
> 
> As written elsewhere in this thread, because we regularly got bitten by pure 
> Python packages without tests.
> […]
> 
> -- 
> Miro Hrončok
> -- 
> Phone: +420777974800
> IRC: mhroncok

I tentatively agree with the idea of requiring an “import foo.bar” smoke test 
in cases where the upstream tests cannot be used, especially since 
pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add 
runtime dependencies as BR’s. (It may not always be practical to explicitly 
import all subpackages/modules in a complicated package, but even importing the 
top-level package is a good start). It would catch a large portion of the FTI 
bugs that appear in practice.

The NodeJS packaging guidelines have a similar requirement 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_build_testing_in_check).

Even just mandating that the runtime requirements must be BR’s 
(%pyproject_buildrequires -r or “better”) would catch most such issues at build 
time. I did this in pipx for exactly that reason.

– Ben Beasley
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Alexander Bokovoy


On ma, 14 kesä 2021, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x

== Summary ==
The Python Packaging guidelines will be rewritten, with the major changes being
'''PyPI parity''' and usage of '''upstream metadata'''.

A new set of macros, {{package|pyproject-rpm-macros}}, written in mind with
the new guidelines and with upstream best practices, are documented in
the new guidelines as practical examples.

The older (a.k.a. "201x-era") Python Packaging guidelines will remain in effect
as an option (until retired by another Fedora Change).

== Owner ==
* Name: [[User:pviktori| Petr Viktorin]]
** Email: pvikt...@redhat.com

* Name: [[SIGs/Python| Python SIG]]
** Email: python-de...@lists.fedoraproject.org


== Detailed Description ==

The proposed new Python packaging guidelines are available at this
Pull request: https://pagure.io/packaging-committee/pull-request/1072

A rendered preview (which might be deleted by the time you read this)
is at: 
http://100.26.217.43/packaging-committee-pr1072/packaging-guidelines/Python/

=== Not removing older guidelines ===

The current ("201x-era") guidelines will stay in effect as an option for
packages that haven't migrated yet or those that cannot follow the new
guidelines for whatever unforeseen reason.

They will be retired in another Fedora Change,
some time after the vast majority of Python packages follow the new guidelines
and there are no known blockers for the remaining ones.
There's no rush; it might well take a decade.

=== Guideline Changes ===

See [https://hackmd.io/@python-maint/rJmQQc4DP an external document]
for a detailed list of changes to '''MUST'''/'''SHOULD''' rules.
The major ones are listed here:

 Requiring python3-devel 

All packages that use Python at run- or build-time will need to
BuildRequire {{package|python3-devel}}.
This package brings the necessary macros and settings,
and it will enable automated or manual checks.
(For example, Python maintainers use this requirement to list packages
that use Python in some way and might be affected by planned changes.)

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat namespace,
and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called requests
is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python packaging
metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be registered
on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI]
to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


=== Burdening packagers ===

[https://lists.fedoraproject.org/archives/list/python-de...@lists.fedoraproject.org/message/IYBF5GX6HVQYPC5EJ6HVK7GJHAIKHVBK/
Neal]
does
“not necessarily disagree that PyPI and Fedora pythonXdist() names
should match, but [he disagrees] on burdening packagers with this.”

However, actually pushing a package to PyPI (and maintaining it there)
is not necessary.
The minimal-effort solution is to ''reserve'' the name on PyPI
so that a conflicting package can not be created there.

The author of this change is willing to do this work for all
Python packages whose upstream is not on PyPI;
packagers need to send a request to the Python SIG list.
If something happens to me, Red Hat's python-maint team is also ready
to do this, and in the event of catastrophic management change,
PyPI maintainers can be e-mailed directly.

This is all described in the new guidelines (see the ''PyPI parity'' section).

The new guidelines discourage the minimal-effort solution because it is
suboptimal ''for users'' (and for the wider Python ecosystem).


In case of Samba I would argue that going with anything but
minimal-effort is actually suboptimal to users.


(It was further suggested that the registration process should be ''automatic''
to minimize burden on packagers. It will be automated if it's too much for
humans to handle, but we

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 19:35, Benjamin Beasley wrote:

I’m still in favor of running every test that is even vaguely practical in 
%check, but upstream Python packaging practices are wildly diverse (arguably, a 
mess) and it seems like a strongly worded SHOULD with a fallback of “trust the 
packager” would be a better approach than forcing packagers to build 
complicated CI configurations and go to great lengths to implement and debug 
network-connected tests they cannot reproduce locally.


I don't disagree with you.

However I think we should at least strictly require a smoke test (such as 
%python3 -c "import foo, foo.bar") in such cases, for reasons described below...



It’s also not clear to me why the Python guidelines should be so much stricter 
than the overall Fedora guidelines 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in 
this area.


As written elsewhere in this thread, because we regularly got bitten by pure 
Python packages without tests.


With Python packages, when there is no (useful) %check, the build basically 
just copy-pastes some files around and bytecompiles them. If there are no 
syntax errors, the bytecompilation always succeeds.


Python packages without tests always cause trouble when we measure impact of 
our changes. E.g. when we continuously rebuild packages with the next Python 
release, pure Python packages with incompatibilities but without %check always 
succeed the build, but they break other packages.


Example Python 3.10 failures that only manifest themselves in the depending 
packages because %check is non-existent or commented out:


https://bugzilla.redhat.com/show_bug.cgi?id=1968737
https://bugzilla.redhat.com/show_bug.cgi?id=1969311
https://bugzilla.redhat.com/show_bug.cgi?id=1919789
https://bugzilla.redhat.com/show_bug.cgi?id=1926112

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Alexander Bokovoy

On ma, 14 kesä 2021, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok  wrote:

On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
> On 14.06.2021 15:32, Ben Cotton wrote:
>> Running upstream tests is mandatory.
>
> What about tests that require network access?

 From the proposed guidelines:

> If a test suite exists upstream, it MUST be run in the %check section and/or
> in Fedora CI. You MAY exclude specific failing tests.
>
> You MUST NOT disable the entire testsuite or ignore the result to solve a
> build failure.
>
> As an exception, you MAY disable tests with an appropriate %if conditional
> (e.g. bcond) when bootstrapping.

Assuming that *all the tests* require network access, you should run them on
Fedora CI instead.

Assuming only some of the tests require network access, you should exclude them
in %check (and possibly run them on Fedora CI instead).

This is unreasonable, since Fedora CI isn't usable across all
architectures. Unless Fedora CI is at platform parity, then we have to
have an allowance for not running tests.

More to that, for packages like FreeIPA, running a FreeIPA installation
-- which is what required to have a reasonable test run -- is simply not
possible in %check. We could run a subset of unit tests but that is not
going to produce a meaningful result.

Current Fedora CI capabilities also pretty limited with regards to
multi-server deployments. We rely on OpenQA testing at the Bodhi update
time for a reason.

--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Benjamin Beasley

I agree that running the tests wherever practical is the best practice. I do
put my time where my mouth is—I maintain 17 packages that now use the
pyproject-rpm-macros, and you’ll find that in general I’ve added a lot of
%check sections to packages that were previously lacking them.

However, let me point to pipx as an example where I recently gave up on
maintaining an ever-shifting list of 50 tests that require interaction with
PyPI
(https://src.fedoraproject.org/rpms/pipx/c/55c6c9debf3039747327686a2f07dc31cd1e966b?branch=rawhide).
It just wasn’t worth it to keep doing this in order to run the small minority
of tests that can work offline. Even worse, upstream recently added an offline
bundle of a huge number of PyPI wheels to make these tests reproducible
offline, but this bundle is arch-specific with compiled binary wheels (a
non-starter for including as a Source# file), and it would now require a
nontrivial patch to run ANY of the tests without either the wheel bundle or a
network connection.

Now, running these fussy tests in CI would be great! I would accept a
reasonable PR to do so. But I’m not excited about learning Yet Another
YAML-Based Programming-As-Configuration Language being an absolute requirement
in order to maintain packages that have impractical upstream tests like these.
There are an increasing number of upstreams with test suites that are really
only designed to run in upstream’s own fickle CI, and reproducing these
environments could be quite a burden. Realistically, I picked up pipx and
cleaned it up after it was orphaned, and I’m likely just to file a Bugzilla and
orphan it again if CI is mandated. Unfortunately, I suspect there are many
packagers who would choose quiet noncompliance instead.

Another example is python-strictyaml. It builds its documentation using an
idiosyncratic build system (see https://hitchdev.com/) that is hopelessly
entangled with the idea of downloading dependencies from PyPI. An offline
documentation build is therefore nearly impossible. Currently, it has no tests
upstream, but if it ever adds them, they will be run via the same system and
will also be effectively impossible to run offline.

A third example is python-pgspecial. Most of the tests require a postgresql
database, which I cannot set up in %check because this requires root
privileges. This package actually seems like it would be compliant today,
because it has a few tests that work without the database connection, and
automatically skips the rest—but I am sure there are others where every single
test requires a privileged operation. That would be the case for
python-databases if it did not support SQLite.

A fourth example is python-absl-py. There are a couple of “smoke tests” that I
can and do run, but the core test suite requires Bazel—which, as has been
discussed before, will probably never be successfully packaged for Fedora.

I’m still in favor of running every test that is even vaguely practical in
%check, but upstream Python packaging practices are wildly diverse (arguably, a
mess) and it seems like a strongly worded SHOULD with a fallback of “trust the
packager” would be a better approach than forcing packagers to build
complicated CI configurations and go to great lengths to implement and debug
network-connected tests they cannot reproduce locally.

It’s also not clear to me why the Python guidelines should be so much stricter
than the overall Fedora guidelines
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in
this area.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Vitaly Zaitsev via devel


On 14.06.2021 19:16, Miro Hrončok wrote:
That is exactly the thing we need to avoid. Python packages without 
tests always cause trouble when we measure impact of our changes. E.g. 
when we continuously rebuild packages with the next Python release, pure 
Python packages with incompatibilities but without %check always succeed 
the build, but they break other packages.


OK, I will orphan all my Python packages than. I don't want to deal with 
tests at all.


--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 18:32, Vitaly Zaitsev via devel wrote:

On 14.06.2021 17:56, Miro Hrončok wrote:
Assuming that *all the tests* require network access, you should run them on 
Fedora CI instead.


And if I don't want to patch tests in downstream or not interested in runing 
tests at all?


For example my package wloc[1] has upstream tests, but they cannot run on 
Fedora due to the lack of a test suite file and secret keys.


Another example is a package that requires special equipment to run tests.

I'm strongly against this part of proposal. The package maintainers should 
decide whether they need tests or not.


There are two situations described here:

1) Running tests in %check is impossible, unreasonably hard or useless

In that case, I am inclined to agree running them should not be hard 
requirement, the packagers should instead be required to document the reasoning 
in the spec file.



2) Packager not interested in running tests at all

That is exactly the thing we need to avoid. Python packages without tests 
always cause trouble when we measure impact of our changes. E.g. when we 
continuously rebuild packages with the next Python release, pure Python 
packages with incompatibilities but without %check always succeed the build, 
but they break other packages.


Example Python 3.10 failures that only manifest themselves in the depending 
packages because %check is non-existent or commented out:


https://bugzilla.redhat.com/show_bug.cgi?id=1968737
https://bugzilla.redhat.com/show_bug.cgi?id=1969311
https://bugzilla.redhat.com/show_bug.cgi?id=1919789
https://bugzilla.redhat.com/show_bug.cgi?id=1926112

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 18:35, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok  wrote:


On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:

On 14.06.2021 15:32, Ben Cotton wrote:

Running upstream tests is mandatory.


What about tests that require network access?


  From the proposed guidelines:


If a test suite exists upstream, it MUST be run in the %check section and/or
in Fedora CI. You MAY exclude specific failing tests.

You MUST NOT disable the entire testsuite or ignore the result to solve a
build failure.

As an exception, you MAY disable tests with an appropriate %if conditional
(e.g. bcond) when bootstrapping.



Assuming that *all the tests* require network access, you should run them on
Fedora CI instead.

Assuming only some of the tests require network access, you should exclude them
in %check (and possibly run them on Fedora CI instead).


This is unreasonable, since Fedora CI isn't usable across all
architectures. Unless Fedora CI is at platform parity, then we have to
have an allowance for not running tests.


Yeah, the situation wrt. platform parity is not good. However I still think 
that running the network-needing tests on x86_64 only is a magnitude better 
than "%%check requires internet, commented out".


That said, requiring to run tests on the Fedora CI if they cannot run in %check 
is a new thing and I would not like the new Python packaging guidelines to be 
blocked exactly on this. I don't event think this was the intention.


However, I have different reasons than platform parity: I don't want Python 
packagers to be blocked by figuring out how to do Fedora CI, since it is 
non-trivial.


Personally, I still prefer tests in %check because they are easily verifiable 
e.g. in Copr when we try to measure the impact of some of our changes. Tests in 
the CI are unusable for this use-case, because there is no tooling around it.


I suppose allowing to not run tests (at all) if they (all of them) require 
resources not available in Koji during rpmbuild (such as internet access or 
secret keys) is a reasonable compromise, however I'd like to give Petr a chance 
to pitch in as well.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok  wrote:
>
> On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
> > On 14.06.2021 15:32, Ben Cotton wrote:
> >> Running upstream tests is mandatory.
> >
> > What about tests that require network access?
>
>  From the proposed guidelines:
>
> > If a test suite exists upstream, it MUST be run in the %check section and/or
> > in Fedora CI. You MAY exclude specific failing tests.
> >
> > You MUST NOT disable the entire testsuite or ignore the result to solve a
> > build failure.
> >
> > As an exception, you MAY disable tests with an appropriate %if conditional
> > (e.g. bcond) when bootstrapping.
>
>
> Assuming that *all the tests* require network access, you should run them on
> Fedora CI instead.
>
> Assuming only some of the tests require network access, you should exclude 
> them
> in %check (and possibly run them on Fedora CI instead).

This is unreasonable, since Fedora CI isn't usable across all
architectures. Unless Fedora CI is at platform parity, then we have to
have an allowance for not running tests.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Vitaly Zaitsev via devel


On 14.06.2021 17:56, Miro Hrončok wrote:
Assuming that *all the tests* require network access, you should run 
them on Fedora CI instead.


And if I don't want to patch tests in downstream or not interested in 
runing tests at all?


For example my package wloc[1] has upstream tests, but they cannot run 
on Fedora due to the lack of a test suite file and secret keys.


Another example is a package that requires special equipment to run tests.

I'm strongly against this part of proposal. The package maintainers 
should decide whether they need tests or not.


[1]: https://github.com/xvitaly/wloc

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)