Re: [HEADS UP] Fedora 39 Python 3.12 rebuilds to start in a side tag this week

[Petr Viktorin]

On 30. 06. 23 18:13, Jerry James wrote:

On Fri, Jun 30, 2023 at 9:02 AM Petr Viktorin  wrote:

The use_tracing was fixed for most code, but still remained in
profiling/tracing support.
I filed a PR for Cython:
https://src.fedoraproject.org/rpms/Cython/pull-request/46

(Hoping for a Cython 3 release soon -- things are getting fixed properly
for 3.0, but the stable branch is pretty much on life support now...)


Ah great, thank you.


Do you have an example of a package that fails on ob_digit?
Uses of that should be behind CYTHON_USE_PYLONG_INTERNALS which should
be off by default (and probably will stay off until Cython 3.0).


I was trying to fix a couple of Paulo's packages, python-cypari2 and
python-fpylll.  They directly access ob_digit.  My attempts to fix
them seemed to be stymied by the Cython definition I mentioned, but
maybe I just don't know what I'm doing. :-)  If you have suggestions
on how to approach the fixes for those two packages, I would be very
grateful.


Sadly these are too involved for me to fix today, but I filed upstream 
issues:


https://github.com/sagemath/cypari2/issues/137
https://github.com/fplll/fplll/issues/514
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: [HEADS UP] Fedora 39 Python 3.12 rebuilds to start in a side tag this week

[Petr Viktorin]

On 29. 06. 23 22:37, Jerry James wrote:

On Tue, Jun 13, 2023 at 6:03 AM Tomas Hrnciar  wrote:

If you'd like to build a package after we already rebuilt it, you should be 
able to build it in the side tag via:

on branch rawhide:
$ fedpkg build --target=f39-python
$ koji wait-repo f39-python --build 


I'm trying to help by fixing packages I maintain that failed to build
on the first attempt.  However, I'm having some issues with Cython
generating incorrect code.  The most recent example is the
python-pytest-cython package, which fails because Cython generates
code that accesses the use_tracing field of _PyCFrame.  That field was
removed in python 3.12.


Thanks for your help!

The use_tracing was fixed for most code, but still remained in 
profiling/tracing support.
I filed a PR for Cython: 
https://src.fedoraproject.org/rpms/Cython/pull-request/46


(Hoping for a Cython 3 release soon -- things are getting fixed properly 
for 3.0, but the stable branch is pretty much on life support now...)




There are a couple of other packages that have issues with the
representation of a long object.  In python 3.11 and before, we had:

struct _longobject {
 PyObject_VAR_HEAD
 digit ob_digit[1];
};

In python 3.12, we have:

typedef struct _PyLongValue {
 uintptr_t lv_tag; /* Number of digits, sign and flags */
 digit ob_digit[1];
} _PyLongValue;

struct _longobject {
 PyObject_HEAD
 _PyLongValue long_value;
};

The Cython package in the side tag has this in Includes/cpython/longintrepr.pxd:

 ctypedef class __builtin__.py_long [object PyLongObject]:
 cdef digit* ob_digit

which is incorrect for python 3.12.  I think I'm stuck until Cython
has been updated for python 3.12.


Do you have an example of a package that fails on ob_digit?
Uses of that should be behind CYTHON_USE_PYLONG_INTERNALS which should 
be off by default (and probably will stay off until Cython 3.0).

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: PyPI name blocking request for vkbasalt-cli

[Petr Viktorin]

On 16. 05. 23 16:32, Sandro wrote:

On 16-05-2023 16:04, Petr Viktorin wrote:
Did I misunderstand/misinterpret the guidelines? The review is 
currently stalled due to the PyPI parity issue.


Should I approach PyPI folks myself and ask them to block 
vkbasalt-cli for me? Any kind of guidance is much appreciated.


Hi! Sorry for the silence.

It seems weird to ask PyPI admins to block a project that has working 
code, so what I would do is upload the project on PyPI, and tell the 
author that I'm happy to hand it over to him.

Do you want me to do it?


Thanks for the offer. I think I'll be able to do that myself. I 
published on PyPI before.


Thank you. Happy to help if you run into a snag.

Should the guidelines be clarified? I read the PyPI parity paragraph 
several times. While it feels weird reserving a name for an existing, 
yet unpublished package, it feels equally, if not more, weird to publish 
someone else's package without their explicit consent.


Fedora has packages that don't make any sense on PyPI. Perhaps the best 
example is RPM bindings, which are built as part of RPM and don't work 
unless you have the corresponding version of RPM installed.

That's the use case for reserving the name.

IMO, if the author doesn't publish on PyPI (yet), putting the package 
there is no different than putting it in Fedora without their consent. 
And you probably didn't think twice about that :)



I'll proceed with it all the same. Without the package landing in 
Fedora, Bottles will be broken forever. And who likes to spill a nice 
drink? ;-)


-- Sandro


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: PyPI name blocking request for vkbasalt-cli

[Petr Viktorin]

On 16. 05. 23 13:15, Sandro wrote:

On 05-05-2023 15:10, Sandro wrote:

On 27-04-2023 14:26, Sandro wrote:

Well, I submitted the package for review [2] and have been told that
vkbasalt not being published on PyPI is an issue wrt naming parity.
So, I filed a bug requesting publication on PyPI [3], but I haven't
heard back from upstream, yet. If upstream remains unresponsive, I would
need to either publish it myself or get the name reserved as described
in PyPI parity [4].


It has been almost two weeks and I haven't heard back from upstream 
regarding my request for publication on PyPI.


Still no news from upstream. But I'd like to move on with packaging 
vkbasalt-cli. Meanwhile bug reports regarding Bottles keep pouring [1] 
in. Well, at least is shows that there's a user base for that package.


What are the next steps I should take in line with PyPI parity? I'm 
leaning towards "Ask Python SIG to block the name on PyPI for you" and 
will consider publishing it myself at a later stage should upstream 
remain unresponsive.


Did I misunderstand/misinterpret the guidelines? The review is currently 
stalled due to the PyPI parity issue.


Should I approach PyPI folks myself and ask them to block vkbasalt-cli 
for me? Any kind of guidance is much appreciated.


Hi! Sorry for the silence.

It seems weird to ask PyPI admins to block a project that has working 
code, so what I would do is upload the project on PyPI, and tell the 
author that I'm happy to hand it over to him.

Do you want me to do it?





[2] https://bugzilla.redhat.com/show_bug.cgi?id=2188653
[3] https://gitlab.com/TheEvilSkeleton/vkbasalt-cli/-/issues/10
[4]
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_pypi_parity


[1] I'm exaggerating ;-)

-- Sandro
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Package naming for vkbasalt-cli

[Petr Viktorin]

Hi!

On 21. 04. 23 12:46, Sandro wrote:

Hi,

I need to package vkbasalt-cli [1] as a dependency for Bottles. Bottles 
will use the Python library, but vkbasalt-cli also provides a command 
for independent/direct usage.


My plan is to name the Fedora package 'vkbasalt-cli' in line with 
upstream and add a virtual provide for 'python3-vkbasalt-cli'.


Could someone confirm that this is the right approach for vkbasalt-cli?

Thanks in advance,

That sounds good. You could also do it the other way around.

In this case, it looks like `vkbasalt-cli` is developed independently of 
vkbasalt itself, is that right? If there's a chance that an “official” 
vkbasalt CLI appears in the future, consider using only 
`python-vkbasalt-cli`, to prevent future confusion.
But I guess if that ever happened, the fact that vkbasalt-cli provides 
`/usr/bin/vkbasalt` would be a bigger issue.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Building Python 3.12 with no-omit-frame-pointer

[Petr Viktorin]

On 22. 11. 22 18:30, Miro Hrončok wrote:

On 21. 11. 22 11:01, Petr Viktorin wrote:
And since the Python slowdown comes from a single weird function, I 
think that Fedora should ignore the Python benchmarks when evaluating 
the distro default -- and if Fedora switches to no-omit-frame-pointer, 
Python 3.11 should be an exception (to be re-evaluated for 3.12).


Do I interpret correctly?

If (and only if) Fedora switches to -fno-omit-frame-pointer, python3.11 
(and possibly older Pythons) will explicitly use -fomit-frame-pointer 
instead (aka opt-out).


Yes. If the change allows software to opt out (e.g. if it's slowed down 
too much), then, based on the benchmarks, Python 3.11 should do that.



python3.12 will do the same for now.


Or not, if that would help testing. I don't think the speed of the 
preview Python matters that much.


Once the massive changes to _PyEval_EvalFrameDefault land in Python 
3.12, we will re-evaluate whether to use -fomit-frame-pointer or 
-fno-omit-frame-pointer (regardless of the Fedora global flags).


That might not be a single point in time, so let's perhaps re-evaluate 
around Beta?

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Building Python 3.12 with no-omit-frame-pointer

[Petr Viktorin]

On 09. 11. 22 12:37, Petr Viktorin wrote:
tl;dr: Python 3.12 should be built with no-omit-frame-pointer if 
upstream recommends it.



Hello,
You might be aware of a Fedora change proposal [0] (discussed on 
fedora-devel [1] and FESCo [2]) are discussing turning on C compiler 
flags that help with performance *measurement*, but might hurt 
performance itself: `-fno-omit-frame-pointer` and 
`-mno-omit-leaf-frame-pointer`.
Apparently there are some benchmarks that make Python look extra slow 
when the flags are turned on -- which I don't quite understand.


Update: Andrii Nakryiko looked at the disassembly: 
https://pagure.io/fesco/issue/2817#comment-826636


Looks like the slowdown comes from a single function, 
_PyEval_EvalFrameDefault, which essentially *is* the Python interpreter 
-- so it's very unlikely to be indicative of other code in the distro 
(including Python extension modules).
That function is being overhauled for Python 3.12 (in the main branch 
it's now autogenerated [5], which should hopefully allow optimizations 
across any common code, and perhaps things like switching to a 
register-based VM [6]).


Given these massive planned changes in the affected function, I think we 
should treat Python 3.11 and 3.12 as entirely separate when it comes to 
performance with no-omit-frame-pointer.


And since the Python slowdown comes from a single weird function, I 
think that Fedora should ignore the Python benchmarks when evaluating 
the distro default -- and if Fedora switches to no-omit-frame-pointer, 
Python 3.11 should be an exception (to be re-evaluated for 3.12).
(Most of the current benchmarks [7] are from Python, so more might be 
needed.)



[5]: https://github.com/python/cpython/issues/98831
[6]: https://github.com/faster-cpython/ideas/issues/489
[7]: https://github.com/DaanDeMeyer/fpbench




Meanwhile, on the upstream side, Python 3.12 (due next year, main Python 
for Fedora 39 [3]) has support for `perf`. Upstream plans to recommend 
compiling with these flags when measuring performance [4], and AFAIK, 
the plan is to recommend *always* compiling with them.
The idea is that possible speedups from "allowing anyone to 
profile/optimize their workflow" are worth the initial slowdown.


I'm not much of a performance expert myself, but I do get drawn into the 
relevant discussions on the CPython side.
As far as I can see, performance geeks are enthusiastic for `perf` 
support, and I'd like to get them to (continue to) use Fedora builds.
If CPython upstream does recommend these flags (or makes them default), 
I'm considering to turn the no-omit options on for Python 3.12 even if 
Fedora as a whole doesn't.
Note that even a 2% slowdown will likely won back by general performance 
improvements – the Faster CPython team is targeting a 20% average 
speedup for pure-Python code in 3.12, on top of the ~25% for 3.11. And 
the people responsible for this speedup have a say in the upstream 
recommendations.


Technically there are three separate places where the flags can be set, 
I think we should turn them on everywhere:

- CPython itself & its standard library
- The debug build (/usr/bin/python3-debug)
- Default for libraries in Fedora (RPM macros)
- Befault for libraries built by users (sysconfig settings)


[0]: https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
[1]: 
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/OOJDAKTJB5WGMOZRXTUX7FTPFBF3H7WE/

[2]: https://pagure.io/fesco/issue/2817
[3]: https://fedoraproject.org/wiki/Changes/Python3.12
[4]: 
https://docs.python.org/3.12/howto/perf_profiling.html#how-to-obtain-the-best-results

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Building Python 3.12 with no-omit-frame-pointer

[Petr Viktorin]

On 15. 11. 22 20:17, Kevin Kofler wrote:

tl;dr: Python 3.12 should be built with no-omit-frame-pointer if
upstream recommends it.


Absolutely not, because…


Apparently there are some benchmarks that make Python look extra slow
when the flags are turned on


… considering those benchmarks, Python is one of the programs for which it 
would be the *least* appropriate to enable frame pointers!


Oh, do you have more info on these benchmarks? The mentions I found were 
pretty vague.
Is it DaanDeMeyer/fpbench? It seems pyperformance is the majority of 
those benchmarks, so there's not much to compare to. It's also the 
benchmark suite that upstream uses, so I'm certain any upstream 
recommendation will take it into account.



Meanwhile, on the upstream side, Python 3.12 (due next year, main Python
for Fedora 39 [3]) has support for `perf`. Upstream plans to recommend
compiling with these flags when measuring performance [4], and AFAIK,
the plan is to recommend *always* compiling with them.
[3]: https://fedoraproject.org/wiki/Changes/Python3.12
[4]: 
https://docs.python.org/3.12/howto/perf_profiling.html#how-to-obtain-the-best-results


Looking at the details in the link, would it not be sufficient to have the perf 
trampolines compiled with frame pointers enabled (which will only affect runs with perf 
support enabled ar runtime)? Or actually, since they are "compiled on the fly", 
do those trampolines not always have a frame pointer anyway, no matter how Python is 
compiled?


That sounds like a question for upstream, do you want to bring it up there?
(I won't, as I don't think I could follow up well on the discussion.)


The idea is that possible speedups from "allowing anyone to
profile/optimize their workflow" are worth the initial slowdown.


I and others heavily dispute this claim. There is no evidence that anybody will 
be doing enough profiling and optimization to compensate for the up to 10% 
performance hit seen on performance-critical Python code in the benchmarks, nor 
that such big optimization is even possible to begin with. Also, profiling and 
optimizing individual Python programs (as opposed to the Python interpreter 
itself) will not help the vast majority of Python code out there at all.


I know enough about the pyperformance benchmarks to suspect that talking 
about the highest slowdown is just a scare tactic. Not all benchmarks 
are indicative of real-world use. The one that got closest to 10% 
(scimark_sparse_mat_mult, with a 9.5% hit [0]) is pure-Python matrix 
multiplication -- perhaps the best example of code you would never put 
in production today.



As far as I can see, performance geeks are enthusiastic for `perf`
support,


As you say yourself, this is a niche feature for "performance geeks".


and I'd like to get them to (continue to) use Fedora builds.


That is in theory a laudable goal, but not if it degrades the performance for 
everyone else.


If CPython upstream does recommend these flags (or makes them default),
I'm considering to turn the no-omit options on for Python 3.12 even if
Fedora as a whole doesn't.


Then I can only hope that FESCo will explicitly disallow that.


Note that even a 2% slowdown will likely won back by general performance
improvements – the Faster CPython team is targeting a 20% average
speedup for pure-Python code in 3.12, on top of the ~25% for 3.11. And
the people responsible for this speedup have a say in the upstream
recommendations.


Those 20% are a stated goal, not a guarantee. Even if attained, they will also 
not necessarily help all programs the same amount. And most importantly, Fedora 
will benefit from those upstream optimizations either way, even if it does not 
ship a build optimized for profiling rather than performance. So I do not see 
this as an argument for shipping Python with frame pointers.

I am also very sceptical of that "grabbing for the stars" approach of accepting a 2-10% 
performance loss in the hopes of getting a 20% performance win that may or may not be actually 
achievable. In German, we have a proverb that literally translates to: "Better the sparrow in 
the hand than the dove on the roof." It means that it is better to take the small thing that 
you already have than to throw it away for a bigger thing that you have yet to catch.


We don't “have” anything now, the Python 3.12 release is a year away.
I'm pretty sure the upstream recommendation will take the then-current 
state into account.


In fact, in your mail I don't really see anything Fedora-specific reason 
to override an upstream recommendation, when it comes.



[0]: 
https://github.com/DaanDeMeyer/fpbench/blob/7233a6cfcd01467b429e1a22d961e3f77b6f4868/README.md

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List 

Building Python 3.12 with no-omit-frame-pointer

[Petr Viktorin]

tl;dr: Python 3.12 should be built with no-omit-frame-pointer if 
upstream recommends it.



Hello,
You might be aware of a Fedora change proposal [0] (discussed on 
fedora-devel [1] and FESCo [2]) are discussing turning on C compiler 
flags that help with performance *measurement*, but might hurt 
performance itself: `-fno-omit-frame-pointer` and 
`-mno-omit-leaf-frame-pointer`.
Apparently there are some benchmarks that make Python look extra slow 
when the flags are turned on -- which I don't quite understand.


Meanwhile, on the upstream side, Python 3.12 (due next year, main Python 
for Fedora 39 [3]) has support for `perf`. Upstream plans to recommend 
compiling with these flags when measuring performance [4], and AFAIK, 
the plan is to recommend *always* compiling with them.
The idea is that possible speedups from "allowing anyone to 
profile/optimize their workflow" are worth the initial slowdown.


I'm not much of a performance expert myself, but I do get drawn into the 
relevant discussions on the CPython side.
As far as I can see, performance geeks are enthusiastic for `perf` 
support, and I'd like to get them to (continue to) use Fedora builds.
If CPython upstream does recommend these flags (or makes them default), 
I'm considering to turn the no-omit options on for Python 3.12 even if 
Fedora as a whole doesn't.
Note that even a 2% slowdown will likely won back by general performance 
improvements – the Faster CPython team is targeting a 20% average 
speedup for pure-Python code in 3.12, on top of the ~25% for 3.11. And 
the people responsible for this speedup have a say in the upstream 
recommendations.


Technically there are three separate places where the flags can be set, 
I think we should turn them on everywhere:

- CPython itself & its standard library
- The debug build (/usr/bin/python3-debug)
- Default for libraries in Fedora (RPM macros)
- Befault for libraries built by users (sysconfig settings)


[0]: https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
[1]: 
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/OOJDAKTJB5WGMOZRXTUX7FTPFBF3H7WE/

[2]: https://pagure.io/fesco/issue/2817
[3]: https://fedoraproject.org/wiki/Changes/Python3.12
[4]: 
https://docs.python.org/3.12/howto/perf_profiling.html#how-to-obtain-the-best-results

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: What if we excluded 32bit ARM from Python 3.12

[Petr Viktorin]

Makes sense. It's a new package. I see nothing wrong with excluding it, 
to see what breaks & who complains.



On 13. 10. 22 11:07, Miro Hrončok wrote:

Hello Pythonistas,

we are probably going to package python3.12 soon for all Fedora releases.

Unfortunately, building Python for 32bit ARM has been very tedious 
lately, as the Koji build keeps restarting and the build takes 24+ hours 
to finish.


Considering 32bit ARM is gone from Fedora 37+, I was considering the 
ExcludeArch it from the python3.12 package even on older Fedoras, to 
make the builds easier.


Would anybody be sad about that?


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Cython build failing on longintrepr.h not found

[Petr Viktorin]

The guidelines have more info: 
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_using_cython



On 27. 09. 22 8:17, Lumír Balhar wrote:

Hi Sandro.

Make sure that the build does not use the pyx file from upstream. It 
seems to me that the file generated by Cython is in the source tarball 
(skmisc/loess/src/_loess.pyx) and I did not find any mention of use of 
Cython in the build log. The file is probably generated by an older 
version of Cython and that is causing you the problem.


Hope this helps.

Have a nice day.

Lumír

On 9/27/22 01:48, Sandro wrote:

Hi Pythonians,

I have a favor to ask. I'm trying to package scikit-misc as a 
dependency for plotnine (for neuro-sig).


The build fails with:

fatal error: longintrepr.h: No such file or directory

I'm aware of the fact that the file has been moved in Python 3.11, but 
was under the impression this change was implemented in Cython 0.29.5.


I opened an issue upstream, but I would appreciate a second pair of 
eyes on the build. The file that tries to include the header file is 
generated, so nothing I could work around with using a patch (with my 
limited knowledge that is).


Build: 
https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4869979/

Issue: https://github.com/has2k1/scikit-misc/issues/17

-- Sandro
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to 
python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Orphaned python packages

[Petr Viktorin]

On 31. 08. 22 13:41, Ali Erdinc Koroglu wrote:
- pytest-capturelog not even exits in pypi [2] so can we retire it and 
use pytest-catchlog instead ?


Yes. Except pytest-catchlog is retired too, because pytest itself 
provides the functionality now: 
https://docs.pytest.org/en/latest/how-to/logging.html#caplog-fixture
And it seems that nothing depends on pytest-capturelog in Fedora, so it 
should be safe to retire.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Replacing pytest -n auto with pytest -n %{_smp_build_ncpus}

[Petr Viktorin]

On 07. 07. 22 14:44, Miro Hrončok wrote:

On 07. 07. 22 14:00, Petr Viktorin wrote:

On 07. 07. 22 11:38, Miro Hrončok wrote:

Hello Pythonistats, packagers,

A handful of Fedora Python packages uses pytest-xdist to run tests in 
parallel like this:


   %pytest -n auto

-n auto means pytest will spawn a number of workers processes equal 
to the number of available CPUs.


In the spirit of other packaging guidelines, I believe we should use 
this instead:


   %pytest -n %{_smp_build_ncpus}

This means the same thing in most of the ceases, but will limit the 
number of workers depending on other constraints in the spec or in 
the environment.


Should I do this in a mass change? Not so many packages use pytest -n 
auto in the spec:


   $ rg -l -- '(-n|--numprocesses)(\s*|=)auto(\s|$)'
   ansible-bender.spec
   azure-cli.spec
   ocrmypdf.spec
   python-cartopy.spec
   python-GridDataFormats.spec
   python-hypothesis.spec
   python-matplotlib.spec
   python-mplcairo.spec
   python-rpmautospec.spec
   python-sqlalchemy.spec
   python-tox.spec
   python-xarray.spec
   python-zstandard.spec
   pythran.spec
   scipy.spec

(Other packages have that in tox.ini or pytest config file, but I am 
not aiming at changing that here.)


And, considering many other packages might want to benefit from that, 
should this be:


  1) encouraged in the Python packaging guidelines
  2) macronized (I was thinking %pytest_parallel, but TBD)


Or pytest-xdist could be taught to check an environment variable for 
`auto`, making this seamless for packagers?


This could work, except that sometimes pytest-xdist is installed and we 
don't want to run tests in parallel because they are not prepared for 
that. But I guess an opt-out would still exist, e.g. setting 
%{_smp_build_ncpus} to 1 in the %check section.


I will have a look if there is an environment variable we could use. We 
can probably adjust PYTEST_ADDOPTS if we detect pytest-xdist is available.


I meant teaching pytest-xdist to look at an envvar, which would just be 
ignored if xdist isn't around.
Actually I just went ahead and proposed it upstream: 
https://github.com/pytest-dev/pytest-xdist/issues/792

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Python 3.11 final release might be delayed to December

[Petr Viktorin]

On 05. 07. 22 15:54, Richard Shaw wrote:
On Tue, Jul 5, 2022 at 8:07 AM Richard W.M. Jones > wrote:


On Tue, Jul 05, 2022 at 01:17:39PM +0200, Miro Hrončok wrote:
 > Hello,
 > forwarding this  message to Fedora.
 >
 > Will know more by the end of this week -- we might need to consider
 > reverting back to Python 3.10 if we don't want to ship Fedora 37 GA
 > with a beta version of Python :(

Is there a reason why shipping a beta version of Python would be bad?
I've been using it on my main development machine for a few weeks and
for the (fairly limited) Python stuff I do it seems to be fine.

It'd be a problem if it was causing bugs.


Would it not make sense then to review the blocker bugs  and then 
determine if they are blockers for us?


The issue is not bug(s but ABI compatibility. We can't update from a 
Beta to (say) 3.11.1 in stable Fedora, because all PYC files and 
extension modules would break if the C-ABI or PYC files would break.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Replacing pytest -n auto with pytest -n %{_smp_build_ncpus}

[Petr Viktorin]

On 07. 07. 22 11:38, Miro Hrončok wrote:

Hello Pythonistats, packagers,

A handful of Fedora Python packages uses pytest-xdist to run tests in 
parallel like this:


   %pytest -n auto

-n auto means pytest will spawn a number of workers processes equal to 
the number of available CPUs.


In the spirit of other packaging guidelines, I believe we should use 
this instead:


   %pytest -n %{_smp_build_ncpus}

This means the same thing in most of the ceases, but will limit the 
number of workers depending on other constraints in the spec or in the 
environment.


Should I do this in a mass change? Not so many packages use pytest -n 
auto in the spec:


   $ rg -l -- '(-n|--numprocesses)(\s*|=)auto(\s|$)'
   ansible-bender.spec
   azure-cli.spec
   ocrmypdf.spec
   python-cartopy.spec
   python-GridDataFormats.spec
   python-hypothesis.spec
   python-matplotlib.spec
   python-mplcairo.spec
   python-rpmautospec.spec
   python-sqlalchemy.spec
   python-tox.spec
   python-xarray.spec
   python-zstandard.spec
   pythran.spec
   scipy.spec

(Other packages have that in tox.ini or pytest config file, but I am not 
aiming at changing that here.)


And, considering many other packages might want to benefit from that, 
should this be:


  1) encouraged in the Python packaging guidelines
  2) macronized (I was thinking %pytest_parallel, but TBD)


Or pytest-xdist could be taught to check an environment variable for 
`auto`, making this seamless for packagers?

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Help needed: liquidctl and python3.11

[Petr Viktorin]

On 30. 06. 22 15:13, Artur Frenszek-Iwicki wrote:

Some time ago we've had python3.11 land in Rawhide, with a rebuild of dependent 
packages.
One of my packages, liquidctl, fails to build with the new Python, and since my 
knowledge of the language
stops at "I used it to rewrite some bash scripts", I'd need some help.


Hello,
I replied on the bug: https://bugzilla.redhat.com/show_bug.cgi?id=2098741#c3
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Should python3dist(...) provides generator fail when the version is = 0?

[Petr Viktorin]

On 29. 01. 22 22:11, Miro Hrončok wrote:

Hello Pythonistas,

today, I've looked up packages in rawhide providing python3dist(...) = 0 
and I opened bugzillas for them:


https://bugzilla.redhat.com/showdependencytree.cgi?id=python3dist0

While version 0 (or equal versions like 0.0 or 0.0.0) is probably 
technically valid, it most certainly indicates a packaging error (most 
likely but not necessarily a downstream packaging error).


Should we prevent this error from happening by explicitly erroring (and 
failing the build) when it happens? I think it would make the dependency 
generators more robust.


In an unlikely scenario when packagers actually want to package version 
0, they can reach out and we can allow it via some configuration (but 
[YAGNI], so I don't want to clutter the generator with yet another 
option right away).


[YAGNI] https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it



Sounds like a great idea.
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Does anybody still use `starship'?

[Petr Viktorin]

On 10. 01. 22 18:41, Dan Čermák wrote:

Otto Urpelainen  writes:


Igor Raits kirjoitti 9.1.2022 klo 13.24:

Hello,

I'm interested to hear if there are any users of the `starship'
application here in Fedora that consume it from the repositories.
Please speak up if you do!


I use it, too.


Same here and very thankful for being able to do that!


Same here! And I'm happy with what the Fedora version can do.
Thank you!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Adding python3.10dist(pp) to python-ppft

[Petr Viktorin]

On 07. 12. 21 22:42, Ankur Sinha wrote:

Hi folks,

I was looking to unretire the unmaintained python-pp because some neuro
packages still use it but then was reminded by @music that we already
provide python-ppft which is a maintained fork and drop-in replacement
for pp.

https://github.com/uqfoundation/ppft

Currently python3-ppft provides these:

$ rpm -q --provides python3-ppft
python-ppft = 1.6.6.4-2.fc35
python3-ppft = 1.6.6.4-2.fc35
python3.10-ppft = 1.6.6.4-2.fc35
python3.10dist(ppft) = 1.6.6.4
python3dist(ppft) = 1.6.6.4

Can we also provide all these for `pp` so that packages in Fedora that
require pp automatically use ppft?

I tried adding `%py_provides pp` to python3-ppft, and while that does
add a few of these, it does not add the python3.10dist(pp) or
python3dist(pp) ones (which are autogenerated from the metadata?):

$ rpm -q --provides -p 
./results_python-ppft/1.6.6.4/3.fc36/python3-ppft-1.6.6.4-3.fc36.noarch.rpm
python-pp = 1.6.6.4-3.fc36
python-ppft = 1.6.6.4-3.fc36
python3-pp = 1.6.6.4-3.fc36
python3-ppft = 1.6.6.4-3.fc36
python3.10-pp = 1.6.6.4-3.fc36
python3.10-ppft = 1.6.6.4-3.fc36
python3.10dist(ppft) = 1.6.6.4
python3dist(ppft) = 1.6.6.4

Would there be a way to include these machine readable provides too
since these forms are used by other packages in the automatically
generated requires etc. (if this is the right thing to do)?

I guess the alternative would be that every package that requires pp be
patched to require ppft, but tweaking a single python-ppft package would
be less work. :)



Hi,
These are generated from Python metadata, e.g. install_requires in 
setuptools: 
https://github.com/uqfoundation/ppft/blob/a4dd5832103512f6df717637b03fb507a00efd84/setup.py#L200


If you only add the RPM provides, tools that read the Python-specific 
metadata might complain about not finding pp. And unfortunately Python 
packaging doesn't have good support for "virtual provides" 
(Provides-Dist) yet.


So it looks like it would be best to patch setup.py (or equivalent) in 
all dependent packages. And perhaps push the change upstream, if ppft is 
better than pp. (I don't know anything about these packages.)

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Python RPM dependency generators and LegacyVersion, time to error?

[Petr Viktorin]

On 04. 11. 21 12:49, Miro Hrončok wrote:

Hello Pythonistas.

After some recent improvements in the Python RPM dependency generators, 
a regression was discovered [0].


Turns out the error happened when the upstream metadata contained a 
requirement with a PEP 440 [1] incompatible version. A fix is pending 
and only one package in Fedora was affected.



[...]


tl;dr:

  - there might be cases where legacy versions are needed
  - but in most cases, they should be avoided

Hence, I propose we do the following in Rawhide:

We turn LegacyVersions to failures, but we provide a stop-gap measure, 
such as (%global python_dependency_allow_legacy_version_provides 1 /  
%global python_dependency_allow_legacy_version_requires 1 ) for packages 
that need to override this. When pypa/packaging actually drops 
LegacyVersion, this stop-gap measure will no longer work.



I suggest removing LegacyVersion support without replacement. I think 
neither creating nor using 
"python_dependency_allow_legacy_version_requires" would be worth the 
extra work.
All the uses of it that we've seen were mistakes, and since upstream 
tools are removing support for it, the stop-gap is just delaying the 
inevitable.

If we find an actual use case for it, it can always be added.
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Using reverse week deps for Python interpreters and tox (Supplements instead of Recommends)?

[Petr Viktorin]

On Wed, Aug 11, 2021 at 3:00 PM Miro Hrončok  wrote:
>
> Hello Pythonistas.
>
> Currently, the tox package has:
>
># Recommend "all the Pythons"
>Recommends: python2.7
>Recommends: python3.6
>Recommends: python3.7
>Recommends: python3.8
>Recommends: python3.9
>Recommends: python3.10
>Recommends: pypy2-devel
>Recommends: pypy3-devel
>Recommends: python2-devel
>Recommends: python3-devel
>
> Every time we add or remove a Python interpreter, we need to change the list:
>
> E.g.
> https://src.fedoraproject.org/rpms/python-tox/c/7220117b812d3b5b009b3b1170ee5242b03efcb2
>
>
> What if we added this to all interpreters that work with our tox instead?
>
>Supplements: tox
>
> That way:
>
>   - When we introduce a new interpreter,
> tox starts to pull it by default right away.
>
>   - When different Fedoras (or even EPELs) have different interpreter sets,
> we don't need to divert the tox spec.
>
> OTOH:
>
>   - When we add a new interpreter that Supplements tox, we would still need to
> adapt tox's CI config to include it. Maybe we could adapt the CI tests for tox
> to automatically test with all interpreters that supplement it.
>
> WDYT?
>

This looks very reasonable!
Generating the list automatically in tests sounds a bit fragile. For
tests, it might be better to have a manually updated list, and
automatically check if it is up to date.
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: tzdata-minimal (Self-Contained Change proposal)

[Petr Viktorin]

On 15. 07. 21 20:33, Carlos O'Donell wrote:

On 7/15/21 6:34 AM, Zbigniew Jędrzejewski-Szmek wrote:

On Tue, Jul 06, 2021 at 01:20:47PM -0400, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/tzdata-minimal

== Summary ==
Split the tzdata package into two parts - tzdata and tzdata-minimal.
tzdata will require tzdata-minimal.  tzdata-minimal provides the
minimal files needed to support UTC on containers.

== Owner ==
* Name: Patsy Griffin (Franklin)
* Email: pa...@redhat.com


== Detailed Description ==
This is the first step towards providing support for a minimal, UTC
only, version of tzdata for containers.  The tzdata-minimal package
will be a stand-alone, UTC only, subset of tzdata. The tzdata package
will require tzdata-minimal.

With this framework in place, other packages can develop code to
detect a minimal tzdata installation.  These packages will also need
to provide appropriate messages when users request timezone
information not available when only tzdata-minimal is installed.


In general, I like the idea of making it easier to not install tzdata.
For many machines it's completely unnecessary, and 5MB is enough to
make this worthwhile.


Thanks for supporting the idea.


I don't know enough about all the consumers to comment on the details.
FWIW, systemd-timedated already has a reasonable message that encompasses
both possible errors:

   $ timedatectl set-timezone Europe/Warsaw2
   Failed to set time zone: Invalid or not installed time zone 'Europe/Warsaw2'

But I think it'd be useful to use a different message if we can
distinguish the two cases. I'd be happy to take a patch...


Yes, you could say "invalid" (zone table files installed) if the zone is not
in one of the tables, and "invalid or not installed" if there are no files
(no tzdata installed).


Without the split (i.e. if either all or none of tzdata is installed), 
it would be similar:
If a zone1970.tab exists but doesn't list the zone (or, hopefully 
equivalently: a zone1970.tab exists but there's no file for the zone): 
it's "Invalid time zone".

If the table doesn't exist: it's "Invalid or not installed".


We are still in the process of checking with upstream tzdata if they are
comfortable with, and will support *not* installing some of the zones, and
document that upstream.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: tzdata-minimal (Self-Contained Change proposal)

[Petr Viktorin]

On 14. 07. 21 12:43, Jonathan Wakely wrote:

On Tue, 13 Jul 2021 at 21:42, Carlos O'Donell wrote:

(1) Missing files vs. No /usr/share/zoneinfo.

All existing third-party software must handle zone name changes today,
so there must be some way to handle the error that a given zone is
missing (changed names).

Let me talk a bit about the new C++ time zone API I have been looking
at with Jonathan Wakely.

Errors based on this will be likely correct e.g. get_tz_dir() from the
currently proposed C++ API for this (see (2)):


N.B. The functions below are not part of the standardized API, they
are implementation details of a specific implementation of that API.



  353 CONSTDATA auto tz_dir_default = "/usr/share/zoneinfo";
  354 CONSTDATA auto tz_dir_buildroot = "/usr/share/zoneinfo/uclibc";
  355
  356 // Check special path which is valid for buildroot with uclibc builds
  357 if(stat(tz_dir_buildroot, ) == 0 && S_ISDIR(sb.st_mode))
  358 return tz_dir_buildroot;
  359 else if(stat(tz_dir_default, ) == 0 && S_ISDIR(sb.st_mode))
  360 return tz_dir_default;
  361 else
  362 throw runtime_error("discover_tz_dir failed to find zoneinfo\n");

Having tzdata uninstalled will throw the generic error as on line 362.

But having tzdata-minimal will instead throw a specific error from line 3624:

3624 throw std::runtime_error(std::string(tz_name) + " not found in timezone 
database");

This probably supports having tzdata removed entirely, since the latter
error makes it seems as-if the zone name is wrong (it's not, it's just
not installed).

Notes:
https://github.com/HowardHinnant/date

(2) Non-rpm packaging.

We have C++ users using Howard Hinnart's 'date' package that implements
 header and it can process complete IANA tzdb files.

Currently Howard's API is being proposed for inclusion
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0355r7.html


N.B. This is incorrect, it was proposed some years ago (the paper
above is from 2018) and is now part of ISO/IEC 14882:2020 i.e. the
C++20 standard.

I'm not yet sure what the C++ runtime will need to do if there are no
zoneinfo files at all, not even UTC. I suspect we will need a fallback
to UTC as well, and std::tzdb::current_zone() will return the info for
the UTC timezone. So I'm not sure that there's much practical
difference between no tzdata at all and tzdata-minimal as far as
libstdc++ will be concerned. This might change once I actually start
to implement it though. Maybe having tzdata-minimal would allow the
initial implementation to omit the fallback (assuming that all
sensible linux environments have at least UTC info), and add the
fallback later.

(It doesn't help that glibc has a fallback to UTC because the C++
runtime can't use glibc for timezone info, it will consume the tzdata
files directly.)


It doesn't help Python's zoneinfo either, for the same reason.
I assume anything that uses the glibc API shouldn't depend on tzdata 
directly.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: tzdata-minimal (Self-Contained Change proposal)

[Petr Viktorin]

On 13. 07. 21 16:31, Carlos O'Donell wrote:

On 7/12/21 12:16 PM, Petr Viktorin wrote:

On 06. 07. 21 20:38, David Cantrell wrote:

On Tue, Jul 06, 2021 at 01:20:47PM -0400, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/tzdata-minimal

== Summary ==
Split the tzdata package into two parts - tzdata and tzdata-minimal.
tzdata will require tzdata-minimal.  tzdata-minimal provides the
minimal files needed to support UTC on containers.

[...]



Python should work well without *any* tzdata installed. But having
only a small subset of timezones would result in issues unique to
Fedora-based systems (assuming those are the only systems that split
tzdata).


It's possible to remove all tzdata.

However, without *some* data it is not easy to distinguish between these
two scenarios if you want to offer a different error message:

- Is this zone provided and correct but missing? e.g. exists but is not 
installed.
   - zone table exists, lists the zone, but the zone is missing.

- Is this zone not correct? e.g. doesn't exist in the current version.
   - zone table doesn't list the zone.


True. But I don't see a use case where this would be important enough: 
when is an error message like "tzdata is not installed, so the zone 
can't be used *or* checked" not enough?


The problem with making it possible to distinguish between these two 
scenarios is that *every* consumer of tzdata must now distinguish 
between them. If they assume tzdata is never split (as many do, AFAIK), 
their error messages will be wrong.
That's not every consumer we distribute in a RPM (we could handle that I 
guess), but also any third-party one people will build/install on their 
systems.



We could theoretically patch Python to give reasonable error
messages. But since the consumer of the tz data (the zoneinfo module)
was only added in Python 3.9 (last year), existing applications
mainly use third-party modules instead of the standard library. I
assume that like Python, these modules expect tzinfo to either be
missing entirely or be all there. And I expect this is the case for
more than just Python modules.
  
Over time some zones become deprecated and invalid names.


One must already handle zone name changes, so if the code can handle
name deprecation then it will report the same error for missing zoneinfo. >

Is it reasonable for glibc to hardcode the +0 fallback timezone,
rather than needing the zoneinfo file for it? If so, we could remove
tzdata from minimal containers entirely. Or is that too naive?


It is not naive.

glibc already falls back to UTC with no data present (and we need to cleanup
what we print).

The question is what kind of errors we want to be able to express to users.
  

== Benefit to Fedora ==
This change will reduce the size of base container installations.

== Scope ==
* Proposal owners: Implement the proposal.
* Other developers: Developers need to ensure that their packages
continue to build and install with the new split tzdata/tzdata-minimal
package changes.

* Release engineering: No coordination required with Release Engineering.
* Policies and guidelines: The policies and guidelines do not need to
be updated.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A

== Upgrade/compatibility impact ==
The only visible change will be a new package tzdata-minimal required by tzdata.


== How To Test ==
Run a dnf upgrade of tzdata and observe that tzdata-minimal is now
also installed as a dependency.


== User Experience ==
Users will see that new updates to tzdata include a new package
dependency on tzdata-minimal.


== Dependencies ==
This change does not require or depend on changes to other packages.
However, we hope that dependent packages will work towards
recommending tzdata for builds and installs rather than requiring it.


== Contingency Plan ==
* Contingency mechanism: If we are unable to complete this feature by
the final development freeze, we will revert to the shipped
configuration.
* Contingency deadline: 100% Code complete deadline
* Blocks release? No

== Documentation ==
No documentation changes are needed at this time.


== Release Notes ==
The tzdata package is now divided into a UTC only package,
tzdata-minimal, and tzdata.


What is supposed to be in tzdata-minimal?  Is it
/usr/share/zoneinfo/UTC or that and more?


Slightly more (based on experience).

* UTC
/usr/share/zoneinfo/UTC

* Zone tables (which can be used to determine what is valid).
/usr/share/zoneinfo/iso3166.tab
/usr/share/zoneinfo/zone.tab
/usr/share/zoneinfo/zone1970.tab

* License file
/usr/share/licenses/tzdata/LICENSE

It saves ~5MiB for base container image sizes in a supportable way
(as opposed to what is done today which is to delete the zoneinfo data).



Forcibly removing tzdata on a fresh Fedora VM that I just set up has
the system fall back to UTC, as expected.  On this incredibly small
install, tzdata is required glibc-common, python3-libs, and
pyt

Re: F35 Change: tzdata-minimal (Self-Contained Change proposal)

[Petr Viktorin]

On 12. 07. 21 18:31, Carlos O'Donell wrote:

On 7/6/21 2:38 PM, David Cantrell wrote:

On Tue, Jul 06, 2021 at 01:20:47PM -0400, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/tzdata-minimal

== Summary ==
Split the tzdata package into two parts - tzdata and tzdata-minimal.
tzdata will require tzdata-minimal.  tzdata-minimal provides the
minimal files needed to support UTC on containers.

== Owner ==
* Name: Patsy Griffin (Franklin)
* Email: pa...@redhat.com


== Detailed Description ==
This is the first step towards providing support for a minimal, UTC
only, version of tzdata for containers.  The tzdata-minimal package
will be a stand-alone, UTC only, subset of tzdata. The tzdata package
will require tzdata-minimal.

With this framework in place, other packages can develop code to
detect a minimal tzdata installation.  These packages will also need
to provide appropriate messages when users request timezone
information not available when only tzdata-minimal is installed.

== Feedback ==
We have had requests for this functionality in order to support
minimal container installations.  Currently some container kickstart
installations already ad hoc remove most of the timezone information
provided by tzdata, leaving only UTC support available.  This change
provides a formal method of providing this support.

Both the glibc and python teams are aware of this proposed change.
This change does not currently require changes in their code.  The
goal is for those packages that currently require tzdata as part of
their build or install, move towards recommending tzdata instead.

== Benefit to Fedora ==
This change will reduce the size of base container installations.

== Scope ==
* Proposal owners: Implement the proposal.
* Other developers: Developers need to ensure that their packages
continue to build and install with the new split tzdata/tzdata-minimal
package changes.

* Release engineering: No coordination required with Release Engineering.
* Policies and guidelines: The policies and guidelines do not need to
be updated.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A

== Upgrade/compatibility impact ==
The only visible change will be a new package tzdata-minimal required by tzdata.


== How To Test ==
Run a dnf upgrade of tzdata and observe that tzdata-minimal is now
also installed as a dependency.


== User Experience ==
Users will see that new updates to tzdata include a new package
dependency on tzdata-minimal.


== Dependencies ==
This change does not require or depend on changes to other packages.
However, we hope that dependent packages will work towards
recommending tzdata for builds and installs rather than requiring it.


== Contingency Plan ==
* Contingency mechanism: If we are unable to complete this feature by
the final development freeze, we will revert to the shipped
configuration.
* Contingency deadline: 100% Code complete deadline
* Blocks release? No

== Documentation ==
No documentation changes are needed at this time.


== Release Notes ==
The tzdata package is now divided into a UTC only package,
tzdata-minimal, and tzdata.


What is supposed to be in tzdata-minimal?  Is it
/usr/share/zoneinfo/UTC or that and more?


A little more is provided.

You need some of the name tables for certain use cases.


What are the use cases (and why are they needed for minimal containers)?


This is based on Intel's clear linux work in splitting the data.
  

Forcibly removing tzdata on a fresh Fedora VM that I just set up has
the system fall back to UTC, as expected.  On this incredibly small
install, tzdata is required glibc-common, python3-libs, and
python3-dateutil.  The last one is reasonable, but for all of them I
ask if tzdata is actually a hard dependency or if it can become a weak
dependency and this change proposal could become "make tzdata
something easily removable" rather than creating more tzdata packages.


If you can minimally provide the tables of possible zones, and provide
an easy way to detect a zone is missing, then the APIs can determine:
"Yes you could do that, but your system is missing the data."
vs.
"That is an invalid zone."
Which is a useful distinction.


Sure, we can provide such an API, but that alone won't make consumers 
use it. Especially if it's specific to Fedora (and Clear Linux -- AFAIK, 
people expect missing bits and pieces are there.)


I think the best place to discuss and document this would be the tz. If 
their README (or just a mail thread) said the data can be split this 
way, and hinted at what should be done when it is, we wouldn't need this 
conversation.
Without that, I don't think it's OK to allow Fedora systems with only 
tzdata-minimal.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List 

Re: F35 Change: tzdata-minimal (Self-Contained Change proposal)

[Petr Viktorin]

On 06. 07. 21 20:38, David Cantrell wrote:

On Tue, Jul 06, 2021 at 01:20:47PM -0400, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/tzdata-minimal

== Summary ==
Split the tzdata package into two parts - tzdata and tzdata-minimal.
tzdata will require tzdata-minimal.  tzdata-minimal provides the
minimal files needed to support UTC on containers.

== Owner ==
* Name: Patsy Griffin (Franklin)
* Email: pa...@redhat.com


== Detailed Description ==
This is the first step towards providing support for a minimal, UTC
only, version of tzdata for containers.  The tzdata-minimal package
will be a stand-alone, UTC only, subset of tzdata. The tzdata package
will require tzdata-minimal.

With this framework in place, other packages can develop code to
detect a minimal tzdata installation.  These packages will also need
to provide appropriate messages when users request timezone
information not available when only tzdata-minimal is installed.

== Feedback ==
We have had requests for this functionality in order to support
minimal container installations.  Currently some container kickstart
installations already ad hoc remove most of the timezone information
provided by tzdata, leaving only UTC support available.  This change
provides a formal method of providing this support.

Both the glibc and python teams are aware of this proposed change.
This change does not currently require changes in their code.  The
goal is for those packages that currently require tzdata as part of
their build or install, move towards recommending tzdata instead.


Python should work well without *any* tzdata installed.
But having only a small subset of timezones would result in issues 
unique to Fedora-based systems (assuming those are the only systems that 
split tzdata).


We could theoretically patch Python to give reasonable error messages. 
But since the consumer of the tz data (the zoneinfo module) was only 
added in Python 3.9 (last year), existing applications mainly use 
third-party modules instead of the standard library. I assume that like 
Python, these modules expect tzinfo to either be missing entirely or be 
all there.

And I expect this is the case for more than just Python modules.


Is it reasonable for glibc to hardcode the +0 fallback timezone, rather 
than needing the zoneinfo file for it? If so, we could remove tzdata 
from minimal containers entirely.

Or is that too naive?



== Benefit to Fedora ==
This change will reduce the size of base container installations.

== Scope ==
* Proposal owners: Implement the proposal.
* Other developers: Developers need to ensure that their packages
continue to build and install with the new split tzdata/tzdata-minimal
package changes.

* Release engineering: No coordination required with Release Engineering.
* Policies and guidelines: The policies and guidelines do not need to
be updated.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A

== Upgrade/compatibility impact ==
The only visible change will be a new package tzdata-minimal required 
by tzdata.



== How To Test ==
Run a dnf upgrade of tzdata and observe that tzdata-minimal is now
also installed as a dependency.


== User Experience ==
Users will see that new updates to tzdata include a new package
dependency on tzdata-minimal.


== Dependencies ==
This change does not require or depend on changes to other packages.
However, we hope that dependent packages will work towards
recommending tzdata for builds and installs rather than requiring it.


== Contingency Plan ==
* Contingency mechanism: If we are unable to complete this feature by
the final development freeze, we will revert to the shipped
configuration.
* Contingency deadline: 100% Code complete deadline
* Blocks release? No

== Documentation ==
No documentation changes are needed at this time.


== Release Notes ==
The tzdata package is now divided into a UTC only package,
tzdata-minimal, and tzdata.


What is supposed to be in tzdata-minimal?  Is it
/usr/share/zoneinfo/UTC or that and more?

Forcibly removing tzdata on a fresh Fedora VM that I just set up has
the system fall back to UTC, as expected.  On this incredibly small
install, tzdata is required glibc-common, python3-libs, and
python3-dateutil.  The last one is reasonable, but for all of them I
ask if tzdata is actually a hard dependency or if it can become a weak
dependency and this change proposal could become "make tzdata
something easily removable" rather than creating more tzdata packages.


For python3-libs, I would have no issue with tzdata a weak dependency.
For users of python3-dateutil, I'd suggest using the Python standard 
library instead, where possible.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: 

Re: An unsuccessful case study: Using pyproject-rpm-macros with PyQt-builder and sip 5

[Petr Viktorin]

Hi,
This looks like the pip issue for passing options to backends (like 
sip-build, presumably): https://github.com/pypa/pip/issues/5771




On 23. 06. 21 13:17, Miro Hrončok wrote:

Hello,
since Scott is working on update to sip 5 (thank you!) I've figured this 
is a good time check if we can use the PyQt-builder Python build backend 
with our pyproject-rpm-macros.


tl;dr It is broken in various ways and I am unsure where to attempt to 
fix things.


-

This was my case study:

I've used the python-pyqtchart package as a test subject, as it is 
fairly simple and has an open pull request to port it to sip5 [1].


I've pretty much reconstructed the spec file as I would normally do:

   %generate_buildrequires
   %pyproject_buildrequires

   %build
   %pyproject_wheel

   %install
   %pyproject_install

To ease things up, I've kept the non-Python-packages BuilRequires 
intact, so the package still BuildRequires python3-qt5-devel, 
qt5-qtcharts-devel and qt5-qtbase-private-devel. I've removed the 
BuildRequires on python3dist(sip) and python3dist(pyqt-builder) as I've 
expected those to be generated.


The simplified pyproject.toml file looks like this:

   [build-system]
   requires = ["sip >=5.3, <7", "PyQt-builder >=1.6, <2"]
   build-backend = "sipbuild.api"

   [tool.sip.metadata]
   name = "PyQtChart"
   version = "5.15.2"
   ...
   requires-dist = "PyQt5 (>=5.15)"

   [tool.sip]
   project-factory = "pyqtbuild:PyQtProject"

   [tool.sip.project]
   tag-prefix = "QtChart"

   [tool.sip.bindings.QtChart]
   qmake-QT = ["charts"]

I've also tried with version 5.15.4, but not much is different.

I've used the copr with sip5 [2] to build the package.

-

[...] build
backend that does not support the prepare_metadata_for_build_wheel hook. 
Bummer, but fair.


We can RFE PyQt-builder to add that hook.

-

[...] PyQt-builder calls
"qmake" and that command was not available. I've looked into the code 
and the "qmake" name is hardcoded, but searched on $PATH.


The "original" non-pyproject spec has:

   sip-build ... --qmake="%{_qt5_qmake}"

But there is no (known) way to pass such options to the backend.


[...] workaround exists.


-

During %pyproject_wheel, "make" is called internally by the backend. The 
build is not parallelized. I was able to workaround this by using 
MAKEFLAGS='%{?_smp_mflags}'.


We might want to set this environment variable to '%{_make_output_sync} 
%{?_smp_mflags} %{_make_verbose}' in the %pyproject_wheel macro, so any 
backed that utilizes make gets the default arguments Fedora packages 
should use.


Anyway, this is not a blocker to build the package.

-

   Processing files: python-pyqtchart-debugsource-5.15.4-1.fc35.x86_64
   RPM build errors:
   error: Empty %files file /builddir/build/BUILD/PyQtChart-5.15.4
   /debugsourcefiles.list

This is a problem I get next. It likely means the build flags are not 
propagated.
Our macros set the CFLAGS and LDFLAGS environment variables, but the 
build does not respect them. I've tried exporting CXXFLAGS as well, but 
no dice.


Apparently, the build backend does not pass the variables to the build.

I've tried a workaround like this:

   export MAKEFLAGS='%{?_smp_mflags} CPPFLAGS="%{build_cxxflags}"'

But it makes the internal make call just dump the targets instead of 
actually building, so I guess I am doing it wrong. Either way, I think 
we need to fix PyQt-builder/sip to respect the flags from the 
environment variables.


Funnily enbough, when I break make like this, it creates the wheel 
anyway, but with no dist-info metadata in it. I guess if users break 
make by setting wrong MAKEFLAGS, the wheel should not be created at all :/


To move forward, I've used %global debug_package %{nil}, but I consider 
not using the Fedora's flag a blocker before we recommend the macros for 
packages that use PyQt-builder.


-

The next problem I got was:

   error: File not found: 
/builddir/build/BUILDROOT/python-pyqtchart-5.15.4-1.fc35.x86_64/usr/share/qt5/qsci/api/python/PyQtChart.api 



The "original" non-pyproject specfile had:

   sip-build ... --api-dir=%{_qt5_datadir}/qsci/api/python

And as said previously, I don't know a way to pass such options to the 
backend.


I am not skilled enough in Qt and SIP, so I have no idea if we need to 
ship the .api file or not. But if we need or want to, we should be able 
to. When the option is not used, the api file is not installed anywhere. 
I was unable to figure this out yet.


-

That's it. I don't really know where to fix things, and I'd appreciate 
some pointers. This is not a very high priority for me right now, but 
once the new Python guidelines [4] are in place, I think we should make 
this work, possibly with some reasonable worarounds.


[1] https://src.fedoraproject.org/rpms/python-pyqtchart/pull-request/6
[2] https://copr.fedorainfracloud.org/coprs/swt2c/pyqt5-sip5/
[3] 

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

On 15. 06. 21 13:48, Neal Gompa wrote:

On Tue, Jun 15, 2021 at 6:50 AM Petr Viktorin  wrote:


Hi Neal,
We had this conversation in the past (and you can see it in the change).
I don't think I can convince you, but I'll reiterate since it's new for
devel@.

Unlike the "mandatory tests" issue elsewhere in this thread, using the
PyPI namespace is the main point of the change. I can't take it out.


On 15. 06. 21 2:11, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:


On 6/14/21 1:53 PM, Neal Gompa wrote:

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.



I don't think we can reasonably separate the two.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

(I believe this is the "namespace" issue mentioned in the proposed change.)


You can. When it comes to distribution packages, we can maintain
consistency of the package names referenced within the distribution.
The point of this was to be able to use the names Python packages call
themselves for dependencies. PyPI is merely a distribution center for
such things. Fedora is another distribution center for such things.


You can only separate the namespaces if you sacrifice interoberability,
i.e. allowing pip-installable packages to depend on Fedora packages.
It turns out you can already do that (venv --system-site-packages), and
users expect that dist-info project names use the PyPI namespace. This
is a current problem that we need to solve.

There are other (harder) solutions for the issue than reusing the PyPI
namespace in Fedora, but why should we bother with them? There already
is an ecosystem-wide namespace; why should we add a Fedora-specific one?
(Or a Linux-distro-specific one, if we're interested in cross-distro
collaboration.)

You seem to assume that registering/blocking a name on PyPI is hard.
I say it is not and I am willing to do it for packagers who need it.
(And to mitigate the bus factor, I'll make it a responsibility of Red
Hat's python-maint team.)


It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.


I agree, in fact, I think Fedora's problems here are a subset of the
problems the private organizations have: if issues of proprietary corps
are solved, we can use the solution as well. (However, it'll need more
work than is necessary for Fedora/FOSS needs, so I don't want to drive
the effort.)
BUT, if the issues are solved, it'll likely be through namespacing: we'd
need to prefix our names with `fedora-` or `fedora:`. I still think it
is better for Fedora to reuse the public PyPI namespace rather than
start its own.


As long as you don't make packagers deal with it, I'm fine with it.
I'm okay with the PyPI contract as long as you don't make me or any
other Fedora packager specifically deal with it. And I strongly
suspect you're going to want automation in the future, but that choice
is up to you.


When packaging something that isn't on PyPI, you can let me know (on 
python-de...@lists.fedoraproject.org ) and I'll get the name blocked. I 
expect this to be rare enough to not need automation.


If the name *conflicts* with something on PyPI, then also let me know. 
Dealing with that issue might be more difficult, and it might involve 
you more. Still, this situation is confusing for users, so I think it's 
fair to want to solve it.

(And again, I'm not expecting existing packages to do this *right* now.)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

On 15. 06. 21 13:32, Alexander Bokovoy wrote:

On ti, 15 kesä 2021, Petr Viktorin wrote:

On 14. 06. 21 20:09, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Ben Cotton wrote:

[...]

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat 
namespace,
and PyPI is ''the'' place where open-source Python packages are 
generally
published, so users and tools assume a package called 
requests

is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes 
sense

for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python 
packaging

metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be 
registered

on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved 
on PyPI]

to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


Then I'd recommend that the project name in dist-info metadata would 
be, for example "samba-bindings".


Samba Python bindings have no dist-info metadata. Does it need to? The
bindings are consumed by few projects around Samba but none of them has
anything related on PyPI. What exists on PyPI is either an outdated
stuff without upstream sources or a reference to a similar outdated
stuff, none of which is using Samba Python bindings.


It doesn't need to have dist-info metadata, but I believe it would be 
better if it did, so that it's *possible* to encode "requires Samba" in 
Python packaging metadata.
(*Should* the bindings be consumed by more than a few projects around 
Samba? If yes, it dist-info would make it easier.)


There should also be a good way to add dist-info metadata to projects 
*like* Samba, even if Samba packagers decide they don't want it.


This matters in cases like a pip-installable package declaring a 
dependency on Samba bindings: it now can't use `samba` because that 
means something else on PyPI.


There are already unrelated projects with Samba name on PyPI. This
change in Fedora guidelines cannot affect those. It cannot also affect
actual use of Samba Python bindings. It looks like there is no real use
of PyPI for our use case either, so why the fuss for this particular
package?

I suspect at best we'd submit a request for exception, if pressed.


No pressure; this is exactly why the old guidelines aren't going anywhere.


I fully agree about not installing Samba itself via pip.

(Also, this is one of the cases that might take a decade -- I'm not 
putting pressure on you to implement this, but pointing out a 
direction for the future.)


Frankly, I do not see a real use case for what you might see in other
projects. We had a similar issue with FreeIPA and what we provide on
PyPI is a crippled subset of FreeIPA Python modules that mostly does not
use real FreeIPA features. To date, I am not sure it is really bringing
any benefit to us (upstream).

This does not mean the whole effort is useless, not at all. It means, in
my opinion, that proposed guidelines are failing to address real world
use cases present in Fedora. Invalidating these use cases is as bad as
dropping the guidelines' changes in one go. ;)


Don't put Samba or FreeIPA up on PyPI. But provide a name in dist-info, 
so packages on PyPI can depend on Samba/FreeIPA (which need to be 
installed separately). And reserve that name, so  no one else can put it 
on PyPI either.


Again, no pressure, just a direction for the future.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:

On 14.06.2021 15:32, Ben Cotton wrote:

Running upstream tests is mandatory.


What about tests that require network access?



Thanks for this and all the other concerns about mandatory tests!
I updated the proposal to mane them not mandatory, but require a smoke 
test if they're not there.
I also removed the mention of Fedora CI: it was meant as an escape hatch 
for tests that need network, but that's now unnecessary.


Diff: 
https://pagure.io/fork/pviktori/packaging-committee/c/ec9643873c989e0ff264128891665f6ad78f3e4a?branch=new-py-guidelines
History including other minor changes: 
https://pagure.io/fork/pviktori/packaging-committee/commits/new-py-guidelines


New wording:

If a test suite exists upstream,
it *SHOULD* be run in the `+%check+` section.
If that is not possible with reasonable effort,
at least a basic smoke test (such as importing the packaged module)
*MUST* be run in `+%check+`.

You *MAY* exclude specific failing tests.
You *MUST NOT* disable the entire testsuite
or ignore its result to solve a build failure.

As an exception,
you *MAY* disable tests with an appropriate `+%if+` conditional
(e.g. http://rpm.org/user_doc/conditional_builds.html[bcond])
when xref:index.adoc#bootstrapping[bootstrapping].

Most errors in Python happen at run-time,
so tests are extremely important to root out issues,
especially when mass rebuilds are required.

Common reasons for skipping tests in `+%check+` include requiring
network access,
dependencies not packaged in Fedora,
and/or specialized hardware or resources.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

I'll address the larger "mandatory tests" issue later; thanks for all 
your concerns!

This point deserves a reply on its own:

On 14. 06. 21 19:35, Benjamin Beasley wrote:
[...]

It’s also not clear to me why the Python guidelines should be so much stricter 
than the overall Fedora guidelines 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in 
this area.


Two reasons:
- Python is a dynamic language; most errors only happen at runtime. In 
C, just compiling and linking might work as a reasonable smoke test.
- I'd like the Python guidelines to be a bit "ahead" of the Fedora-wide 
ones. The proposal contains a few ideas that might be better 
Fedora-wide, but I'd rather introduce and test them in the ecosystem I know.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

On 14. 06. 21 20:09, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Ben Cotton wrote:

[...]

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat 
namespace,

and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called 
requests

is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python 
packaging

metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be 
registered

on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on 
PyPI]

to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


Then I'd recommend that the project name in dist-info metadata would be, 
for example "samba-bindings".
This matters in cases like a pip-installable package declaring a 
dependency on Samba bindings: it now can't use `samba` because that 
means something else on PyPI.


I fully agree about not installing Samba itself via pip.

(Also, this is one of the cases that might take a decade -- I'm not 
putting pressure on you to implement this, but pointing out a direction 
for the future.)



=== Burdening packagers ===

[https://lists.fedoraproject.org/archives/list/python-de...@lists.fedoraproject.org/message/IYBF5GX6HVQYPC5EJ6HVK7GJHAIKHVBK/ 


Neal]
does
“not necessarily disagree that PyPI and Fedora pythonXdist() names
should match, but [he disagrees] on burdening packagers with this.”

However, actually pushing a package to PyPI (and maintaining it there)
is not necessary.
The minimal-effort solution is to ''reserve'' the name on PyPI
so that a conflicting package can not be created there.

The author of this change is willing to do this work for all
Python packages whose upstream is not on PyPI;
packagers need to send a request to the Python SIG list.
If something happens to me, Red Hat's python-maint team is also ready
to do this, and in the event of catastrophic management change,
PyPI maintainers can be e-mailed directly.

This is all described in the new guidelines (see the ''PyPI parity'' 
section).


The new guidelines discourage the minimal-effort solution because it is
suboptimal ''for users'' (and for the wider Python ecosystem).


In case of Samba I would argue that going with anything but
minimal-effort is actually suboptimal to users.


I agree; Samba is a special case.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

Hi Neal,
We had this conversation in the past (and you can see it in the change). 
I don't think I can convince you, but I'll reiterate since it's new for 
devel@.


Unlike the "mandatory tests" issue elsewhere in this thread, using the 
PyPI namespace is the main point of the change. I can't take it out.



On 15. 06. 21 2:11, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:


On 6/14/21 1:53 PM, Neal Gompa wrote:

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.



I don't think we can reasonably separate the two.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

(I believe this is the "namespace" issue mentioned in the proposed change.)


You can. When it comes to distribution packages, we can maintain
consistency of the package names referenced within the distribution.
The point of this was to be able to use the names Python packages call
themselves for dependencies. PyPI is merely a distribution center for
such things. Fedora is another distribution center for such things.


You can only separate the namespaces if you sacrifice interoberability, 
i.e. allowing pip-installable packages to depend on Fedora packages.
It turns out you can already do that (venv --system-site-packages), and 
users expect that dist-info project names use the PyPI namespace. This 
is a current problem that we need to solve.


There are other (harder) solutions for the issue than reusing the PyPI 
namespace in Fedora, but why should we bother with them? There already 
is an ecosystem-wide namespace; why should we add a Fedora-specific one? 
(Or a Linux-distro-specific one, if we're interested in cross-distro 
collaboration.)


You seem to assume that registering/blocking a name on PyPI is hard.
I say it is not and I am willing to do it for packagers who need it. 
(And to mitigate the bus factor, I'll make it a responsibility of Red 
Hat's python-maint team.)



It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.


I agree, in fact, I think Fedora's problems here are a subset of the 
problems the private organizations have: if issues of proprietary corps 
are solved, we can use the solution as well. (However, it'll need more 
work than is necessary for Fedora/FOSS needs, so I don't want to drive 
the effort.)
BUT, if the issues are solved, it'll likely be through namespacing: we'd 
need to prefix our names with `fedora-` or `fedora:`. I still think it 
is better for Fedora to reuse the public PyPI namespace rather than 
start its own.



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

[Petr Viktorin]

On 14. 06. 21 17:39, Richard W.M. Jones wrote:

Questions, questions ...

These new guidelines seem to be fine for pure Python packages, but I'm
maintaining a couple of packages where Python bindings are built as
subpackages of existing C libraries:

https://src.fedoraproject.org/rpms/libnbd
https://src.fedoraproject.org/rpms/libguestfs

[Yes there are good reasons for this.  No we are not going to decouple
them and package the Python stuff separately.]

Running %pyproject_install etc is not really an option for us.
However I guess some of these macros could make sense
(%pyproject_save_files?  %pyproject_files?).  Also I don't think these
projects generate the extra *.egg-info files or how to create them.


Note that the macros are helpers/implementation details. They're not 
necessary; you can use other tools to be compatible with the guidelines.

Some of those tools are still to be written.
That's the main reason why adopting the guidelines might take a decade. 
We do intend to add helpers for more use cases, but it's not a priority 
right now.


Like the change says, there's no rush; the old guidelines will be 
retired "some time after the vast majority of Python packages follow the 
new guidelines and there are no known blockers for the remaining ones".
I hope it's clear that I do not expect individual packagers to solve 
this problem. (Though if y'all want to help, I won't object.)



=== Not removing older guidelines ===
There's no rush; it might well take a decade.


I guess we've got a while ...

While I've got your attention, one other package is interesting:

https://src.fedoraproject.org/rpms/nbdkit

This has Python 3 bindings but they work in completely the opposite
way to normal bindings, namely a Python interpreter is embedded in a
C program.  As far as I know none of the Python guidelines in Fedora
address this scenario directly, and indeed we don't currently use any
of the special %python* macros.


Thanks for the examples!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

I've proposed the new guidelines as a Fedora change: 
https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x


A discussion thread should pop up on de...@lists.fedoraproject.org soon.
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

License correction for pyparsing-doc: MIT -> MIT and GPLv2+ and GPLv3+ and BSD

[Petr Viktorin]

Hi!

The documentation package pyparsing-doc includes several examples that 
aren't under the project licence (MIT). The License field was corrected.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Draft of New Python Packaging Guidelines - 0.3

[Petr Viktorin]

Hello,
After (other parts of) $DAY_JOB delayed this for a while, a slightly 
updated draft is at https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg (and attached).


More importantly, here is the requested summary of changes from the old 
guidelines, roughly sorted by "importance": 
https://hackmd.io/LPfT6FKQSRyMcnqNU-zw2g (and attached).


There were two major objections to the earlier versions. The discussion 
died ou, so I'll repeat them. (Both of these should go in a change 
document rather than the guidelines).


- Why are the new macros better than the old ones?
They allow other build tools than just Setuptools, use upstream metadata 
for BuildRequires.
And they're being improved, with the vision that metadata should be 
taken from upstream rather than duplicated in the spec file. (If 
upstream is wrong, metadata should be patched just as you'd patch code.)



- Why do project names need to match PyPI?

This is because everywhere else (except in linux distros or tightly 
controlled corporate environments), a package name in a Python 
requirements list means a name on PyPI. Even at run time, when Python 
code checks if a package is installed, it assumes PyPI names.
Maintaining a separate namespace of project names, which could be mixed 
with the PyPI namespace, is *hard*. And building automation on top of 
such mixed namespaces is harder still.
Companies hit by the "dependency confusion" attack are to pin versions, 
use private proxies and/or block PyPI names.
There are ideas to add namespace support to PyPI, and Fedora should 
start using that when/if that happens. But until then, I really think 
the best we can do as Fedora packagers is in the draft:



If your package is not or cannot be published on PyPI, you can:
- Ask upstram to publish it
- If you wish: publish it to PyPI yourself and maintain it
- Ask Python SIG to block the name on PyPI for you
- Email PyPI admins to block the name for you, giving the project name and 
explaining the situation


Packages that aren't on PyPI are so rare, I'll be happy to handle the 
blocking myself (and pass the responsibility on within Red Hat 
python-maint when I retire from Fedora).


Also, no other ecosystem in Fedora does this. Python would be the first. 
If the other ecosystems also use a flat package namespace, and want to 
automate packaging, they'll probably run into the same problem.


Ultimately, syncing Fedora with the wider Python ecosystem is the main 
idea behind the draft. I'd be glad to hear how it can be done better, 
but to me, the new guidelines wouldn't make sense without this part.




On 4/30/20 3:41 PM, Petr Viktorin wrote:

Hello!
Below is a draft of new Packaging Guidelines! It's full of unfinished 
areas (marked with XXX), but it's ready for scrutiny.
A possibly updated version is on 
https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view


Generally, for rules marked **SHOULD** we know of cases where they 
should be broken; for things marked **MUST** we don't.


We have tried to only take the Good Existing Things™ (macros, practices) 
and revise the rest without thinking about the past much. Some used 
technology is unfortunately not compatible with current EPELs, but we 
have considered Fedora 31+. Using the current Python guidelines will 
still be an option for people who target EPEL 6/7/8.


The main controversial idea (of many) is synchronizing Fedora's names 
(a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python 
Package Index (PyPI, pypi.org), which has by now become the de-facto 
canonical namespace for freely redistributable Python packages.
We believe that this will help both Fedora and the greater Python 
ecosystem, but there will definitely be some growing pains.


Most of Fedora Python packages already are on PyPI, but more than 250 
are missing. There is software developed within Fedora (e.g. pagure, 
fedpkg, rust2rpm); projects that aren't primarily Python packages (e.g. 
perf, ceph, xen, setroubleshoot, iotop); and others.
A full list is attached. The names have been temporarily blocked on PyPI 
to keep trolls from taking them while this is discussed.
Over at the Python Discourse we are discussing how to properly handle 
these; once that discussion is settled they should be unblocked: 
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a-k-a-pypi-namesquatting-e-g-for-fedora-packages/4045 



Another general idea is that package metadata should be kept upstream; 
the spec file should duplicate as little of it as possible. Any 
adjustments should be done with upstreamable patches.


The draft lives on hackmd.io, which we found easy to collaborate with. 
If you have an account there, we can add you. If you'd like to 
collaborate some other way, let us know.


Petr and Miro

---

Current draft for inline comments:


 > [IMPORTANT]
 > This is a DRAFT; it is not in effect yet.

# Python Packaging Guidelines (draft)

 > [IMPORTANT]
 > This i

Re: Follow-up - Please BuildRequire python3-setuptools explicitly

[Petr Viktorin]

On 2020-10-05 13:53, Richard Shaw wrote:
I updated fail2ban and gmsh and committed to master. I assume that is 
sufficient?


Thank you!
As I understand it, it is sufficient.  But since repo queries look at 
"live" packages, you might get some spurious notifications about this in 
the future.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines - 0.2

[Petr Viktorin]

On 2020-08-12 18:19, Neal Gompa wrote:

On Wed, Aug 12, 2020 at 12:02 PM Petr Viktorin  wrote:

On 2020-08-12 17:22, Neal Gompa wrote:

On Wed, Aug 12, 2020 at 11:19 AM Petr Viktorin  wrote:

On 2020-08-12 16:53, Neal Gompa wrote:

On Wed, Aug 12, 2020 at 10:23 AM Petr Viktorin  wrote:

@Conan-Kudo:

Is there something you've done here to make the new macros unequivocally 
better? Do you have automated flavor builds, for example? Right now there is no 
effective difference.



The new macros allow other build tools than just setuptools.
They also use upstream metadata for BuildRequires & file lists.

Of course, where possible the changes were backported to the existing
macros. I don't necessarily see the macros as the main thing that
changes. But when you look at a specfile, you can usually tell what
conventions it uses by what macros you see.



Those are all nice things, for sure. I think it'd be important to
spell that out somewhere.


Do you mean providing a summary of the changes between the existing
guidelines and these?



Yes. Also indicating the *why* for pyproject stuff is useful within
the context of why the old macros are deprecated.


Is that necessary for the beta release of the packaging guidelines?
(i.e. when they would be in effect, but optional?)

You're asking for quite a lot of work around describing a document that
might still change.




I am only asking for what makes sense if the document as it stands
were to be finalized.


OK, you're right. I added a note to the checklist.


## PyPI parity

Every Python package in Fedora **SHOULD** also be available
on [the Python Package Index](https://pypi.org) (PyPI).

The command `pip install PROJECTNAME` **MUST** install the same package
(possibly in a different version),
install nothing,
or fail with a reasonable error message.

If this is not the case, the packager **MUST** contact upstream about this.
The goal is to get the project name registered or blocked on PyPI,
or to otherwise ensure the rule is followed.

To block a project name on PyPI, email
[ad...@pypi.org](mailto:ad...@pypi.org),
giving the project name and explaining the situation
(for example: the package cannot currently be installed via `pip`).
You can ask questions and discuss the process at the [Python
Discourse](https://discuss.python.org/t/block-names/4045).

> NOTE: Project names that were in Fedora but not on PyPI
> when these guidelines were proposed
> are *blocked* from being uploaded to PyPI.
> This prevents potential trolls from taking them,
> but it also blocks legitimate owners.
> If your package is affected, contact the Python SIG
> or [file a PyPA
issue](https://github.com/pypa/pypi-support/issues/new?labels=PEP+541=pep541-request.md=PEP+541+Request%3A+PROJECT_NAME)

> and mention `@encukou`.

This is necessary to protect users,
avoid confusion,
and enable automation as Fedora and upstream ecosystems grow more
integrated.

As always, [specific exceptions can be granted by the Packaging
Committee](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_general_exception_policy).



@Conan-Kudo:

This is not reasonable to ask of packagers to do. Such a check is difficult to 
do, and there is no particularly compelling reason for making everything 
written in Python using Python build system available in PyPI. Unless you plan 
to provide an automated system to inform PyPI of these things, this is not only 
unreasonable, it is unenforceable.


A lot of stuff in the guidelines is unenforceable, so let's focus on the
"unreasonable" part.

There is no reason to have everything available on PyPI, but I do
believe it's reasonable to *reserve the name* in such cases.

Here's a reason why I want this:

The issue here is that Python tools have access to project names. For
example, I can get the version of Requests using:

>>> from importlib.metadata import version
>>> version('requests')
'2.22.0'

Things like this are only useful if we use a common namespace. Upstream,
that namespace *is* PyPI; there's little we can do about that.
If project names mean something else in Fedora than in the wider
ecosystem, we'll get user confusion at best.

(If you use a private package index, like they do at CERN or some
closed-source shops, there can be some collisions -- but in that case
there's a limited number of software authors and users, and a lot more
control over the package set. Conflicts in global repositories of
free/open-source software are much harder to manage.)



Lately, I think about another way to handle this: packages that aren't
on PyPI could not ship the .dist-info at all, and so, they would not
have things like `python3dist(...)` provides. They'd only be usable with
Fedora tooling, not in the wider Python ecosystem.
It's a major case to think out and test, but maybe it would be worth it?



I think omitting the Provides is unfair.


Why?
In my view, `python3dist(..

Re: Draft of New Python Packaging Guidelines - 0.2

[Petr Viktorin]

On 2020-08-12 17:22, Neal Gompa wrote:

On Wed, Aug 12, 2020 at 11:19 AM Petr Viktorin  wrote:


On 2020-08-12 16:53, Neal Gompa wrote:

On Wed, Aug 12, 2020 at 10:23 AM Petr Viktorin  wrote:


I'll move some discussion here, where it doesn't become part of the
document:



On 2020-08-11 14:19, Petr Viktorin wrote:

These Guidelines represent a major rewrite and paradigm shift, and not
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages
**MAY** use them instead of this document:
* 201x-era [Python packaging
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/)
(for packages that use e.g. `%py3_install` or `setup.py install`)
* [Python 2
appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_python_2_packages)
(for e.g. `%py2_install`) (Python 2 packages require a FESCo exception.)


@Conan-Kudo:

Is there something you've done here to make the new macros unequivocally 
better? Do you have automated flavor builds, for example? Right now there is no 
effective difference.



The new macros allow other build tools than just setuptools.
They also use upstream metadata for BuildRequires & file lists.

Of course, where possible the changes were backported to the existing
macros. I don't necessarily see the macros as the main thing that
changes. But when you look at a specfile, you can usually tell what
conventions it uses by what macros you see.



Those are all nice things, for sure. I think it'd be important to
spell that out somewhere.


Do you mean providing a summary of the changes between the existing
guidelines and these?



Yes. Also indicating the *why* for pyproject stuff is useful within
the context of why the old macros are deprecated.


Is that necessary for the beta release of the packaging guidelines? 
(i.e. when they would be in effect, but optional?)


You're asking for quite a lot of work around describing a document that 
might still change.




## PyPI parity

Every Python package in Fedora **SHOULD** also be available
on [the Python Package Index](https://pypi.org) (PyPI).

The command `pip install PROJECTNAME` **MUST** install the same package
(possibly in a different version),
install nothing,
or fail with a reasonable error message.

If this is not the case, the packager **MUST** contact upstream about this.
The goal is to get the project name registered or blocked on PyPI,
or to otherwise ensure the rule is followed.

To block a project name on PyPI, email
[ad...@pypi.org](mailto:ad...@pypi.org),
giving the project name and explaining the situation
(for example: the package cannot currently be installed via `pip`).
You can ask questions and discuss the process at the [Python
Discourse](https://discuss.python.org/t/block-names/4045).

   > NOTE: Project names that were in Fedora but not on PyPI
   > when these guidelines were proposed
   > are *blocked* from being uploaded to PyPI.
   > This prevents potential trolls from taking them,
   > but it also blocks legitimate owners.
   > If your package is affected, contact the Python SIG
   > or [file a PyPA
issue](https://github.com/pypa/pypi-support/issues/new?labels=PEP+541=pep541-request.md=PEP+541+Request%3A+PROJECT_NAME)

   > and mention `@encukou`.

This is necessary to protect users,
avoid confusion,
and enable automation as Fedora and upstream ecosystems grow more
integrated.

As always, [specific exceptions can be granted by the Packaging
Committee](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_general_exception_policy).



@Conan-Kudo:

This is not reasonable to ask of packagers to do. Such a check is difficult to 
do, and there is no particularly compelling reason for making everything 
written in Python using Python build system available in PyPI. Unless you plan 
to provide an automated system to inform PyPI of these things, this is not only 
unreasonable, it is unenforceable.


A lot of stuff in the guidelines is unenforceable, so let's focus on the
"unreasonable" part.

There is no reason to have everything available on PyPI, but I do
believe it's reasonable to *reserve the name* in such cases.

Here's a reason why I want this:

The issue here is that Python tools have access to project names. For
example, I can get the version of Requests using:

   >>> from importlib.metadata import version
   >>> version('requests')
'2.22.0'

Things like this are only useful if we use a common namespace. Upstream,
that namespace *is* PyPI; there's little we can do about that.
If project names mean something else in Fedora than in the wider
ecosystem, we'll get user confusion at best.

(If you use a private package index, like they do at CERN or some
closed-source shops, there can be some collisions -- but in that case
there's a limited number of software authors and users, and a lot more
control over the package set. Conflicts in global repositorie

Re: Draft of New Python Packaging Guidelines - 0.2

[Petr Viktorin]

On 2020-08-12 16:53, Neal Gompa wrote:

On Wed, Aug 12, 2020 at 10:23 AM Petr Viktorin  wrote:


I'll move some discussion here, where it doesn't become part of the
document:



On 2020-08-11 14:19, Petr Viktorin wrote:

These Guidelines represent a major rewrite and paradigm shift, and not
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages
**MAY** use them instead of this document:
* 201x-era [Python packaging
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/)
(for packages that use e.g. `%py3_install` or `setup.py install`)
* [Python 2
appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_python_2_packages)
(for e.g. `%py2_install`) (Python 2 packages require a FESCo exception.)


@Conan-Kudo:

Is there something you've done here to make the new macros unequivocally 
better? Do you have automated flavor builds, for example? Right now there is no 
effective difference.



The new macros allow other build tools than just setuptools.
They also use upstream metadata for BuildRequires & file lists.

Of course, where possible the changes were backported to the existing
macros. I don't necessarily see the macros as the main thing that
changes. But when you look at a specfile, you can usually tell what
conventions it uses by what macros you see.



Those are all nice things, for sure. I think it'd be important to
spell that out somewhere.


Do you mean providing a summary of the changes between the existing 
guidelines and these?



## PyPI parity

Every Python package in Fedora **SHOULD** also be available
on [the Python Package Index](https://pypi.org) (PyPI).

The command `pip install PROJECTNAME` **MUST** install the same package
(possibly in a different version),
install nothing,
or fail with a reasonable error message.

If this is not the case, the packager **MUST** contact upstream about this.
The goal is to get the project name registered or blocked on PyPI,
or to otherwise ensure the rule is followed.

To block a project name on PyPI, email
[ad...@pypi.org](mailto:ad...@pypi.org),
giving the project name and explaining the situation
(for example: the package cannot currently be installed via `pip`).
You can ask questions and discuss the process at the [Python
Discourse](https://discuss.python.org/t/block-names/4045).

  > NOTE: Project names that were in Fedora but not on PyPI
  > when these guidelines were proposed
  > are *blocked* from being uploaded to PyPI.
  > This prevents potential trolls from taking them,
  > but it also blocks legitimate owners.
  > If your package is affected, contact the Python SIG
  > or [file a PyPA
issue](https://github.com/pypa/pypi-support/issues/new?labels=PEP+541=pep541-request.md=PEP+541+Request%3A+PROJECT_NAME)

  > and mention `@encukou`.

This is necessary to protect users,
avoid confusion,
and enable automation as Fedora and upstream ecosystems grow more
integrated.

As always, [specific exceptions can be granted by the Packaging
Committee](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_general_exception_policy).



@Conan-Kudo:

This is not reasonable to ask of packagers to do. Such a check is difficult to 
do, and there is no particularly compelling reason for making everything 
written in Python using Python build system available in PyPI. Unless you plan 
to provide an automated system to inform PyPI of these things, this is not only 
unreasonable, it is unenforceable.


A lot of stuff in the guidelines is unenforceable, so let's focus on the
"unreasonable" part.

There is no reason to have everything available on PyPI, but I do
believe it's reasonable to *reserve the name* in such cases.

Here's a reason why I want this:

The issue here is that Python tools have access to project names. For
example, I can get the version of Requests using:

  >>> from importlib.metadata import version
  >>> version('requests')
'2.22.0'

Things like this are only useful if we use a common namespace. Upstream,
that namespace *is* PyPI; there's little we can do about that.
If project names mean something else in Fedora than in the wider
ecosystem, we'll get user confusion at best.

(If you use a private package index, like they do at CERN or some
closed-source shops, there can be some collisions -- but in that case
there's a limited number of software authors and users, and a lot more
control over the package set. Conflicts in global repositories of
free/open-source software are much harder to manage.)



Lately, I think about another way to handle this: packages that aren't
on PyPI could not ship the .dist-info at all, and so, they would not
have things like `python3dist(...)` provides. They'd only be usable with
Fedora tooling, not in the wider Python ecosystem.
It's a major case to think out and test, but maybe it would be worth it?



I think omitting the Provides is unfair.


Wh

Re: Draft of New Python Packaging Guidelines - 0.2

[Petr Viktorin]

I'll move some discussion here, where it doesn't become part of the 
document:




On 2020-08-11 14:19, Petr Viktorin wrote:
These Guidelines represent a major rewrite and paradigm shift, and not 
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages 
**MAY** use them instead of this document:
* 201x-era [Python packaging 
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/) 
(for packages that use e.g. `%py3_install` or `setup.py install`)
* [Python 2 
appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_python_2_packages) 
(for e.g. `%py2_install`) (Python 2 packages require a FESCo exception.)



@Conan-Kudo:

Is there something you've done here to make the new macros unequivocally 
better? Do you have automated flavor builds, for example? Right now there is no 
effective difference.



The new macros allow other build tools than just setuptools.
They also use upstream metadata for BuildRequires & file lists.

Of course, where possible the changes were backported to the existing 
macros. I don't necessarily see the macros as the main thing that 
changes. But when you look at a specfile, you can usually tell what 
conventions it uses by what macros you see.





## PyPI parity

Every Python package in Fedora **SHOULD** also be available
on [the Python Package Index](https://pypi.org) (PyPI).

The command `pip install PROJECTNAME` **MUST** install the same package
(possibly in a different version),
install nothing,
or fail with a reasonable error message.

If this is not the case, the packager **MUST** contact upstream about this.
The goal is to get the project name registered or blocked on PyPI,
or to otherwise ensure the rule is followed.

To block a project name on PyPI, email 
[ad...@pypi.org](mailto:ad...@pypi.org),

giving the project name and explaining the situation
(for example: the package cannot currently be installed via `pip`).
You can ask questions and discuss the process at the [Python 
Discourse](https://discuss.python.org/t/block-names/4045).


 > NOTE: Project names that were in Fedora but not on PyPI
 > when these guidelines were proposed
 > are *blocked* from being uploaded to PyPI.
 > This prevents potential trolls from taking them,
 > but it also blocks legitimate owners.
 > If your package is affected, contact the Python SIG
 > or [file a PyPA 
issue](https://github.com/pypa/pypi-support/issues/new?labels=PEP+541=pep541-request.md=PEP+541+Request%3A+PROJECT_NAME) 


 > and mention `@encukou`.

This is necessary to protect users,
avoid confusion,
and enable automation as Fedora and upstream ecosystems grow more 
integrated.


As always, [specific exceptions can be granted by the Packaging 
Committee](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_general_exception_policy). 



@Conan-Kudo:

This is not reasonable to ask of packagers to do. Such a check is difficult to 
do, and there is no particularly compelling reason for making everything 
written in Python using Python build system available in PyPI. Unless you plan 
to provide an automated system to inform PyPI of these things, this is not only 
unreasonable, it is unenforceable.


A lot of stuff in the guidelines is unenforceable, so let's focus on the 
"unreasonable" part.


There is no reason to have everything available on PyPI, but I do 
believe it's reasonable to *reserve the name* in such cases.


Here's a reason why I want this:

The issue here is that Python tools have access to project names. For 
example, I can get the version of Requests using:


>>> from importlib.metadata import version
>>> version('requests')
'2.22.0'

Things like this are only useful if we use a common namespace. Upstream, 
that namespace *is* PyPI; there's little we can do about that.
If project names mean something else in Fedora than in the wider 
ecosystem, we'll get user confusion at best.


(If you use a private package index, like they do at CERN or some 
closed-source shops, there can be some collisions -- but in that case 
there's a limited number of software authors and users, and a lot more 
control over the package set. Conflicts in global repositories of 
free/open-source software are much harder to manage.)




Lately, I think about another way to handle this: packages that aren't 
on PyPI could not ship the .dist-info at all, and so, they would not 
have things like `python3dist(...)` provides. They'd only be usable with 
Fedora tooling, not in the wider Python ecosystem.

It's a major case to think out and test, but maybe it would be worth it?
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedo

Draft of New Python Packaging Guidelines - 0.2

[Petr Viktorin]

Hello!
Here is another draft of new Python Packaging guidelines. It is now very 
close to complete – see the checklists at the beginning.


The text is pasted for you to quote.
An possibly updated version is still at 
https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg


Thanks to Miro and the rest of the team for the work on this and on the 
related macro improvements — some of which made it to the current, 
"non-pyproject" macros as well; see 
https://pagure.io/packaging-committee/pull-request/1011



On 2020-04-30 15:41, Petr Viktorin wrote:
...
Generally, for rules marked **SHOULD** we know of cases where they 
should be broken; for things marked **MUST** we don't.


We have tried to only take the Good Existing Things™ (macros, practices) 
and revise the rest without thinking about the past much. Some used 
technology is unfortunately not compatible with current EPELs, but we 
have considered Fedora 31+. Using the current Python guidelines will 
still be an option for people who target EPEL 6/7/8.


The main controversial idea (of many) is synchronizing Fedora's names 
(a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python 
Package Index (PyPI, pypi.org), which has by now become the de-facto 
canonical namespace for freely redistributable Python packages.
We believe that this will help both Fedora and the greater Python 
ecosystem, but there will definitely be some growing pains.

...> Another general idea is that package metadata should be kept upstream;
the spec file should duplicate as little of it as possible. Any 
adjustments should be done with upstreamable patches.


The draft lives on hackmd.io, which we found easy to collaborate with. 
If you have an account there, we can add you. If you'd like to 
collaborate some other way, let us know.


Petr and Miro



---


> [IMPORTANT]
> This is a DRAFT; it is not in effect yet.
>
> Checklist for public Beta:
> * Update the [Python section on the general Naming 
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules) 
– add link to [Name limitations](#Name-limitations), [Library 
naming](#Library-naming), [Application naming](#Application-naming); and 
the old guidelines
> * `%pyproject_buildrequires` should take multiple comma-separated 
extras for `-x`

> * Pello
>   * Triage (and possibly fix) [Pello 
issues](https://github.com/hrnciar/pello/issues)

>   * Update the example spec
> * Convert to to AsciiDoc and sembr
>   * Update internal links (esp. link directly to individual macros)
>
> Checklist for going out of Beta:
> * Add a note to the [Manual bytecompilation 
Appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#manual-bytecompilation) 
that it's only for the old guidelines. (The [Manual byte 
compilation](#Manual-byte-compilation) section and macro docs should 
cover everything needed for Python 3.)
>   * Keep "Byte compilation reproducibility" 
(https://pagure.io/packaging-committee/pull-request/1013) as is; it 
applies to both old and new guidelines.
> * Make `python3-devel` BuildRequire `pyproject-rpm-macros`; remove 
text about requiring it manually

> * Write an automated check for [PyPI parity](#PyPI-parity).
> * Move old guidelines to a sub-page
> * Move these guidelines to the main page
> * Update links in both
>  * Rewrite the intro


# Python Packaging Guidelines (draft)

> [IMPORTANT]
> This is a *beta* version of the Python Packaging Guidelines and the 
associated RPM macros.
> Packagers that opt in to following this version **MUST** be prepared 
to change their packages frequently when the guidelines or macros are 
updated.
> These packagers **SHOULD** join [Python SIG mailing 
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/) 
and monitor/start conversations there.



These Guidelines represent a major rewrite and paradigm shift, and not 
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages 
**MAY** use them instead of this document:
* 201x-era [Python packaging 
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/) 
(for packages that use e.g. `%py3_install` or `setup.py install`)
* [Python 2 
appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_python_2_packages) 
(for e.g. `%py2_install`) (Python 2 packages require a FESCo exception.)


> [NOTE]
> These guidelines only support Fedora 31+. For older releases (such as 
in EPEL 8), consult the [older 
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/). 



The two "[Distro-wide guidelines](#Distro-wide-guidelines)" below apply 
to all software in Fedora that uses Python at build- or run-time.


The rest of the Guidelines apply to packages that ship code that can be 
importe

Re: Suppress "running pip install with root privileges" warning in RPM macros?

[Petr Viktorin]

On 2020-07-07 19:54, Miro Hrončok wrote:

1) Add a custom --no-warn-root-privileges option
2) Hide the warning when $RPM_BUILD_ROOT is set.
3) Introduce an environment variable (e.g. PIP_NOWARN_ROOT)
4) Introduce our warning upstream, but make it opt-in only.
5) Hide the warning when --root is set.

What do you think?


I like option 5 as well.
It's just a warning, not an error, because there are cases where you 
"know what you're doing", such as containers/VMs, where "sudo pip" may 
be appropriate. All uses of --root that come to my mind are such cases.


Users who follow tutorials without "knowing what they're doing" are very 
unlikely to use --root.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-11 13:22, Tomas Orsava wrote:

On 6/9/20 12:15 PM, Petr Viktorin wrote:

On 2020-06-08 12:15, Tomas Orsava wrote:

On 6/8/20 11:58 AM, Petr Viktorin wrote:

[...] https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg


Looks good, thank you.

All in all, really nice document. Let me know if you could use more 
help with it!


Tomas


Well, any of the XXX could use help :)
Except maybe the links; those can be dded when we convert from Markdown.



Is there some way to do reviews? If someone just replaces XXX with text, 
it's hard to notice.


There is a "versions" view on HackMD, which has diffs.
I don't think it's yet time to move to Git and do proper reviewed pull 
requests.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-08 12:15, Tomas Orsava wrote:

On 6/8/20 11:58 AM, Petr Viktorin wrote:



On 2020-06-05 16:12, Tomas Orsava wrote:

On 6/5/20 2:26 PM, Petr Viktorin wrote:



On 2020-06-05 11:51, Tomas Orsava wrote:

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the 
document.


Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:



### Dist-info metadata

Each Python package **MUST** include *Package Distribution 
Metadata* conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)). 



This applies to libraries (e.g. `python-requests`) as well as 
tools (e.g. `ansible`).



> XXX what with splitting into subpackages? 1) dist-info always 
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, 
fedpkg)

>
> e.g.
> When software is split into several subpackages, it is OK to 
only ship metadata in one built RPM.



dist-info usually corresponds to an importable module, so let's 
say that it SHOULD be in the same subpackage as the importable 
module?


But if you split that module, which submodule does the dist-info 
go to?

I'd leave it to the packager; all these cases are different.


That's why I suggested the SHOULD, because there will be 
exceptions. But I think these guidelines might be read by people 
who will not be actively aware of the relationship between 
dist-info and a Python importable module, so I would add guidance 
that these should be together if possible.


Please suggest the wording you'd like to see.
The one you have has the problem that there can be more importable 
modules in one project. It doesn't give any guidance for what to do 
if you split the project.


Of course, if you put importable module(s) in one subpackage and 
documentation in another, the dist-info should be with the module(s).



I'm trying to read this guide through the eyes of someone starting 
out with Fedora and/or Python, so I'd rather not assume people know 
these details.


How about:

    When software is split into several subpackages, it is OK to only
    ship metadata in one built RPM. If the project contains an
    importable module(s), the metadata SHOULD be included in the same
    subpackage as the (main) importable module.

(First sentence already was in the text, I included it for context.)


I added it. I put the **SHOULD** sentence near the top of the section 
to align to the organization of the document: rules first, 
explanations/examples under them.


Looks good, thank you.

All in all, really nice document. Let me know if you could use more help 
with it!


Tomas


Well, any of the XXX could use help :)
Except maybe the links; those can be dded when we convert from Markdown.
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-05 16:12, Tomas Orsava wrote:

On 6/5/20 2:26 PM, Petr Viktorin wrote:



On 2020-06-05 11:51, Tomas Orsava wrote:

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the 
document.


Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:



### Dist-info metadata

Each Python package **MUST** include *Package Distribution 
Metadata* conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)). 



This applies to libraries (e.g. `python-requests`) as well as 
tools (e.g. `ansible`).



> XXX what with splitting into subpackages? 1) dist-info always 
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, 
fedpkg)

>
> e.g.
> When software is split into several subpackages, it is OK to 
only ship metadata in one built RPM.



dist-info usually corresponds to an importable module, so let's say 
that it SHOULD be in the same subpackage as the importable module?


But if you split that module, which submodule does the dist-info go to?
I'd leave it to the packager; all these cases are different.


That's why I suggested the SHOULD, because there will be exceptions. 
But I think these guidelines might be read by people who will not be 
actively aware of the relationship between dist-info and a Python 
importable module, so I would add guidance that these should be 
together if possible.


Please suggest the wording you'd like to see.
The one you have has the problem that there can be more importable 
modules in one project. It doesn't give any guidance for what to do if 
you split the project.


Of course, if you put importable module(s) in one subpackage and 
documentation in another, the dist-info should be with the module(s).



I'm trying to read this guide through the eyes of someone starting out 
with Fedora and/or Python, so I'd rather not assume people know these 
details.


How about:

When software is split into several subpackages, it is OK to only
ship metadata in one built RPM. If the project contains an
importable module(s), the metadata SHOULD be included in the same
subpackage as the (main) importable module.

(First sentence already was in the text, I included it for context.)


I added it. I put the **SHOULD** sentence near the top of the section to 
align to the organization of the document: rules first, 
explanations/examples under them.





## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check` section 
and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the result 
to solve a build failure.


As an exception, you **MAY** disable tests with an appropriate 
`%if` conditional (e.g. bcond) when 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping). 





I would like to see either that the bcond SHOULD be named `tests` 
(or possibly `check`), or if that's too strong, at least recommend 
these two as common bcond names.


I agree, but it should be a Fedora-wide guideline.



True. Does anyone know if this is already in progress somewhere? I 
remember it being talked about. Otherwise I guess the best way will 
be for me to open a new thread about this.


Please do.


Here we go:
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/443LZIX4XLZL6NMF3M7HAGHKPNA4TRYN/
 >>
In the meantime, I would at least list these as common bcond names, 
as Fedora-wide guidelines might take a while.


The Python guidelines will also take a while, and I'd like to avoid 
giving different guidelines than what we come up with for Fedora.


Makes sense. I've added an "XXX" note to the text so we don't forget to 
address this later.




Thanks!
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-05 15:37, Tomas Orsava wrote:

On 6/5/20 2:22 PM, Petr Viktorin wrote:



On 2020-06-05 13:58, Tomas Orsava wrote:

On 6/5/20 1:43 PM, Miro Hrončok wrote:

On 05. 06. 20 11:51, Tomas Orsava wrote:

[...]



I see what you mean.
On the other hand, that's a pretty horrible error message format 
(posting in it's entirety for others to consider).


Is there no better way to achieve this? For a few packages it's ok, 
but I would be weary of introducing this to too many packages.


In the proposal there's talk of blocking the name on PyPI. Is this 
the way the blocking will be achieved?


You can talk to PyPI admins to block packages.

But all in all, I think what "fedora", "ldap" or "microsoft are doing 
is the best option right now.
I started some discussion upstream, if you want to read it: 
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a-k-a-pypi-namesquatting-e-g-for-fedora-packages/4045/2 


(But let's discuss Fedora issues here first.)


Ah, that's sad. Thanks for raising the topic upstream, hopefully it'll 
improve down the road.


As fer Fedora: Do I understand it correctly that:
- the names of Python packages in Fedora have been now blocked on PyPI 
using the admin intervention, and
- we want to advise people to get those names on PyPI and either publish 
the projects there or do this fedora/ldap-sort of namesquatting?


Yes.

I agree it's the least-worst option now. However, it will need a 
detailed instructions.


Yes, but I want to discuss it first before writing the instructions :)
I'm on it next week again; it just ot stalled by work around the 3.9 
side tag builds/merging.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-05 11:51, Tomas Orsava wrote:

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the 
document.


Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:



### Dist-info metadata

Each Python package **MUST** include *Package Distribution Metadata* 
conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)). 



This applies to libraries (e.g. `python-requests`) as well as tools 
(e.g. `ansible`).



> XXX what with splitting into subpackages? 1) dist-info always 
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, fedpkg)
>
> e.g.
> When software is split into several subpackages, it is OK to only 
ship metadata in one built RPM.



dist-info usually corresponds to an importable module, so let's say 
that it SHOULD be in the same subpackage as the importable module?


But if you split that module, which submodule does the dist-info go to?
I'd leave it to the packager; all these cases are different.


That's why I suggested the SHOULD, because there will be exceptions. But 
I think these guidelines might be read by people who will not be 
actively aware of the relationship between dist-info and a Python 
importable module, so I would add guidance that these should be together 
if possible.


Please suggest the wording you'd like to see.
The one you have has the problem that there can be more importable 
modules in one project. It doesn't give any guidance for what to do if 
you split the project.


Of course, if you put importable module(s) in one subpackage and 
documentation in another, the dist-info should be with the module(s).





If this is not the case, the packager **MUST** contact upstream 
about this. The goal is to get the project name registered or 
blocked on PyPI, or to otherwise ensure the rule is followed.


> XXX Note that project names that were in Fedora but not on PyPI 
when these guidelines were proposed are *blocked* as we discuss how 
they should be handled.
> This prevents potential trolls, but also legitimate owners, from 
taking them.


This is necessary to protect users, avoid confusion, and enable 
automation as Fedora and upstream ecosystems grow more integrated.


As always, specific exceptions can be granted by FPC (XXX link 
exceptions rules).


> XXX Write an automated check for this.





## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check` section 
and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the result 
to solve a build failure.


As an exception, you **MAY** disable tests with an appropriate `%if` 
conditional (e.g. bcond) when 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping). 




I would like to see either that the bcond SHOULD be named `tests` (or 
possibly `check`), or if that's too strong, at least recommend these 
two as common bcond names.


I agree, but it should be a Fedora-wide guideline.



True. Does anyone know if this is already in progress somewhere? I 
remember it being talked about. Otherwise I guess the best way will be 
for me to open a new thread about this.


Please do.

In the meantime, I would at least list these as common bcond names, as 
Fedora-wide guidelines might take a while.


The Python guidelines will also take a while, and I'd like to avoid 
giving different guidelines than what we come up with for Fedora.




___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-05 13:58, Tomas Orsava wrote:

On 6/5/20 1:43 PM, Miro Hrončok wrote:

On 05. 06. 20 11:51, Tomas Orsava wrote:

[...]



I see what you mean.
On the other hand, that's a pretty horrible error message format 
(posting in it's entirety for others to consider).


Is there no better way to achieve this? For a few packages it's ok, but 
I would be weary of introducing this to too many packages.


In the proposal there's talk of blocking the name on PyPI. Is this the 
way the blocking will be achieved?


You can talk to PyPI admins to block packages.

But all in all, I think what "fedora", "ldap" or "microsoft are doing is 
the best option right now.
I started some discussion upstream, if you want to read it: 
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a-k-a-pypi-namesquatting-e-g-for-fedora-packages/4045/2

(But let's discuss Fedora issues here first.)
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the document.

Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:



### Dist-info metadata

Each Python package **MUST** include *Package Distribution Metadata* 
conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)). 



This applies to libraries (e.g. `python-requests`) as well as tools 
(e.g. `ansible`).



> XXX what with splitting into subpackages? 1) dist-info always 
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, fedpkg)
>
> e.g.
> When software is split into several subpackages, it is OK to only 
ship metadata in one built RPM.



dist-info usually corresponds to an importable module, so let's say that 
it SHOULD be in the same subpackage as the importable module?


But if you split that module, which submodule does the dist-info go to?
I'd leave it to the packager; all these cases are different.


The metadata takes the form of a `dist-info` directory installed in 
`%{python3_sitelib}` or `%{python3_sitearch}`, and contains 
information that tools like 
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) 
use to introspect installed libraries.


> XXX example %files with manual dist-info entry

Note that some older tools instead put metadata in an `egg-info` 
directory, or even a single file.

This won't happen if you use the `%pyproject_wheel` macro.
If your package uses a build system that generates an `egg-info` 
directory or file, please contact Python SIG.


> XXX We need a better solution before we go out of beta.

As an exception, the Python standard library **MAY** ship without this 
metadata.





## PyPI parity

Every Python package in Fedora **SHOULD** also be available on [the 
Python Package Index](https://pypi.org) (PyPI).


The command `pip install PROJECTNAME` **MUST** install the same 
package (possibly in a different version), install nothing, or fail 
with a reasonable error message.



What should I imagine under "reasonable error message"?


One that explains the situation. Not a segfault.
I think it's fine to leave this to the packager.


If this is not the case, the packager **MUST** contact upstream about 
this. The goal is to get the project name registered or blocked on 
PyPI, or to otherwise ensure the rule is followed.


> XXX Note that project names that were in Fedora but not on PyPI when 
these guidelines were proposed are *blocked* as we discuss how they 
should be handled.
> This prevents potential trolls, but also legitimate owners, from 
taking them.


This is necessary to protect users, avoid confusion, and enable 
automation as Fedora and upstream ecosystems grow more integrated.


As always, specific exceptions can be granted by FPC (XXX link 
exceptions rules).


> XXX Write an automated check for this.





## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check` section 
and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the result to 
solve a build failure.


As an exception, you **MAY** disable tests with an appropriate `%if` 
conditional (e.g. bcond) when 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping). 


I would like to see either that the bcond SHOULD be named `tests` (or 
possibly `check`), or if that's too strong, at least recommend these two 
as common bcond names.


I agree, but it should be a Fedora-wide guideline.


A popular testing tool, and one which is well integrated in Fedora, is 
`tox`. Upstream, it is commonly used to test against multiple Python 
versions. In a Fedora package, BuildRequire test dependencies (see 
*Test dependencies* below) and run `tox` with:


```
%tox
```




### Test dependencies

One part of the Python packaging ecosystem that is still not 
standardized is specifying test dependencies (and development 
dependencies in general).


The best practice to specify tests is using an extra (XXX link to 
section above, which should be fleshed out) like `[test]` or `[dev]`. 
In this case, upstream's instructions to install test dependencies 
might look like `pip install -e.[test]`.


Projects using `tox` usually specify test dependencies in a 
`tox`-specific format: a 
[requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires) 
key in the configuration.


Both forms are handled by the `%pyproject_buildrequires` macro, see 
below.


If upstream does not use either form, list test dependencies as manual 
*BuildRequires* in the `spec` file.



Should these be manually listed as Fed

Re: Many packages unnecessarily link to libpython

[Petr Viktorin]

On 2020-05-31 10:24, Honggang LI wrote:

On Fri, May 29, 2020 at 06:47:35PM +0200, Petr Viktorin wrote:



On 2020-05-18 03:53, Honggang LI wrote:

On Fri, May 15, 2020 at 02:12:00PM -0400, Charalampos Stratakis wrote:

rdma-coredledford honli jwilson


rdma-core pyverbs must be linked with libpython38. For example,

build]$ nm ./python/pyverbs/mem_alloc.cpython-38-x86_64-linux-gnu.so | grep -w 
U |  grep  PyInterpreterState_GetID
U PyInterpreterState_GetID

Python-3.8.3]$ nm -a ./build/debug/libpython3.8d.so | grep 
PyInterpreterState_GetID
000c6a82 T PyInterpreterState_GetID


This is exactly the case where the module should *not* be linked to
/usr/lib64/libpython3.8.so.
mem_alloc.cpython-38-x86_64-linux-gnu.so is a Python module, so it is loaded
from Python when imported. The import will link link it to the particular
Python interpreter it's imported into, which could be using a different
libpython3.8.so (such as the debug version).


OK, I see the problem now. But I don't know how to handle this.

Let's take ../python/pyverbs/srq.cpython-39-x86_64-linux-gnu.so as
example.

1) Link with debug version.
[honli@localhost pyverbs (master)]$ gcc  -fPIC  -Wall -Wextra -Wno-sign-compare 
-Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations 
-Wwrite-strings -Wformat=2 -Wcast-function-type -Wformat-nonliteral -Wdate-time 
-Wnested-externs -Wshadow -Wstrict-prototypes -Wold-style-definition 
-Wredundant-decls -O2 -g -DNDEBUG  -Wl,--as-needed -Wl,--no-undefined -shared 
-Wl,-soname,srq.cpython-39-x86_64-linux-gnu.so -o 
../python/pyverbs/srq.cpython-39-x86_64-linux-gnu.so 
CMakeFiles/srq.cpython-39-x86_64-linux-gnu.dir/srq.c.o  
-Wl,-rpath,/home/honli/upstream/rdma-core/build/lib:  
../lib/librdmacm.so.1.2.30.0 ../lib/libibverbs.so.1.9.30.0 -lpython3.9d

[honli@localhost pyverbs (master)]$ ldd 
../python/pyverbs/srq.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x7fff2fbe4000)
libibverbs.so.1 => 
/home/honli/upstream/rdma-core/build/lib/libibverbs.so.1 (0x7feed22c3000)
libpython3.9d.so.1.0 => /lib64/libpython3.9d.so.1.0 (0x7feed1e6e000)
libc.so.6 => /lib64/libc.so.6 (0x7feed1ca3000)
libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x7feed1c1b000)
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x7feed1bf7000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x7feed1bd5000)
libdl.so.2 => /lib64/libdl.so.2 (0x7feed1bcc000)
libutil.so.1 => /lib64/libutil.so.1 (0x7feed1bc7000)
libm.so.6 => /lib64/libm.so.6 (0x7feed1a81000)
/lib64/ld-linux-x86-64.so.2 (0x7feed22f7000)

[honli@localhost pyverbs (master)]$ ldd 
../python/pyverbs/srq.cpython-39-x86_64-linux-gnu.so | grep libpython3.9d.so.1.0
libpython3.9d.so.1.0 => /lib64/libpython3.9d.so.1.0 (0x7f32570ac000)

[honli@localhost pyverbs (master)]$ rpm -qf /lib64/libpython3.9d.so.1.0
python3-debug-3.9.0~b1-3.fc33.x86_64

2) link with normal version
[honli@localhost pyverbs (master)]$ gcc  -fPIC  -Wall -Wextra -Wno-sign-compare 
-Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations 
-Wwrite-strings -Wformat=2 -Wcast-function-type -Wformat-nonliteral -Wdate-time 
-Wnested-externs -Wshadow -Wstrict-prototypes -Wold-style-definition 
-Wredundant-decls -O2 -g -DNDEBUG  -Wl,--as-needed -Wl,--no-undefined -shared 
-Wl,-soname,srq.cpython-39-x86_64-linux-gnu.so -o 
../python/pyverbs/srq.cpython-39-x86_64-linux-gnu.so 
CMakeFiles/srq.cpython-39-x86_64-linux-gnu.dir/srq.c.o  
-Wl,-rpath,/home/honli/upstream/rdma-core/build/lib:  
../lib/librdmacm.so.1.2.30.0 ../lib/libibverbs.so.1.9.30.0 -lpython3.9

[honli@localhost pyverbs (master)]$ ldd 
../python/pyverbs/srq.cpython-39-x86_64-linux-gnu.so | grep libpython3.9
libpython3.9.so.1.0 => /lib64/libpython3.9.so.1.0 (0x7f7f8a7c7000)

[honli@localhost pyverbs (master)]$ rpm -qf /lib64/libpython3.9.so.1.0
python3-libs-3.9.0~b1-3.fc33.x86_64

As you see, "-lpython3.9" or "-lpython3.9d" library must be provided to
the linker. Otherwise, a lot of "undefined reference to xxx" error
messages show up.

/home/honli/upstream/rdma-core/build/pyverbs/srq.c:10425: undefined reference 
to `PyErr_GivenExceptionMatches'
/home/honli/upstream/rdma-core/build/pyverbs/srq.c:8987: undefined reference to 
`_PyObject_GenericGetAttrWithDict'
/home/honli/upstream/rdma-core/build/pyverbs/srq.c:7786: undefined reference to 
`PyModuleDef_Init'

[honli@localhost pyverbs (master)]$ grep undefined error.txt |wc -l
876

Any hits how to handle this?


Judging from the file name, mem_alloc.cpython-38-x86_64-linux-gnu.so is 
a Python module. If that's so, you shouldn't link it using gcc, but use 
a Python import statement to load it.

Is there some reason to link it using gcc?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to dev

Re: [HEADS UP] Fedora 33 Python 3.9 rebuilds have started in a side tag

[Petr Viktorin]

On 2020-05-29 16:39, Miro Hrončok wrote:

On 29. 05. 20 16:25, Richard Shaw wrote:
On Fri, May 29, 2020 at 9:18 AM Miro Hrončok > wrote:



    The side tag has been merged. Thank you all for your patience.


Woohoo! So now Python can be rebuilt with the fix for PySide2? :)


Technically blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1839826

But I am sure Petr is working on it.


It's progressing, though slower than I hoped due to other issues. I'll 
rebuild PySide and close the bug when done.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: How to determine maintainer of a package en mass?

[Petr Viktorin]

On 2020-05-29 01:34, Fabio Valentini wrote:

On Fri, May 29, 2020 at 1:16 AM Richard Shaw  wrote:


Specifically, I would like a way to determine which packages I am the sole 
maintainer of or the main maintainer.

Long story short, I've been spending far too much time on packaging work and it's taking 
away from $LIFE, $DAYJOB, and other hobbies I used to enjoy. I need to offload the 
primary maintenance of some of them to "lighten the load" so to speak.

No worries, I'm not disappearing or anything and will continue to do what I can.


This shows all the packages for which you are "owner" / "main admin":
https://src.fedoraproject.org/dashboard/projects?acl=main+admin

This shows all packages for which you have "admin" access (whether
directly or via group membership):
https://src.fedoraproject.org/dashboard/projects?acl=admin

Do those two pages show the information you need?


There's also a JSON with all maintainers for all packages here: 
https://src.fedoraproject.org/extras/pagure_owner_alias.json

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Many packages unnecessarily link to libpython

[Petr Viktorin]

On 2020-05-18 03:53, Honggang LI wrote:

On Fri, May 15, 2020 at 02:12:00PM -0400, Charalampos Stratakis wrote:

rdma-coredledford honli jwilson


rdma-core pyverbs must be linked with libpython38. For example,

build]$ nm ./python/pyverbs/mem_alloc.cpython-38-x86_64-linux-gnu.so | grep -w 
U |  grep  PyInterpreterState_GetID
U PyInterpreterState_GetID

Python-3.8.3]$ nm -a ./build/debug/libpython3.8d.so | grep 
PyInterpreterState_GetID
000c6a82 T PyInterpreterState_GetID


This is exactly the case where the module should *not* be linked to 
/usr/lib64/libpython3.8.so.
mem_alloc.cpython-38-x86_64-linux-gnu.so is a Python module, so it is 
loaded from Python when imported. The import will link link it to the 
particular Python interpreter it's imported into, which could be using a 
different libpython3.8.so (such as the debug version).

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Location of executable code

[Petr Viktorin]

On 2020-05-22 17:31, Steve Grubb wrote:

On Friday, May 22, 2020 10:39:43 AM EDT Petr Viktorin wrote:

On 2020-05-22 16:30, Steve Grubb wrote:

Hello,

I am working on our application whitelisting daemon. It uses the rpmdb to
derive trust in what's on disk. If we use the whole rpmdb, then the
number of files is large. So, to prune the amount of entries in the
trust db down to a reasonable number, I thought we could jettison
anything in /usr/share.

According to the Filesystem Hierarchy Standard [1] it says this about
/usr/ share:

The /usr/share hierarchy is for all read-only architecture independent
data files.

But what I'm finding in practice is that cinnamon places its javascript
there, there are libexec dirs that contain executable code, there are
python and byte compiled python over there. In short, the system doesn't
work because critical executables are in /usr/share.

The question is what should be done about this? Do we care that things
are in /usr/share that are not following the Filesystem Hierarchy
Standard? If we do, what is the proper fix this this? Should bz be
opened against each component?


Does "read-only architecture independent data" mean the files must not
be programs?


I personally think of data and applications/modules/libraries as distinct
entities.


Javascript, Python scripts and Python bytecode are all read-only and
architecture independent.


But they are not data. What I'm trying to do is constrain the notion of what
can be executed.


Interesting. Where do you draw the line?

I guess /usr/lib*/python3.*/pydoc_data/topics.py is not data?
What about a YAML file? An Ansible playbook? A PostScript image?
A JavaScript file intended to be minified and sent to a browser?

Not that those are very practical questions. I'm just curious.


And everything on disk is data.
The document you linked explicitly gives troff macros and "perl" as
examples of what goes in /usr/share/.


Indeed it does. But it also suggests /usr/lib as another home for that. If
there was a /usr/share/code  or /usr/share/perl or some convention like that,
it would make the job easier because applications are discoverable. Instead
they are hidden and in unexpected places. And its hard to tell what are
example programs from something that the system depends on. There needs to be
clarity about the intentions of any system code that lives in /usr/share. At
least the libexec dirs are obvious. Is it possible to make this system code
discoverable?


Possible, but not easy. There's currently no clear distinction, and even 
if you come up with a clear spec, it won't be easy to move all the files.




-Steve


IMO, /usr/share/ would be a better place for *all* pure-Python modules
than /usr/lib/, where they are now. The main reason they weren't moved
is that the move would cause a lot of disruption.


1 - https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Location of executable code

[Petr Viktorin]

On 2020-05-22 16:30, Steve Grubb wrote:

Hello,

I am working on our application whitelisting daemon. It uses the rpmdb to
derive trust in what's on disk. If we use the whole rpmdb, then the number of
files is large. So, to prune the amount of entries in the trust db down to a
reasonable number, I thought we could jettison anything in /usr/share.

According to the Filesystem Hierarchy Standard [1] it says this about /usr/
share:

The /usr/share hierarchy is for all read-only architecture independent data
files.

But what I'm finding in practice is that cinnamon places its javascript there,
there are libexec dirs that contain executable code, there are python and
byte compiled python over there. In short, the system doesn't work because
critical executables are in /usr/share.

The question is what should be done about this? Do we care that things are in
/usr/share that are not following the Filesystem Hierarchy Standard? If we
do, what is the proper fix this this? Should bz be opened against each
component?


Does "read-only architecture independent data" mean the files must not 
be programs?
Javascript, Python scripts and Python bytecode are all read-only and 
architecture independent. And everything on disk is data.
The document you linked explicitly gives troff macros and "perl" as 
examples of what goes in /usr/share/.


IMO, /usr/share/ would be a better place for *all* pure-Python modules 
than /usr/lib/, where they are now. The main reason they weren't moved 
is that the move would cause a lot of disruption.



1 - https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: [HEADS UP] Fedora 33 Python 3.9 rebuilds have started in a side tag

[Petr Viktorin]

On 2020-05-22 13:59, Tomasz Kłoczko wrote:
On Fri, 22 May 2020 at 12:02, Miro Hrončok > wrote:


On 22. 05. 20 12:43, Tomasz Kłoczko wrote:
 > [mode=serious]
 > On making transition from 3.8.x to 3.9.x all what would be
necessary to do would
 > be just create compat-python3.8 package -> upgrade python.spec to
3.9 -> monitor
 > number of packages still dependent on compat-python3.8 to focus
what still needs
 > to be ported/rebuild. ...snip

This doesn't work. As long as you rebuild setuptools and other
essential
packages with 3.9, the entire 3.8 stack would be useless. You would
need compat
packages for hundreds of packages to make this work.


Is that not *the*target/goal?.. make whole old stack useless/obsolete ASAP?

And no you would not need that because as long as you will not remove 
from repository all python---.f22 
packages all dependencies will be fulfilled.
Remember that each package is not rebuild on entire system but on very 
limited one with (theoretically) necessary dependencies. With that it is 
possible to rebuild all packages one by one and move everything in one 
batch. To make that whole transition easier all what will be 
necessary to do is clone current repository to something like 
fedora-transition-python and add keep all rebuild packages in that repo 
and merge all that packages in one go when all will be finished. 
No stress or time pressure. such transition can take weeks if not months 
and decision about merge could be done instantly basing on only one 
metric which is number of packages still dependent on compat-python3.8. 


How is this "fedora-transition-python" different from a side tag?

The problem with "cloning" is that people build new versions in Fedora 
all the time, so the "clone" gets out of date (with some additional 
problems related to version/release numbers). Unless the "clone" is kept 
up to date, merging the repo would mean downgrading some packages.


That is the main reason for the time pressure: the longer the rebuild 
takes, the more the repos diverge.


Than if at the and will be still some remains hard quickly to port it 
should be made decision about create some limited number of packages 
like compat-python3.8- and/or to put some of those packages on 
obsoletes list. > With single metric decision about actual transition will be no longer in

hands of anyone (as single person or comity).
It will be strict technical decision based on quite well defined set of 
conditions hanging on that metric.


So, what conditions do you propose?

Why would a condition (decided by one person or commitee) be better than 
that person (or comittee) deciding on the fly -- based on not just the 
number, but the actual packages that still fail to build?


If we are talking about using that methodology not only for python in 
some simple enough cases such decision could be done .. automatically!
Basing on my already +25y exp on building rpm packages I can even bet 
that after some cleanups in all python related stuff it should be 
possible to make python major release upgrade *automatically.*


I'm afraid the clean-ups, and keeping the clean-ups up to date, would be 
more work than doing the work manually.


Whoever would want to help on that transition will be asked to add that 
fedora-transition-python repository to own build systems or own personal 
system. Initial set of rebuilds made automatically should provide enough 
set of new dependencies allowing fixing one by one each package which 
will need manual intervention in form of some patches.


That's very similar to what's been happening in COPR for months now.

Above and what I wrote in my prev email could be general methodology on 
making any future transition of bigger groups of packages from one major 
version to another one (not only python specific).


All this is just like git branching but on packages with packages 
repositories as well.


Branching is not hard; merging is. Both Git and CVS could branch; Git 
won because the *merging* (and thus the whole branch-based workflow) is 
easier.


It is yet another consequence of what I've sketched. With that it 
would be possible to remove all python packages bootstrap bconds:


[tkloczko@barrel SPECS.fedora]$ grep bootstrap python* | grep bcond
python3.spec:%bcond_with bootstrap
python-BTrees.spec:%bcond_with bootstrap
python-dask.spec:%bcond_without bootstrap
python-fsspec.spec:%bcond_without bootstrap
python-pbr.spec:%bcond_with bootstrap
python-setuptools.spec:%bcond_with bootstrap
python-setuptools.spec:# Warning, different bootstrap meaning here, has 
nothing to do with our bcond

python-sphinx_rtd_theme.spec:%bcond_with bootstrap
python-stestr.spec:%bcond_without bootstrap
python-wheel.spec:%bcond_with bootstrap
python-zbase32.spec:%bcond_with bootstrap


Sorry, I don't see how this would be possible.

Of course to that picture needs to be added yet another small bit which 
is 

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-05-06 11:24, Lumir Balhar wrote:

I've took a look and the new guidelines look good to me.

The only thing I am afraid of is that there is a lot of magic behind new 
macros. Previously, macros were a way how to use standard Python 
commands like "python3 setup.py build" without memorizing them and 
without a fear that you forget an recommended/standard/important 
command-line option.


The problem is that "setup.py" is not really the standard any more. It 
only works with packages that use setuptools. Currently that's most of 
them, but maintainers of setuptools are very unhappy with the monopoly.


But now, the macros are much more complex and 
newcomers might not understand what is behind them. Moreover, given the 
fact that some of them are MUST, we might be too strict there. It is, of 
course, a matter of taste, but what if I simply prefer to list all my 
runtime dependencies manually to have a comprehensive list which also 
includes all the dependencies outside Python world?


How do you ensure that list stays in sync with upstream?


I mean, sometimes I like to implement the latest possibilities and 
macros to test them and see how they can help me and sometimes I like to 
use the old way, write everything explicitly/manually and don't use 
automation.


Everything should be written explicitly, but in setup.py (or wherever 
the build backend looks for this info). Why do you want to duplicate it 
in the spec?

Do you have any concrete use cases? "Sometimes" is hard to reason about.

Are there any specific MUSTs you would change to SHOULDs?
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Macronize %py3_shebang_fix

[Petr Viktorin]

On 2020-05-06 13:44, Miro Hrončok wrote:

In this change:

https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error

We have advised the following:

   pathfix.py -pni "%{__python3} %{py3_shbang_opts}" 

To fix the shebangs.

  -p preserves timestamps
  -n prevents creating ~backup files
  -i specifies the interpreter for the shebang

Now we have new features and we ultimately want this for the best 
experience:


   pathfix.py -pni %{python3} -ka %{py3_shbang_opts_nodash}

  -k keeps existing flags
  -a adds our flags (if not already there)
  %py3_shbang_opts_nodash is derived from %py3_shbang_opts

This is very tedious copypasta.

I propose we macronize this as follows:

We create %py3_shebang_flags. By default, it is the same as 
%py3_shbang_opts without dash to avoid confusion, but it can be 
overridden. Note the proper "e" in the name.


We create %py3_shebang_fix:


Bikeshed: "%py3_fix_shebang" is a verb



   pathfix.py -pni %{__python3} -k%{?py3_shebang_flags:a 
%py3_shebang_flags}


Usage:

   %prep
   %autosetup
   %py3_shebang_fix .

Or:

   %install
   ...install stuff...
   %py3_shebang_fix %{buildroot}%{python3_sitearch} 
%{buildroot}%{_bindir}/*


+1
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-04-30 15:41, Petr Viktorin wrote:

Hello!
Below is a draft of new Packaging Guidelines! It's full of unfinished 
areas (marked with XXX), but it's ready for scrutiny.
A possibly updated version is on 
https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view

[...]
The draft lives on hackmd.io, which we found easy to collaborate with. 
If you have an account there, we can add you. If you'd like to 
collaborate some other way, let us know.



There were some comments on the hackmd pad, which might be more visible 
in e-mail, so I'll quote & answer them here.

They seem like details, though. Do the overall ideas look good?


One comment was about the "Dots in package names" comment. That's in 
progress; see the mail from Tomáš about renaming `python39` to 
`python3.9`: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/VIUS7WMQMDX6H2WEIH7TVTMBB6SUHY7E/




### Canonical project name

Most of these names are case-sensitive machine-friendly identifiers, but 
the *project name* has human-friendly semantics: it is case-insensitive 
and treats some sets of characters (like `._-`) specially.
For automated use, it needs to be 
[normalized](https://www.python.org/dev/peps/pep-0503/#normalized-names) 
to a canonical format used by Python tools and services such as 
setuptools, pip and PyPI.

The `%{py_dist_name}` macro implements this normalization.

Elsewhere in this text, the metavariable `DISTNAME` refers to the 
canonical form of the project name.


 > XXX
 > ```spec
 > # XXX specfile
 > %py_set_name Django
 > -> %distname django
 > -> %pojectname Django
 > ```


This is quite a cryptic note, but the idea is that you'd do 
`%py_set_name Django`, and %distname & %projectname would get defined 
for you, so that you can use them in the rest of the spec file.


Zbyszek said:

Please don't advertise macros like %distname, %srcname, %origname, and
the countless other variants in use. They made sense when people used > editors 
which didn't support search easily. Having the name
verbatim in there makes the whole thing easier to read and copy


The idea was to standardize these, so we'd get well known names instead 
of each packager defining their own. Then, macros like %pypi_url could 
use the well-known name by default. This would also take care of the 
subtle differences between names: in Django's case, you need to use 
capital D in the URL, but automatic provides use the canonicalized 
lowercase. Even search/replace is likely to miss these (but, inevitably 
packagers themselves will also be confused :/)


It's definitely possible to get rid of %py_set_name, and deprecate 
%pypi_url's weird default -- you'd be expected to write out `%pypi_url 
Django` every time.
I'm not sure if that would prevent people from using 
%distname/%srcname/%origname/%pypiname, but it looks like the right 
thing to do.




### Machine-readable provides

Every Python package **MUST** provide `python3dist(DISTNAME)` **and** 
`python3.Xdist(DISTNAME)`, where `X` is the minor version of the 
interpreter and `DISTNAME` is the *canonical project name* corresponding 
to the *dist-info metadata*, for example `python3.9dist(requests)`. 
(XXX: add links to previous sections)


This is generated automatically from the dist-info metadata.

The provide **SHOULD NOT** be added manually: if the generator fails to 
add it, the metadata **MUST** be fixed.


If necessary, the automatic generator can be disabled by undefining 
`%__pythondist_provides`.


Neal writes:

There should be absolutely _no_ circumstance where the Provides
generator needs to be completely disabled. What cases were you
thinking about that might require this?


IMO, when there is automation, you should be able to disable it. It's 
for the cases I *don't* know about.


But I'll update to some stronger language.
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

[Petr Viktorin]

On 2020-05-01 23:41, Michel Alexandre Salim wrote:

Hi,

On 4/30/20 6:41 AM, Petr Viktorin wrote:
The draft lives on hackmd.io, which we found easy to collaborate with. 
If you have an account there, we can add you. If you'd like to 
collaborate some other way, let us know.


Please add mic...@michel-slm.name, thanks!

(I'm assuming hackmd.io allows commenting on the document without 
overwriting it willy-nilly)


It's versioned and rollbacks are reasonably cheap, so go ahead and write 
into the document. Preferably in "XXX" notes, as in the rest of the text.


There is a "comment" feature, but it's not that great IMO. On the other 
hand, you can comment as a guest.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Draft of New Python Packaging Guidelines

[Petr Viktorin]

Hello!
Below is a draft of new Packaging Guidelines! It's full of unfinished 
areas (marked with XXX), but it's ready for scrutiny.
A possibly updated version is on 
https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view


Generally, for rules marked **SHOULD** we know of cases where they 
should be broken; for things marked **MUST** we don't.


We have tried to only take the Good Existing Things™ (macros, practices) 
and revise the rest without thinking about the past much. Some used 
technology is unfortunately not compatible with current EPELs, but we 
have considered Fedora 31+. Using the current Python guidelines will 
still be an option for people who target EPEL 6/7/8.


The main controversial idea (of many) is synchronizing Fedora's names 
(a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python 
Package Index (PyPI, pypi.org), which has by now become the de-facto 
canonical namespace for freely redistributable Python packages.
We believe that this will help both Fedora and the greater Python 
ecosystem, but there will definitely be some growing pains.


Most of Fedora Python packages already are on PyPI, but more than 250 
are missing. There is software developed within Fedora (e.g. pagure, 
fedpkg, rust2rpm); projects that aren't primarily Python packages (e.g. 
perf, ceph, xen, setroubleshoot, iotop); and others.
A full list is attached. The names have been temporarily blocked on PyPI 
to keep trolls from taking them while this is discussed.
Over at the Python Discourse we are discussing how to properly handle 
these; once that discussion is settled they should be unblocked: 
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a-k-a-pypi-namesquatting-e-g-for-fedora-packages/4045


Another general idea is that package metadata should be kept upstream; 
the spec file should duplicate as little of it as possible. Any 
adjustments should be done with upstreamable patches.


The draft lives on hackmd.io, which we found easy to collaborate with. 
If you have an account there, we can add you. If you'd like to 
collaborate some other way, let us know.


Petr and Miro

---

Current draft for inline comments:


> [IMPORTANT]
> This is a DRAFT; it is not in effect yet.

# Python Packaging Guidelines (draft)

> [IMPORTANT]
> This is a *beta* version of the Python Packaging Guidelines and the 
associated RPM macros.
> Packagers that opt in to following this version **MUST** be prepared 
to change their packages frequently when the guidelines or macros are 
updated.
> These packagers **SHOULD** join [Python SIG mailing 
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/) 
and monitor/start conversations there.


These Guidelines represent a major rewrite and paradigm shift, and not 
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages 
**MAY** use them instead of this document:
* 201x-era Python packaging guidelines (for packages that use e.g. 
`%py3_install` or `setup.py install`)

* Python 2 appendix (for e.g. `%py2_install`) (requires FESCo exception)

> [NOTE]
> These guidelines only support Fedora 31+. For older releases (such as 
in EPEL 8), consult the older guidelines (XXX link).


The two "Distro-wide guidelines" below apply to all software in Fedora 
that uses Python at build- or run-time.


These rest of the Guidelines apply to packages that ship code that *can* 
be imported in Python.
Specifically, that is all packages that install files under 
`/usr/lib*/python*/`.


Except for the two "Distro-wide guidelines", these Guidelines do not 
apply to simple one-file scripts or utilities, especially if these are 
included with software not written in Python.
However, if an application (e.g. CLI tool, script or GUI app) needs a 
more complex Python library, the library **SHOULD** be packaged as an 
importable library under these guidelines.


A major goal for Python packaging in Fedora is to *harmonize with the 
wider Python ecosystem*, that is, the [Python Packaging 
Authority](https://pypa.io) (PyPA) standards and the [Python Package 
Index](https://pypi.org) (PyPI).
Packagers **SHOULD** be prepared to get involved with upstream projects 
to establish best practices as outlined here. We wish to improve both 
Fedora and the wider Python ecosystem.


> [NOTE]
> Fedora's Python SIG not only develops these guidelines, but it's also 
involved in PyPA standards and Python packaging best practices. Check 
out [the wiki](https://fedoraproject.org/wiki/SIGs/Python) or [mailing 
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/) 
if you need help or wish to help out.



## Distro-wide guidelines

### BuildRequire python3-devel

**Every** package that uses Python (at runtime and/or build time), 
and/or installs Python modules **MUST** explicitly include 
`BuildRequires: python3-devel` in its `.spec` file, even if Python is 

Re: Redesigning the %python_provide macro from scratch

[Petr Viktorin]

I finally got around to this mail...

On 2020-04-19 16:55, Miro Hrončok wrote:

Hello Python packagers.

After touching the %python_provide topic in:

https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/SSJLPWSGFGPYRSHXQZDR7JNQXSDGGX3Z/ 



I have realized several things I don't like about %python_provide:

1) It must be used conditionally (it is not defined in python-srpm-macros).
That means you always wrap it in %{?python_provide:...} in order to have 
a "valid" specfile even when the macro is not yet defined (e.g. during 
SRPM creation in Koji or on a packager's machine without 
python-rpm-macros installed).


2) You cannot use it with arbitrary versions. Suppose you package 
python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason 
-- this is not very common, but it happens. %python_provide only takes 
the "name" as an argument, always using %{?epoch} %{version} and 
%{release}.


3) You need to both add a virtual provide + use the macro. Suppose you 
want to provide python3-pkg_resources from python3-setuptools. 
Currently, you need to add:


   Provides: python3-pkg_resources = %{version}-%{release}
   %{?python_provide:%python_provide python3-pkg_resources}

4) When used with (sub)package name, it generates a duplicate dependency 
on Fedora 33+ (and an rpmlint error).


5) It produces Obsoletes, but that might no longer be necessary nor 
desired.


6) Broken expectations about %_isa. It used to add %_isa provides based 
on wrong data, it no longer does that (except on old releases and 
EPELs), can be used manually with name%{?_isa}, but not on the old 
releases.


7) Undocuemnted error handling (e.g. the macro expands to nothing when 
used with pypy-foo, but errors when used with foo).



Hence, I was thinking (for the sake of backwards compatibility) to 
provide a new mechanism to do this and preserve the old macro as is, 
deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 
goes EOL (or never, which is basically the same from today's perspective).


The new macro should solve the problems from above, my current (quick, 
untested) ideas are:



1) Define the macro in python-srpm-macros. No need to use it 
conditionally. We can backport it to EPEL 8 and define a "stub" macro in 
EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes 
EOL, we can make the macro behave consistently across all Fedora versions.)


2) Accept version identifier as an optional argument, use %{?epoch} 
%{version} and %{release} as default. (See for example %{pypi_source} on 
how this can be done.)


3) Make the macro also produce the provide for the given name and 
document that. E.g. when you call it with python3-pkg_resources, it also 
provides python3-pkg_resources (not only python-pkg_resources etc.).


4) Make it so that for given arguments, the macro will only expand to 
something once per build. Hence when you use it with package name, the 
automatic provides won't re-add the same provide again. This also means 
you cannot have 2 different (sub)packages provide the same 
name-version-release, but that shall be very very very uncommon need and 
can always be workarounded somehow if needed.


5) No obsoletes with the new macro. Packagers use manual obsoletes when 
desired.


6) Document clearly that there is no %{?_isa} (and there is no 
"backwards compatibility" load to carry). When absolutely desired, 
packagers can call the macro with %{name}%{?_isa}.


7) Support arbitrary names. Only provide the given name and nothing else 
if not "recognized".


Is that better than erroring out when something is not recognized?


As a bonus, I think the current if-elif-elif-elif-elif code can be 
replaced with lua patterns (imagine regex).



As always, this leaves us with the name problem, but I'd very much like 
to use %python_provides (note the s). The only problem I see is that it 
is likely to be mistaken for the old one, but IMHO it shouldn't really 
hurt that much.


I guess something like %py_provides would also work, and maybe fit more 
nicely with the new generation of macros. Do you like full "python" prefix?

But that's bikeshedding.



Usage example:

   %package -n python3-setuptools
   %python_provides python3-pkg_resources

Resulting provides:

   python3-setuptools = 46.6.6-6.fc33
   python-setuptools = 46.6.6-6.fc33
   python39-pkg_resources = 46.6.6-6.fc33
   python3-pkg_resources = 46.6.6-6.fc33
   python-pkg_resources = 46.6.6-6.fc33
   python39-pkg_resources = 46.6.6-6.fc33


I assume you meant "python39-setuptools" there.



Another example:

   %package -n python3-pillow
   %python_provides python3-PIL 1.1.6-100

Resulting provides:

   python3-pillow = 7.1.1-1.fc33
   python-pillow = 7.1.1-1.fc33
   python39-pillow = 7.1.1-1.fc33
   python3-PIL = 1.1.6-100
   python-PIL = 1.1.6-100
   python39-PIL = 1.1.6-100


Dummy when used with package name:

   %package -n python3-pip
   %python_provides python3-pip

Provides:

Re: Why is python(abi) generated? (Re: Rawhide python-rpm-generators news)

[Petr Viktorin]

On 2020-04-07 12:40, Miro Hrončok wrote:

On 07. 04. 20 11:06, Petr Viktorin wrote:

On 2020-04-03 20:44, Miro Hrončok wrote:

Hello Python packagers.

I have just updated python-rpm-generators to 
python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff 
from RPM 4.16 and will not be backported to older releases.



The python(abi) requirement is now added from a RPM Lua macro, 
instead of by executing a Shell script. This is a bit faster 
especially for packages with a lot of files. For 10 000 files in one 
package, the speedup is roughly from ~80 to ~2 seconds (time of the 
entire package build incl. creating the files from a Python script). 
That is a lot, but most of the real packages have much less files, so 
I am afraid you won't feel it.


Hi,
One thing I'm wondering about is: why is python(abi) provided by a 
generator, anyway?
It's provided by a very small set of packages (python2 and python3 in 
Fedora). Couldn't it just be listed in those specfiles?


Just to put this into more context: The time overhead is for the 
requires. The provides are filtered on a certain path and even if we are 
automating stuff for 2 packages which indeed might not be needed, it 
doesn't really hurt anything.


See also 
https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org/thread/UFKUM5UKCTNGIT3KJVYEI5VXPI23QMBN/ 
for me trying to figure out when this provides are useful -- depending 
on the answer there, we might want to keep the automated provides of 
python(abi) only and remove the manual ones.


The part I don't get is: why keep the automated provides over manual 
ones. Compared to a line in the relevant spec files, the macro looks 
like it adds complexity for no benefit.


I get the automatic *requires* on python(abi) -- that's used by 
thousands of packages and easy to forget. But the provides will only 
ever be in CPython.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Why is python(abi) generated? (Re: Rawhide python-rpm-generators news)

[Petr Viktorin]

On 2020-04-03 20:44, Miro Hrončok wrote:

Hello Python packagers.

I have just updated python-rpm-generators to 
python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff 
from RPM 4.16 and will not be backported to older releases.



The python(abi) requirement is now added from a RPM Lua macro, instead 
of by executing a Shell script. This is a bit faster especially for 
packages with a lot of files. For 10 000 files in one package, the 
speedup is roughly from ~80 to ~2 seconds (time of the entire package 
build incl. creating the files from a Python script). That is a lot, but 
most of the real packages have much less files, so I am afraid you won't 
feel it.


Hi,
One thing I'm wondering about is: why is python(abi) provided by a 
generator, anyway?
It's provided by a very small set of packages (python2 and python3 in 
Fedora). Couldn't it just be listed in those specfiles?

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change Proposal: ELN Buildroot and Compose V3

[Petr Viktorin]

On 2020-04-03 14:43, Aleksandra Fedorova wrote:

On Fri, Apr 3, 2020 at 11:59 AM Petr Viktorin  wrote:


On 2020-04-02 20:07, Stephen Gallagher wrote:

On Thu, Apr 2, 2020 at 12:56 PM Miro Hrončok  wrote:

The change proposal received overly negative feedback by the packager community
as represented both by RHEL¹ and non-RHEL maintainers. Despite being reworked
several times, none of this feedback was reflected in the proposal, only new
reasons why this is going to be done the original way were added. It seems that
feedback is collected not to find the best technical solution, but rather to
find ways to justify a solution that's already decided.



This is needlessly antagonistic, Miro. We've collected the feedback in
good faith, examined it and then identified shortcomings with it. For
the sake of clarity in the Change Proposal, I've recorded that
reasoning there.


Key questions in the *Detailed Description* remain "out of scope of ELN" or not
answered clearly.


If something is not clear, please constructively offer that
information. I've been doing my best to rephrase and clarify anything
that has come up. Anything remaining that is "out of scope of ELN" is
literally just that. ELN's scope is largely "Make Fedora Rawhide more
useful to downstream so that downstream developers will work there
instead of internally".


# What if packages don't build on ELN?

The question isn't actually answered. Instead, the proposal answers
*how* packages will fail to build in ELN.


These are linked topics.

ELN is a "build profile" applied to Fedora Rawhide sources. If we take
a working Fedora Rawhide content and apply build profile to it, and it
breaks, then there two reasons why it may break:
1) the content is different because of the conditional, and this
conditional is broken. So we fix it.
2) the build profile is different, then we fix the buildroot and build
flags, and pungi config.

As we have a working baseline, it means that eventually if we reduce
the number of differences we reach the working state. Once we have
that working state we can start _adding_ differences again while
keeping it in a working state.


# If a package fails to build on ELN, what will happen?

What will the time limit be?
What exactly happens is a maintainer rejects the pull request?

# What if there is a new upstream feature RHEL wants in a package, how
will that go in?

Why is this not in scope? We're bringing RHEL development closer to
Fedora development. Non-Red Hatters don't know how a feature gets into
RHEL, and I don't blame them if they think this will affect their
workflow. Will it? How?


It is not a new thing which ELN changes. Red Hat developers have
always been working with Fedora on changes. If feature makes sense to
Fedora, it is not a matter of the ELN build, it is a matter of
bringing it to Fedora Rawhide.
ELN is not involved in this process.


OK, so would the following answer be correct?
That is not in the scope of ELN. As in the past, the feature can be 
either added to Fedora (and ELN) on its own merits, or it can be added 
directly to RHEL without affecting Fedora (and ELN) at all.



# What if I do not want to have %if's in my spec files?

What happens if ELN SIG cannot find a solution the maintainer is
comfortable with?


Again, we use raw Fedora Rawhide packages which work.


And if they don't work in ELN, it's up to the ELN SIG to make Rawhide 
packages work there. Right?



# What if I do not want to have %if's in my spec files, but I want to
see how changes show up in RHEL?

Sorry, but the answer reminds me of Modularity promises that the missing
pieces will be ready in the future.
What will the maintainer actually be expected to do in this case?


In Fedora - nothing. ELN is not the answer to all the wishes. And we
don't promise that we will ever cover this case in ELN. I don't see
any similarities with Modularity here.


The proposal should address the impact on all major packaging
styles, and while one side of the spectrum (packagers preferring single-spec
with conditionals) is covered, there are very few concrete details on the other
(packagers interested in simple spec files or completely uninterested in RHEL).
The affected packagers are concerned that the current proposal does not in fact
benefit Fedora (to an extent that would justify the disruptions), but rather
addresses mainly a downstream RHEL concern using Fedora's resources.


This is a great example of the Nirvana Fallacy. Essentially, you are
arguing that improvements for one group is not acceptable unless it
improves things for every other group too.


It doesn't have to *improve* things for the other group, but if not, it
should clearly acknowledge that it makes things worse (or the same), and
say how. Otherwise that group will justifiably feel excluded.

Say a new package is added to RHEL, which was previously only in Fedora.
The packager doesn't want RHEL conditionals in the spec. The change

Re: Fedora 33 System-Wide Change Proposal: ELN Buildroot and Compose V3

[Petr Viktorin]

On 2020-04-02 20:07, Stephen Gallagher wrote:

On Thu, Apr 2, 2020 at 12:56 PM Miro Hrončok  wrote:

The change proposal received overly negative feedback by the packager community
as represented both by RHEL¹ and non-RHEL maintainers. Despite being reworked
several times, none of this feedback was reflected in the proposal, only new
reasons why this is going to be done the original way were added. It seems that
feedback is collected not to find the best technical solution, but rather to
find ways to justify a solution that's already decided.



This is needlessly antagonistic, Miro. We've collected the feedback in
good faith, examined it and then identified shortcomings with it. For
the sake of clarity in the Change Proposal, I've recorded that
reasoning there.


Key questions in the *Detailed Description* remain "out of scope of ELN" or not
answered clearly.


If something is not clear, please constructively offer that
information. I've been doing my best to rephrase and clarify anything
that has come up. Anything remaining that is "out of scope of ELN" is
literally just that. ELN's scope is largely "Make Fedora Rawhide more
useful to downstream so that downstream developers will work there
instead of internally".


# What if packages don't build on ELN?

The question isn't actually answered. Instead, the proposal answers 
*how* packages will fail to build in ELN.


# If a package fails to build on ELN, what will happen?

What will the time limit be?
What exactly happens is a maintainer rejects the pull request?

# What if there is a new upstream feature RHEL wants in a package, how 
will that go in?


Why is this not in scope? We're bringing RHEL development closer to 
Fedora development. Non-Red Hatters don't know how a feature gets into 
RHEL, and I don't blame them if they think this will affect their 
workflow. Will it? How?


# What if I do not want to have %if's in my spec files?

What happens if ELN SIG cannot find a solution the maintainer is 
comfortable with?


# What if I do not want to have %if's in my spec files, but I want to 
see how changes show up in RHEL?


Sorry, but the answer reminds me of Modularity promises that the missing 
pieces will be ready in the future.

What will the maintainer actually be expected to do in this case?



The proposal should address the impact on all major packaging
styles, and while one side of the spectrum (packagers preferring single-spec
with conditionals) is covered, there are very few concrete details on the other
(packagers interested in simple spec files or completely uninterested in RHEL).
The affected packagers are concerned that the current proposal does not in fact
benefit Fedora (to an extent that would justify the disruptions), but rather
addresses mainly a downstream RHEL concern using Fedora's resources.


This is a great example of the Nirvana Fallacy. Essentially, you are
arguing that improvements for one group is not acceptable unless it
improves things for every other group too.


It doesn't have to *improve* things for the other group, but if not, it 
should clearly acknowledge that it makes things worse (or the same), and 
say how. Otherwise that group will justifiably feel excluded.



Say a new package is added to RHEL, which was previously only in Fedora. 
The packager doesn't want RHEL conditionals in the spec. The change 
doesn't make sense upstream or in Fedora (for example, the crypto 
backend needs to be replaced to comply with some regulation). Without 
the change, the package (and anything depending on it) cannot be built 
in ELN.
As I read the proposal, ELN will work with the packager and try to find 
a good solution. However, the discussion might be under pressure of RHEL 
guidelines.


What *actually* happens in this worst case? Will the macros be forced 
into the package?
(If yes, please say it explicitly, so we can discuss that! If not, 
hooray -- alse say that explicitly!)


Even if the worst case scenatio is very unlikely, it would help thinking 
about the not-ideal-but-not-worst cases as well.




Downstream RHEL concerns *are* Fedora concerns. I cannot stress this
enough: Fedora and Red Hat have a highly symbiotic relationship. The
more RHEL succeeds, the more support Fedora gets from Red Hat. The
more Fedora succeeds, the better the resulting RHEL product. It is
disingenuous to imply that something that "addresses mainly a
downstream RHEL concern using Fedora's resources" is a net-negative
for Fedora.


Yes, that is true! The question is, does the goal justify the means? And 
even before that, what exactly are the means (specifically, in terms of 
limiting packager autonomy)?



While benefiting the entire Fedora/RHEL/CentOS/EPEL ecosystem is certainly a
good goal, I believe that doing this in a way that alienates a significant part
of our packagers is a disservice to Fedora. The concerned packagers believe that
Fedora is (and should remain) the upstream for RHEL, not RHEL itself. This might
be shifting us to the 

Full history of linux kernel (Re: CPE Git Forge Decision)

[Petr Viktorin]

On 2020-03-31 21:39, Przemek Klosowski via devel wrote:
[...]> Still, the
official Linux git log ( 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?ofs=905000 
) lacks the development history preceding 2005-04-16, and starts with 
one heck of a commit:





FWIW, this is mainly because early Git was unable to cope with huge 
histories, converting old history to another format isn't 
straighforward, and the history is immutable.


Earlier history is out there and there are ways to `graft` it to a 
current repo to locally join the histories together. See e.g. 
https://stackoverflow.com/a/8130239/99057

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: ELN Buildroot and Compose

[Petr Viktorin]

On 2020-03-25 17:33, Aleksandra Fedorova wrote:

[Branching] removes community maintainer from the conversation about what
downstream is doing. While we want to give community member a voice in
that conversation.


I fear that this proposal *forces* the community member to participate 
in the discussion. That is a very different thing than giving them a voice.



On 2020-03-25 17:10, Miro Hrončok wrote:

On 25. 03. 20 16:47, Stephen Gallagher wrote:

I think we managed to miss a few key points in the Change Proposal
which is directly resulting in a bit of the confusion here.


Reading the rest of you e-mail I must agree that this is what happened. 
Would you mind putting the change back to incomplete state and resend it 
once more for feedback once this is addressed?



The first and most important of these is that ELN will*not*  be
building the complete Fedora package set. We're going to be building a
selected subset of packages (derived from packages in RHEL 8 and
EPEL). Our current expectation is that we are going to be looking at
fewer than 2500 source packages. We are still working up the initial
list of these and we will update the Change Proposal with it (as well
as providing a fedora-devel post breaking down the known owners).


Glad to hear that.


Second, from this reduced subset, we expect that the overwhelming
majority of the maintainers are Red Hat employees that already work on
RHEL. With this in mind, we think that the level of concern about how
this will affect non-Red Hat contributors is premature. We will reach
out directly to those non-Red Hat contributors we identify.


This actually worries me very much.

The subset of packages will need to form a working system; it will 
include the most important packages. I hope this doesn't end up limiting 
maintainers of Fedora's most central packages to only Red Hatters or 
those that agree to play by RHEL rules that they have no say in.


I am pretty sure that most packages/packagers will be OK with the ELN 
change. But "most" is not enough. We do need to take the "worst cases" 
into account. Consider a package that:
- is needed in RHEL, but only as a dependency of something vital; the 
RHEL maintainer doesn't care too much about it outside their work duties
- has an enthusiastic packager in Fedora, who prefers to leave RHEL work 
to paid developers


(As member of a team that, some time ago, used to be catch-all RHEL 
maintainer for anything that happened to be written in Python, I can 
definitely imagine situations similar to this.)


For such a package, we need to avoid pushing ELN macros into the 
package, otherwise we lose an enthusiastic contributor who does amazing 
work in Fedora.


Such a package might even be hypothetical right now, but the case still 
needs to be adressed. A package can fall into this category at any time 
in the future. It's not enough to identifying packagers that are 
affected now, and work with them. There needs to be a process for cases 
like these, and it can't be "merge these ELN macros now, there's a 
deadline coming; maybe we'll come up with a better process in the 
future". (Especially if Red Hat is unlikely to drive setting up a better 
process.)


IMO, any changes needed in EL need to be opt-in, with the understanding 
that opting in will be (almost always) beneficial for everyone involved. 
They need to feel like a gift ("Hello fedora packager, please accept 
this PR with EL macros; it'll make it easier to work on this together!") 
rather than forced ("Red Hat needs this; merge it or else").


To borrow Neal Gompa's words, ELN needs to not feel "like it's written 
for Red Hat people to move their sandbox outside of Red Hat into Fedora 
without anyone else to be able to play in the sandbox."



For ELN changes to be to be opt-in, there essentially needs to be a 
place to diverge -- a place for ELN hotfixes to live while polishing 
them and debating their usefulness in Fedora and further upstreams.
Here by "hotfix" I need something required in RHEL that is not yet ready 
for upstream. It is healthy to have a place for such things: it removes 
time pressure from discussions (people from the community very often 
have good ideas but less time), but allows the fix to be used now (so 
the rest of ELN isn't blocked).
Forks are good if they're not permanent (but, making them not permanent 
needs effort).



As Miro Hrončok continues:
As a Red employee if I need to diverge a RHEL package from Fedora, I 
prefer to use branches as well. And I know for a fact I am not alone in 
this. Please don't forget to identify and reach out to Red Hat 
contributors as well, not just the non-Red Hat contributors.


[Stephen Gallagher]:

Finally, of the set of packages that we're going to be including, we
anticipate around 200-300 of them will have distro conditionals that
need investigation (with fewer needing actual modification). The ELN
SIG will be doing this initial investigation and will provide guidance
(and/or PRs) to 

Re: Copr Build System - review of 2019 and vote for features in 2020

[Petr Viktorin]

On 2020-01-22 12:03, Kevin Kofler wrote:

Kevin Kofler wrote:

IMHO, this whole "delete by default" concept is inherently flawed and
dangerous and cannot be fixed. Notification e-mails can be lost in so many
ways (wrong Fedora notification settings, e-mail provider issues, spam
filter false positives, out-of-quota mailbox, etc.) or be missed due to
being offline for a prolonged period of time. It should never be allowed
to delete users' data without their explicit confirmation. Especially in
this case where it is not even possible to reupload the data because Copr
can no longer build for those EOL chroots (which is another quite annoying
limitation of Copr – allowing to build for EOL releases would also allow
people to try backporting select security fixes to those releases Fedora
no longer wants to support).


PS: I also think that at the very least, there ought to be a way to
permanently opt-out a Copr repository out of all future cleanups. Some Coprs
such as the Kannolo Copr should just always be preserved.

I also do not understand why 6 TB of disk space is such an issue in times
where one single HDD can carry up to 16 TB.


AFAIU, the problem is that if something should *always* be preserved, 
you're looking at very different hardware (and associated maintenance 
work) than a single HDD.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Self-introduction

[Petr Viktorin]

On 2019-12-31 13:55, Fabian Affolter wrote:

Hi all,

I don't want to repeat myself, please take a look at my wiki page [1].

Why I want to join? I have one too many Python package and I started to
add the python-sig as admin. But it goes both ways, if the SIG has
access to my packages I want to be part of the SIG.  My name is in the
member list [2] for quite a while but I guess that doesn't come with the
right permissions.


Welcome! As the list says, you've been a member of Python SIG for a long 
time. That's basically just people who care about Python on Fedora and 
are on this list :)


The "python-sig" FAS group [3] is something slightly different. It's 
confusingly named (IIRC only groups with "-sig" in their name can get 
some permissions). It's there for people who want to fix Python-related 
issues in all the packages. Something like provenpackagers for Python.
Is that what you want? (From your mail it seems the confusing name got 
you...)


I'm not a sponsor in the FAS group, but I'd be glad to have you on board :)


[1] https://fedoraproject.org/wiki/User:Fab
[2] https://fedoraproject.org/wiki/SIGs/Python/Members_list


[3] https://fedoraproject.org/wiki/SIGs/Python#Python_SIG_FAS_group
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Python bindings for protobuf on RHEL/CentOS/EPEL 8

[Petr Viktorin]

On 2019-11-14 16:13, Denis Arnaud wrote:



Thanks Petr for your answer!

Let's give the COPR short-term solution a try, then :)

What is frustrating is that we cannot get our hands on the package 
source for Protobuf for RHEL/CentOS, and we cannot therefore know who 
maintains it at RedHat.


The maintainer is (usually, effectively) also the Bugzilla assignee.

For Boost for instance, which is also a RedHat 
core package, I know the RedHat maintainer (Jon Wakely) and can 
co-maintain Boost with him and have an influence on (non-EPEL) 
RHEL/CentOS Boost.


In my personal opinion, that's the way it should be; it benefits both 
Red Hat and the community.


At least, we get visibility on when, why and how 
Boost is rebuilt on RHEL/CentOS. It would be nice to achieve the same 
with Protobuf. But I have no clue on how to identify who at RedHat is 
(are) the main maintainer(s) of Protobuf. I am sure they would be 
interested if I/we can build protobuf with Python bindings for 
RHEL/CentOS 8 (at least on EPEL).
And it seems strange to me that RHEL/CentOS misses Python bindings for 
Protobuf (which is the cornerstone to many frameworks (e.g., gRPC)). I 
wonder how many applications work at all. Or maybe those Python bindings 
are build-time only in the new modularization framework?
It would help to have a way, for Fedora/EPEL volunteers, to be able to 
reach out to RedHat core package maintainers, at least for 
cross-awareness, knowledge sharing and potential collaboration.


Kind regards

Denis


14 Nov. 2019 15:27, Petr Viktorin mailto:pvikt...@redhat.com>> wrote:

On 2019-11-11 11:45, Denis Arnaud wrote:
 > Hi,
 >
 > the Python (3) bindings are missing on RHEL/CentOS/EPEL 8 for
the
 > protobuf package (https://src.fedoraproject.org/rpms/protobuf).
 > A bug request has been created on Bugzilla
 > (https://bugzilla.redhat.com/show_bug.cgi?id=1765844), but as
no status
 > has been given, I was wondering whether someone could shed
some light on
 > the context.
 >
 > Since protobuf is a RedHat core package (maintained by RedHat
and
 > therefore not managed by Fedora/EPEL), it appears as a kind
of black box
 > from Fedora perspective. On Fedora (Rawhide, 31), the Python (3)
 > bindings are generated/packaged (see for instance
 >
https://bodhi.fedoraproject.org/updates/FEDORA-2019-e3a662fe8b and
 > https://koji.fedoraproject.org/koji/rpminfo?rpmID=19440119),
but for
 > some reason, those Python bindings are not generated by
RedHat for
 > RHEL/CentOS 8.
 >
 > 1. Would anyone from RedHat be able to provide some heads up
on why
 > those Python 3 bindings are missing for Protobuf, and/or an
approximate
 > timeline for when it would be generated?

Hi!
We just talked about this among Red Hat python-maint. (Sorry for
the
delay!) We don't have influence over this package; unfortunately we
can't help directly.

 > 2. Would RedHat need help with packaging protobuf on
RHEL/CentOS/EPEL 8?

At this level "Red Hat" is not a single entity. You're probably not
reaching the right people on this list.
If the packager isn't active in Fedora or on Bugzilla, then
unfortunately, the way to get them to respond is to file a customer
support ticket.
Another way to approach this is change EPEL rules somehow to
make it
possible to build subpackages like this. From what I understand,
it's
being discussed, but it'll take time to get sorted out.

 > 3. Would you recommend another way for Fedora packagers/users
to get
 > their hands on the python3-protobuf/protobuf-python3 package?
For
 > instance, through COPR, or some module we may have missed.

Building a replacement protobuf package in COPR is the best
short-term
solution.

Sorry I couldn't be more helpful.


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

List of Python 2 packages to be removed mid-November (= in a week)

[Petr Viktorin]

Dear maintainers,
here is an updated list of packages that (transitively, at build or run 
time) require Python 2 and have not yet got a FESCo exception to do so.

If you were bcced on this e-mail, it affects one or more of your packages.

The default action will be to remove such packages starting November 15.

If this took you by surprise, don't panic. It's possible to change the 
default. Let us know and we'll work things out.
The mid-November deadline is not for removing *all* of Python 2, but for 
getting exceptions.


If you are already working to port to Python 3, sorry for the spam! But, 
time is running out. Consider getting an exception so the package isn't 
removed in November. Or remove the package (if nothing depends on it) 
and re-introduce it when it's ported. (We'll be happy to help, with 
reviews or otherwise.)
If you want to remove the Python 2 package, but you're waiting for 
something that depends on it, please make sure you know what the 
maintainer(s) of the dependent package are planning to do. (Hopefully 
you have fewer dependents than "python27" or "python2-setuptools", and 
your questions can be more targeted than this e-mail.)
If you want to continue using Python 2, please let us know ASAP. We can 
guide you through filing the FESCo exception.


If you talked to us (on e-mail or Bugzilla) and think your package is 
fine as it is, but you don't have a FESCo exception (or a request for 
one), then there was a misunderstanding. We're sorry for our side of it. 
Please get a FESCo exception for your package.


Note: Packages that BuildRequire python27, and have no other Python 2 
dependencies, have a blanket exception for Fedora 32:

https://pagure.io/fesco/issue/2250
They aren't listed below.
Also note: Orphaned Python 2 packages also aren't listed below, but will 
be removed unless someone adopts them and gets an exception.



## Why the FESCo exception?

It's a bit of process meant to ensure all the relevant people know that 
a package *and its dependencies* will stay in Fedora, despite using an 
(upstream-)unmaintained interpreter.
With hundreds of packages, all different, we unfortunately do need a bit 
of bureaucracy.
Due to the volume, the Python 2 removal will be *automated*, so if you 
have an "obvious exception", it's still important to let *humans* know, 
so they can adapt their automation.



## Why now?

The current maintainers of python27 don't want to maintain it forever, 
without upstream support. But we also don't want to just drop it and let 
everything burn, hence all this fuss.
The policy and messaging around deprecating and removing Python 2 has 
been getting stronger and stronger ever since the Python 3.0 release in 
2008.

If you still need more time, please let us know.


## What exactly is happening?

The formal change proposal is here:
https://fedoraproject.org/wiki/Changes/RetirePython2

Packages requiring Python 2 will be removed starting November 15 (unless 
they have an exception).

Components with all essential subpackages removed will be retired.
The removal will be (semi-)automated.

Source packages only BuildRequiring removed packages will fail to build, 
and will be removed according to the regular FTBFS policy.


https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/


## The list

Here is the package breakdown sorted by maintainers.
The list contains the shortest dependency path to Python 2. The arrow 
means "depends on".


The data is based on the latest rawhide compose, so it might be a bit 
out of date.


If you find a bogus dependency, such as a dependency that can be 
resolved in a non-Python 2 way, please let us know, so we can blacklist it.


aarem
  pdf-stapler
(→ PY2)
python2-staplelib (→ PY2)
  python-PyPDF2
python2-PyPDF2 (→ PY2)
  python2-more-itertools
(→ PY2)
abbot
  protobuf
python2-protobuf (→ PY2)
abompard
  python-coverage
python2-coverage (→ PY2)
  python-mako
python2-mako (→ PY2)
  python-pysocks
python2-pysocks (→ PY2)
  python-urllib3
python2-urllib3 (→ PY2)
ajmitchell
  NFStest
(→ PY2)
alexl
  dbus
(BuildRequires: python2-gobject → PY2)
  dbus-python
python2-dbus (→ PY2)
  gnome-python2
gnome-python2-canvas (→ PY2)
gnome-python2-devel (→ PY2)
gnome-python2-gconf (→ PY2)
gnome-python2-gnome (→ PY2)
gnome-python2-gnomevfs (→ PY2)
  gnome-python2-desktop
(→ gnome-python2-canvas → PY2)
gnome-python2-gnomekeyring (→ PY2)
gnome-python2-libwnck (→ PY2)
gnome-python2-rsvg (→ PY2)
alsadi
  dumb-init
(BuildRequires: python2-mock → PY2)
amigadave
  dbus
(BuildRequires: python2-gobject → PY2)
amluto
  python-musicbrainzngs
python2-musicbrainzngs (→ PY2)
anishpatil
  gnome-transliteration
(→ python2-gobject → PY2)
aperezbios
  sugar
(→ PY2)
sugar-cp-* (→ sugar → PY2)
  telepathy-salut
(BuildRequires: python2-dbus → PY2)
apevec
  pyparsing
python2-pyparsing (→ PY2)
  python-distutils-extra
  

Re: Next F31 push?

[Petr Viktorin]

On 2019-10-18 09:18, Leigh Scott wrote:

On Fri, 2019-10-18 at 07:50 +0200, Kevin Kofler wrote:

I mean, in the end it would be self-defeating, because the high chance
that it would introduce more problems would just mean we'd need to
freeze again for longer.



So you get a working ISO for release then break it by releasing untested 
updates  after GA :-(


Untested but fixable. The ISO needs to be usable on its own, but a 
0-day-after-GA update can be amended by 1-day-after-GA update. Or a 
7-day one.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Python 3.8 porting help

[Petr Viktorin]

On 2019-09-26 14:03, Richard Shaw wrote:
On Thu, Sep 26, 2019 at 3:55 AM Petr Viktorin <mailto:pvikt...@redhat.com>> wrote:


Actually, this is in FreeCAD code:

FreeCAD-0.18.3/src/Base/swigpyrun.inl: PyObject *modules =
interp->modules;

Change "interp->modules" to "PyImport_GetModuleDict()" there.


Not being a c/c++ programmer I wasn't sure if it was literally that 
simple. :)


I'll give it a try.


If it doesn't work, please share a COPR that runs into this error.
I tried the change, but (as you warned) I ran into issues that didn't 
look Python-releated.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Python 3.8 porting help

[Petr Viktorin]

Actually, this is in FreeCAD code:

FreeCAD-0.18.3/src/Base/swigpyrun.inl: PyObject *modules = interp->modules;

Change "interp->modules" to "PyImport_GetModuleDict()" there.


On 2019-09-24 16:33, Victor Stinner wrote:

Hi,

It sounds like an issue in SWIG which access directly the
PyInterpreterState structure which became opaque in Python 3.8.

SWIG should use PyImport_GetModuleDict() public function instead, to
access PyImport_GetModuleDict(). In short, it returns interp->modules.

Victor

Le mar. 24 sept. 2019 à 14:57, Richard Shaw  a écrit :


I maintain FreeCAD for Fedora and there appears to have been a change in the 
API for how Python handles thread management in 3.8, specifically 
PyInterpreterState which is giving the the following error when trying to build 
FreeCAD:

In file included from 
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.cpp:41:
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.inl: In function 'void 
Swig_python::cleanupSWIG_T(const char*)':
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.inl:75:31: error: 
invalid use of incomplete type 'PyInterpreterState' {aka 'struct _is'}
75 | PyObject *modules = interp->modules;
   |   ^~
In file included from /usr/include/python3.8/genobject.h:11,
  from /usr/include/python3.8/Python.h:121,
  from 
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/PyExport.h:42,
  from 
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.cpp:25:
/usr/include/python3.8/pystate.h:20:8: note: forward declaration of 
'PyInterpreterState' {aka 'struct _is'}
20 | struct _is;
   |^~~

Any tips?

I can provide a full build log or setup a COPR but it's somewhat difficult 
since I'm building a whole new stack of some dependencies:

Coin3->Coin4 (in review)
- Which SoQt, SIMVoleon, OpenSceneGraph and Pivy need to be ported to.
Pyside->PySide2 (which implies Qt4->Qt5)

Thanks,
Richard
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Python 3.8 porting help

[Petr Viktorin]

Victor, do you know the answer here?

Blender has the same issue: 
https://bugzilla.redhat.com/show_bug.cgi?id=1734980

(but I forgot to CC you there)


On 2019-09-24 14:57, Richard Shaw wrote:
I maintain FreeCAD for Fedora and there appears to have been a change in 
the API for how Python handles thread management in 3.8, specifically 
PyInterpreterState which is giving the the following error when trying 
to build FreeCAD:


In file included from 
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.cpp:41:
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.inl: In function 
'void Swig_python::cleanupSWIG_T(const char*)':
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.inl:75:31: 
error: invalid use of incomplete type 'PyInterpreterState' {aka 'struct 
_is'}

    75 |     PyObject *modules = interp->modules;
       |                               ^~
In file included from /usr/include/python3.8/genobject.h:11,
                  from /usr/include/python3.8/Python.h:121,
                  from 
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/PyExport.h:42,
                  from 
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.cpp:25:
/usr/include/python3.8/pystate.h:20:8: note: forward declaration of 
'PyInterpreterState' {aka 'struct _is'}

    20 | struct _is;
       |        ^~~

Any tips?

I can provide a full build log or setup a COPR but it's somewhat 
difficult since I'm building a whole new stack of some dependencies:


Coin3->Coin4 (in review)
- Which SoQt, SIMVoleon, OpenSceneGraph and Pivy need to be ported to.
Pyside->PySide2 (which implies Qt4->Qt5)

Thanks,
Richard

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Notes from the "Fedora Loves Python" session at Flock

[Petr Viktorin]

Hello!
I ended up running the "Fedora Loves Python" session at Flock. As the 
abstract says:


> It seems that the Fedora ❤ Python initiative has recently become more
> stagnated. It's not that Fedora no longer loves Python, but the
> relationship has become rather boring, after several years
> of marriage. I'd like to figure out, with people who understand
> metrics and marketing more than we do, what to do next.

(or alternatively: We love the F♥P stickers, but the sticker budget 
people want reasons to print more!)


This session was not recorded. (Ithers were, and the recordings might be 
available later.)


Here are some rough notes for anyone who'd like to run through them. 
I'll most likely end up prioritizing and fleshing out some of the ideas 
later.

Please ask if something interesting has too little info :)



Fedora Loves Python discussion at Flock Budapest 2019

• Introduction round

• Mirek Suchy’s suggestion
• `setup.py bdist_rpm` produces ugly results. Even though we have pypa2rpm, 
most people don’t know about it. Let’s improve this
• Petr’s response: that part of setuptools can’t really be improved, it’s 
going to be dismantled piece by piece into different projects
• pyp2rpm is unmaintainable, it does too much guessing. We’re trying to 
move to pyproject macros, which are standard based

• Discussion about single-specfile vs. different specfiles for Fedora/EPEL/etc.
• Petr recommends different specfiles, not merging branches, at most 
cherry-picking commits.
• Especially if it’s hard to coordinate between Fedora and EPEL, it’s 
better to have different specfiles.
• You want EPEL stable, so you generally won’t cherry-pick that many 
changes from Fedora anyway.

• Questions to answer from the sticker budget person:
“What would help me help here is an engineering understanding of (subject 
to revision and extension): ”
“a) why is working in Python a better experience than other platforms;”
“b) what changes/configs/etc. does Fedora do that makes this better (in an 
edition or spin or whatever);”
• Fedora does better than other distros because it tries to NOT MODIFY the 
upstream Python. Fedora tries to work upstream and do the reasolable changes 
there and not downstream in Fedora. Other distros by and large don’t do this.
• We should really market this more (and better). Offering the upstream 
experience is a lot of work.

“c) do we have docs on the definitive way to work with python to build an 
app for deployment (I’m thinking about system packages vs pypi kind of stuff).”
• Petr: Should we even have this?
• Neal Gompa: Yes, for example if you have crypto in your module, a 
system module is better
• Neal: Also talk about Containers, very important
• Petr: We need to figure out Fedora package names as dependencies upstream
• Tomas Orsava: Having the documentation on how to build an app would help 
us with newcomers
• Petr: Yes, if we want the stickers we should have that docs. Who’s 
gonna write it?
• Mention also the Fedora Python Tox container

• Neal Gompa shares why he likes Python on Fedora
• In Fedora it’s way easier to test Python code than on other distros: It 
has *working* PyPy, micropython, tox, Python 2, new Python 3 and also really 
old versions of Python 3 - it’s really easy to test on everything at the same 
time
• Fedora uses Python for its infrastructure

• Meta discussion about the session: We need marketing people to help us 
promote Fedora Loves Python, but this session has only engineers.

• Lumir Balhar’s idea: We have Fedora Python Tox container, which as Tox 
pre-loaded and ready to test all versions
• https://github.com/frenzymadness/fedora-python-tox
• To be moved to fedora-python project on github soon
• Usable for various testing and also on a CI pipeline
• Petr’s suggestions: Make it run with podman, make it into a GitHub app

• Pavol Babincak’s idea: Have axillary modules packaged for every available 
Python version in Fedora, e.g. rpm, dnf bindings that you can’t get from pypi
• Neal Gompa says it would take too much maintenance, building gets super 
long, you have to install so many Python versions and their developer packages, 
it would really slow down development
• Petr suggests to package these packages (even bindings) on PyPI. It is 
not straight forward, but it’s doable, and we can try to even improve it.
• Important: Libvirt for non-default Python
• autotools is a problem in this regard, meson (autotools replacement) is 
starting to be a problem

• Neal Gompa: Jython is in danger because Java is exploding.
• Petr: Does anybody actually use Jython? [crickets]
• Lumir: Even Pytest doesn’t work with Jython

• Petr: PyPI problem - anybody can upload wheels to it and there’s no way to 
check it actually corresponds to the published sources
• Petr: Do we want to build wheels?
• Neal: - In RPM? No point, just harder to 

Re: Why retire Python 2 packages and games that still work to end user ?

[Petr Viktorin]

On 8/11/19 2:28 PM, Kevin Kofler wrote:

Miro Hrončok wrote:

We are still planning to maintain the interpreter. As is documented in the
change. So can we please stop arguing about maintaining the interpreter
over and over? It is staying and our team will maintain it at least until
RHEL 7 EOL, possibly longer.


Then why do you require all this FESCo exception bureaucracy, including a
Python 3 migration plan, for every single package requiring Python 2, even
if it is only the interpreter (and the shipped standard library)?


Python 2 will not get the attention, fixes, and security updates that 
people have been used to in the past decade. That's a big deal, and 
unfortunately we know no good way to properly communicate this to all 
the affected packagers.
We hope the bureaucracy works for the hundreds of packages with inactive 
maintainers; the flip side is that the active maintainers do have to do 
some more work, even if it's just filing a ticket and switching a 
BuildRequires. Sorry for that.


As for the Python 3 migration plan: we can agree to maintain Python 2 
for you to depend on, if there's a feasible plan of moving away from 
Python 2. If there's no plan, you'll just run into trouble in a few more 
years. If you consider your package important, we really want to know 
about the situation.


The Python 3 migration plan is not a requirement, but a very common and 
useful piece of information that we want to hear about. Every package is 
different; we can't know where each upstream does their planning. As a 
packager, you know best where to ask for this info, so we ask you to 
find it and paste the link in a Fedora place -- Bugzilla or a FESCo ticket.
If there's another plan instead of a "Python 3 porting plan" (like port 
to Rust instead, or to Tauthon, or maintain Python 2 forever), we'd also 
like to know about it.




Yes the concern is about maintaining the python2-* packages.

The difference between EL and Fedora is that package sin Fedora get recent
rebases / updates to newer version. You cannot just package them and call
it "done". And most of the upstreams are coming to a point where you
cannot update the Python 2 package because the new version doe snot
support Python 2.

That at the end means you need to go over nonobvious bumps to split the
Python 2 package out of the "common" package, in order to update the
"common" (now Python 3 only) package. And this problem spreads further
with dependencies (even transitive ones). I don't have the energy or time
to split hundreds of packages and maintain a separate stack of Python 2
packages. If somebody else has, go for it.


And that is a perfectly valid reason to orphan the Python 2 parts. My issue
with the policy you proposed and FESCo approved for F31 is that it does not
require this to be the case, but allows maintainers to drop the Python 2
subpackage just because they don't like Python 2 (or simply want to avoid
going through the bureaucracy you're requiring for F32), which is a quite
antisocial thing to do.

If the partial orphaning policy were limited to packages where upstream
dropped or is in the process of dropping Python 2 support, I would not
object.


The policy is for those *and their dependencies*.

See, I have a "package where upstream is in the process of dropping 
Python 2 support". It's called "python2".
I could just drop it and call it a day, but since people still want it, 
I'll maintain it. But only for those who care and who understand the 
situation; not for the hundreds of inactive maintainers.
(To be clear: making it possible and straightforward to build Fedora 
RPMs with python3 *is* more work than just maintaining the interpreter 
for users.)



That said, those are also the cases where the split out python2-* package
WOULD in fact be "done" and never require getting updated to a newer
version, unless upstream maintains some legacy/LTS branch. So the situation
for whomever ends up stuck maintaining the python2-* package would not be
that different from the RHEL situation.


Sure, but this "python2" one does need ongoing maintenance.


The set of python2-* packages to remain is determined by the FESCo
exceptions. It is fairly easy really: We don't have the necessary
information to understand what Python packages are "important" and what
are "cruft". Hence if your package is important, you just need to explain
that. Obviously, if you need to maintain the package as a dependency, for
another important package, the exception can be requested at once, you
don't need to request dozens of exception to keep e.g. chromium around.


But why does this have to go through FESCo and a formal approval process?
Why is it not sufficient that the maintainer says "yes, this is still
needed"? If a maintainer wants to keep the package, this clearly means it is
important to SOMEBODY, so why do we need an approval by committee to confirm
this (or worse, veto this against the wishes of the maintainer)?


The maintainer needs to say "yes, 

Orphaned python2-django1.11

[Petr Viktorin]

Hello,
Back when [Django 2.0] was released in Fedora 28, I took over Django 
1.11 LTS as some important (to me) packages depended on it. I'm no 
longer interested in maintaining it, so I've orphaned it.
Let me know if you want to take it. If no one does, it may be removed 
from Rawhide in 6 weeks.


Depending packages and their maintainers are:
- fts-monitoring (andreamanzi, simonm)
- python-oauth2client (mbaldessari, ralph)
- cobbler-web (jimi, kwizart, orion, shenson)

Let me know if you need help with these packages.


[Django 2.0]: https://fedoraproject.org/wiki/Changes/Django20
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Don't lint in %check (Was: python-flake8 package not available in F30)

[Petr Viktorin]

On 3/9/19 7:33 PM, Chris wrote:

Thank you both for your fast reply!

 >  Why do you need to BuildRequire a linting tool? What are you trying 
to achieve?


I just use python-flake8 as an OCD way of having my build fail if i fail 
pep8 :)  It's just used in conjunction of my unit tests.


Running a linter is fine upstream, but I'll argue that you'll be much 
better off disabling it for the distro build.


Newer versions of flake8 can cause your tests to suddenly fail. We've 
seen that happen relatively often -- style standards themselves evolve, 
and the way they're codified in automatic tools evolves.


Upstream, please do check code style or unused imports if that's your 
thing. But after you cut a release (on GitHub/PyPI), linting the code 
doesn't really bring any benefit. (Unlike running the test suite, of 
course.)


Even if you're currently maintaining this both upstream and in Fedora, 
and have the energy to watch for & fix any failures, after some years 
you might want to pass the package on to someone else. Be nice to your 
successor. And be nice your potential Debian (et al.) maintainers by 
making your specfile show how to package for a distro :)



(Note: AFAIK this is not official Fedora policy; please 
disagree/discuss/argue.)

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Python BuildRequires Generators (Re: F31 System-Wide Change proposal)

[Petr Viktorin]

On 2/22/19 2:39 PM, Neal Gompa wrote:

On Fri, Feb 22, 2019 at 6:01 AM Petr Viktorin  wrote:


On 2/18/19 9:19 PM, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/BuildRequires_Generators

= BuildRequires Generators =

== Summary ==
Add possibility to generate build-time dependencies within RPM spec
file and teach RPM and mock how to handle this.



I am very excited about this change. Count with me for Python. Is there
a (draft) API to start coding against?

In Python, there's an effort to enable things like this generator, e.g.
[PEP 517].

Here, gathering BuildRequires is a two-step process:

- the "build backend" is specified in declarative metadata;
- then the build backend can query backend-specific metadata for
additional BuildRequires.

Is that possible with the current plans?


BTW: New versions of Python's usual build system (setuptools/setup.py)
will be forward compatible with PEP 517. Please do not write generators
that use setup.py directly.


We currently have no BuildRequires generators written yet, so we can
Do It Right(TM) for this. :)

That said, I'm not sure how we will be broadly compatible for divining
setup_requires if they are specified...


No divining in the generator, please.
We should use only the (newly) standardized interfaces, PEP 518, 517, 
etc. Issues with those should be solved upstream (or patched 
temporarily, with a bug filed).
Nothing in the generator should be Setuptools-specific (except using 
Setuptools as the default build backend, for backwards compatibility 
reasons).
setup_requires (and all of setup.py) are Setuptools-specific, so 
Setuptools is responsible for all the divining and for making the info 
available through standardized interfaces.

And yes, this aligns with Setuptools upstream plans.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F31 System-Wide Change proposal: BuildRequires Generators

[Petr Viktorin]

On 2/22/19 12:01 PM, Petr Viktorin wrote:

On 2/18/19 9:19 PM, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/BuildRequires_Generators

= BuildRequires Generators =

== Summary ==
Add possibility to generate build-time dependencies within RPM spec
file and teach RPM and mock how to handle this.



I am very excited about this change. Count with me for Python. Is there 
a (draft) API to start coding against?


In Python, there's an effort to enable things like this generator, e.g. 
[PEP 517].


Missing footnote:
[PEP 517]: https://www.python.org/dev/peps/pep-0517/
specifically: 
https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel




Here, gathering BuildRequires is a two-step process:

- the "build backend" is specified in declarative metadata;
- then the build backend can query backend-specific metadata for 
additional BuildRequires.


Is that possible with the current plans?


BTW: New versions of Python's usual build system (setuptools/setup.py) 
will be forward compatible with PEP 517. Please do not write generators 
that use setup.py directly.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F31 System-Wide Change proposal: BuildRequires Generators

[Petr Viktorin]

On 2/18/19 9:19 PM, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/BuildRequires_Generators

= BuildRequires Generators =

== Summary ==
Add possibility to generate build-time dependencies within RPM spec
file and teach RPM and mock how to handle this.



I am very excited about this change. Count with me for Python. Is there 
a (draft) API to start coding against?


In Python, there's an effort to enable things like this generator, e.g. 
[PEP 517].


Here, gathering BuildRequires is a two-step process:

- the "build backend" is specified in declarative metadata;
- then the build backend can query backend-specific metadata for 
additional BuildRequires.


Is that possible with the current plans?


BTW: New versions of Python's usual build system (setuptools/setup.py) 
will be forward compatible with PEP 517. Please do not write generators 
that use setup.py directly.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F31 System-Wide Change proposal: BuildRequires Generators

[Petr Viktorin]

On 2/19/19 7:04 PM, Raphael Groner wrote:

Hi,

maybe you already noticed, there's a project called pipreqs that parses python 
code for import statements. We've already a review request:
https://bugzilla.redhat.com/show_bug.cgi?id=1665749

Although, I doubt in general that any automagic with dependency generators 
brings a huge benefit in the long run. In case of python projects, I see 
sometimes conditional dependencies that enable optional features by awareness 
of any existance of a library, e.g. SecretStorage that parses for alternative 
desktops and optional password storage. Further, mostly there are up to 5 
dependencies to note but sometimes 2 to ignore anyways.


Yes, automagic generators don't work in the long run. Requirements are 
always better listed explicitly.
However, for most Python projects they *are* listed explicitly -- in the 
Python package metadata. And there are efforts upstream to make this 
metadata more usable for automation.


Currently, that metadata needs to be copied, by hand, to RPM 
BuildRequires. That's what a generator can automate.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: pypy

[Petr Viktorin]

Hi,

On 2/19/19 3:14 PM, Antonette Caldwell wrote:

Hi all

I commented on https://bugzilla.redhat.com/show_bug.cgi?id=1673127 that 
I will be working on building pypy. So far, I was able to build pypy and 
pypy3 on rawhide, but this is my first time creating the package. I 
copied the spec from both of the rpms/pypy packages[1], but I wanted to 
know if I will have to create new spec, since I can see there are 
changes between the 6.0.0 and 7.0.0 versions.


No, this is an update. Don't create a new package.
If you are in the packagers groups, you can open a pull request on Pagure:
- https://src.fedoraproject.org/rpms/pypy
- https://src.fedoraproject.org/rpms/pypy3

If not, you can simply attach the updated specfile to the Bugzilla.



Here's what I have done so far from building pypy on rawhide

pypy
A build of pypy (with jit) on x86_64
[Timer] Timings:
[Timer] annotate                       ---  990.8 s
[Timer] rtype_lltype                   --- 1797.3 s
[Timer] pyjitpl_lltype                 --- 2112.1 s
[Timer] backendopt_lltype              ---  400.0 s
[Timer] stackcheckinsertion_lltype     ---  995.6 s
[Timer] database_c                     ---  907.0 s
[Timer] source_c                       ---  413.2 s
[Timer] compile_c                      --- 1274.7 s
[Timer] build_cffi_imports             ---   16.7 s
[Timer] ===
[Timer] Total:                         --- 8907.5 s

A build of pypy without jit on x86_64
[Timer] Timings:
[Timer] annotate                       --- 1233.1 s
[Timer] rtype_lltype                   --- 1554.2 s
[Timer] backendopt_lltype              ---  407.9 s
[Timer] stackcheckinsertion_lltype     ---  147.9 s
[Timer] database_c                     ---  698.6 s
[Timer] source_c                       ---  304.9 s
[Timer] compile_c                      ---  700.5 s
[Timer] build_cffi_imports             ---   10.2 s
[Timer] ===
[Timer] Total:

pypy3
A build of pypy with jit on x86_64

[translation:info] usession directory: /tmp/usession-py3.5-0
[Timer] Timings:
[Timer] annotate                       ---  260.5 s
[Timer] rtype_lltype                   ---  342.1 s
[Timer] pyjitpl_lltype                 ---  560.0 s
[Timer] backendopt_lltype              ---  106.2 s
[Timer] stackcheckinsertion_lltype     ---   53.4 s
[Timer] database_c                     ---  232.3 s
[Timer] source_c                       ---  101.7 s
[Timer] compile_c                      ---  547.3 s
[Timer] build_cffi_imports             ---   17.9 s
[Timer] ===
[Timer] Total:                         --- 2221.5 s

Please let me know if I need to include devel in this email as well, 
since I am new and I have no built an rpm package, so I will not be able 
to push the new package once it is done.


python-devel is fine for this conversation :)
It might, sadly, take some time before you get a reply. But we're here; 
let us know if you need any help :)



[1] https://src.fedoraproject.org/rpms/pypy/blob/master/f/pypy.spec

Antonette Caldwell


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Python packages with extras dependencies

[Petr Viktorin]

On 2/5/19 12:44 AM, Eli Young wrote:

Python packages can specify extras dependencies, which are sets of dependencies not 
required for core functionality, and which generally correspond to some feature. These 
can then be specified by downstream consumers of the package. For example, requests has 
an entry in extras called security[1], which currently adds requirements of python 
packages pyOpenSSL >= 0.14, cryptography >= 1.3.4, and idna >= 2.0.0. A 
downstream consumer that wants to use this would add a dependency on requests[security].

 From what I can tell, the current practice in Fedora packaging is to ignore 
these. This simplifies packaging Python modules that have extras specified, but 
ultimately pushes the specification of those dependencies down into every 
consumer of the package, whether users or other packages.

As an example of this, I currently maintain the python-dns-lexicon package, 
which provides a common CLI and API for various different DNS providers. Some 
of the providers have additional dependencies that are necessary to function, 
and which are specified as extras. The Plesk provider, for example, also 
requires python-xmltodict[2]. In line with what appears to standard practice, 
extra dependencies are not currently installed with the broader 
python-dns-lexicon package. If, however, a user or dependent package wants to 
utilize the Plesk functionality of python-dns-lexicon, they now need to know 
that python-xmltodict needs to be installed, and will need to check whenever 
the package updates as to whether or not that has changed.

How should we be handling this? Right now, it seems that most packages follow 
this behavior of punting on the responsibility to package consumers. Should 
this continue? If not, how should we handle things? Should we just include all 
extras dependencies in the parent package? Alternatively, should we have 
dummy/meta subpackages for extras that require the parent package as well as 
any extras dependencies (e.g. python-dns-lexicon-plesk would require 
python-dns-lexicon and python-xmltodict)?

[1]: https://github.com/requests/requests/blob/v2.21.0/setup.py#L105
[2]: https://github.com/AnalogJ/lexicon/blob/v3.0.6/setup.py#L101



Hello,

AFAIK, there are currently no official guidelines for Python extras, and 
there's some fame & glory waiting if you'd like to help draft them :)


That said, I believe subpackages are the answer here.
In addition you could use weak dependencies. I think the main package 
should have "Suggests:" for all the extras, and even "Recommends:" for 
ones that are almost always useful.
I believe the cost of extra dependencies is lower in Fedora than on 
PyPI. Some common reasons for punting deps to extras don't apply (like 
bootstrapping issues or requiring compiled modules from pure-Python 
ones), so sometimes it's good to just go for hard Requires.


Purely as a packager, I'd like to see "boolean Provides", something like:
Provides: (python3dist(dns-lexicon[plesk]) if python3dist(xmltodict))
... but at a chat with RPM devs on the last Flock, I learned that is not 
feasible.


The Python SIG (python-devel@lists.fedoraproject.org) would be a good 
place to discuss specific details.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Python packages with extras dependencies

[Petr Viktorin]

On 2/5/19 12:44 AM, Eli Young wrote:

Python packages can specify extras dependencies, which are sets of dependencies not 
required for core functionality, and which generally correspond to some feature. These 
can then be specified by downstream consumers of the package. For example, requests has 
an entry in extras called security[1], which currently adds requirements of python 
packages pyOpenSSL >= 0.14, cryptography >= 1.3.4, and idna >= 2.0.0. A 
downstream consumer that wants to use this would add a dependency on requests[security].

 From what I can tell, the current practice in Fedora packaging is to ignore 
these. This simplifies packaging Python modules that have extras specified, but 
ultimately pushes the specification of those dependencies down into every 
consumer of the package, whether users or other packages.

As an example of this, I currently maintain the python-dns-lexicon package, 
which provides a common CLI and API for various different DNS providers. Some 
of the providers have additional dependencies that are necessary to function, 
and which are specified as extras. The Plesk provider, for example, also 
requires python-xmltodict[2]. In line with what appears to standard practice, 
extra dependencies are not currently installed with the broader 
python-dns-lexicon package. If, however, a user or dependent package wants to 
utilize the Plesk functionality of python-dns-lexicon, they now need to know 
that python-xmltodict needs to be installed, and will need to check whenever 
the package updates as to whether or not that has changed.

How should we be handling this? Right now, it seems that most packages follow 
this behavior of punting on the responsibility to package consumers. Should 
this continue? If not, how should we handle things? Should we just include all 
extras dependencies in the parent package? Alternatively, should we have 
dummy/meta subpackages for extras that require the parent package as well as 
any extras dependencies (e.g. python-dns-lexicon-plesk would require 
python-dns-lexicon and python-xmltodict)?

[1]: https://github.com/requests/requests/blob/v2.21.0/setup.py#L105
[2]: https://github.com/AnalogJ/lexicon/blob/v3.0.6/setup.py#L101



Hello,

AFAIK, there are currently no official guidelines for Python extras, and 
there's some fame & glory waiting if you'd like to help draft them :)


That said, I believe subpackages are the answer here.
In addition you could use weak dependencies. I think the main package 
should have "Suggests:" for all the extras, and even "Recommends:" for 
ones that are almost always useful.
I believe the cost of extra dependencies is lower in Fedora than on 
PyPI. Some common reasons for punting deps to extras don't apply (like 
bootstrapping issues or requiring compiled modules from pure-Python 
ones), so sometimes it's good to just go for hard Requires.


Purely as a packager, I'd like to see "boolean Provides", something like:
Provides: (python3dist(dns-lexicon[plesk]) if python3dist(xmltodict))
... but at a chat with RPM devs on the last Flock, I learned that is not 
feasible.


The Python SIG (python-de...@lists.fedoraproject.org) would be a good 
place to discuss specific details.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Python setuptools and files to be installed in the installation prefix

[Petr Viktorin]

On 10/2/18 10:06 PM, Miro Hrončok wrote:

On 2.10.2018 16:30, Federico Bruni wrote:

Hi folks

While trying to figure out a way to install some desktop specific 
files (such as APP.desktop or APP.appdata.xml) within the installation 
prefix (instead of python installation) using setuptools, I started 
wondering if it's actually a good idea.


AFAIK setuptools (PyPA) considers this a bad practice and doesn't (want 
to) support this. It's up to the Linux package maintainers to do this.


I don't remember where this information comes from, but I'm 90% sure 
it's accurate.


Peter and/or Nick might know more. Cced.


If you use Python's upstream packaging mechanisms, the expectation is 
that the result is installable into a self-contained environment like 
venv, Conda, or a user's home.

To install these files system-wide, put it in the RPM spec.

It's probably possible to do this correctly from setuptools, but that 
would put you in the business of figuring out how to do it correctly 
from setuptools. I don't think you want that :)

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

python3-debug: Should we use -Og rather than -O0?

[Petr Viktorin]

Hi,
I've been looking at documentation of these GCC flags:

-O0
Reduce compilation time and make debugging produce the expected 
results. This is the default


-Og
Optimize debugging experience. -Og enables optimizations that do 
not interfere with debugging. It should be the optimization level of 
choice for the standard edit-compile-debug cycle, offering a reasonable 
level of optimization while maintaining fast compilation and a good 
debugging experience.


(from https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html)


Currently, we use -O0 for the debug build of Python, but it seems that 
-Og would actually be a better option. Is there something I'm missing?

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Future of Python 2.6 and Jython in Fedora

[Petr Viktorin]

On 9/6/18 5:32 PM, Miro Hrončok wrote:
In Fedora, we advertise "all the Pythons" are available and developers 
may use them with venv, virtualenv, tox.


https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.html 



Two of the interpreters we ship are quite problematic in this regard.

### Python 2.6

  - virtualenv 16.x dropped support for 2.6 and we dirty patched it back
  - tox 3 dropped support for 2.6
  - pip/setuptools dropped support for 2.6 and our patched virtualenv 
uses an older version of pip and setuptools to make work


However, I know there are developers who use Python 2.6 in Fedora 
(namely because their software maintains RHEL 6 compatibility).


I propose that we indicate on Fedora Developer Portal that Python 2.6 is 
provided as is and that it may not work with virtualenv/tox. We maintain 
the current patch of virtualenv at least for the lifetime of Fedora 29, 
but if an upgrade of virtualenv is done, we ditch that patch from Fedora 
30+.


We also mark it deprecated and schedule the removal for EPEL 6 EOL 
(AFAIK late 2020).


### Jython

  - virtualenv somehow works but is not officially supported
  - pip is not supported and has hesisenbugs

I know nobody who uses Jython with virtualenv and tox.

I propose that we indicate on Fedora Developer Portal that Jython is 
provided as is and that it may not work with virtualenv/tox.


(No scheduled removal.)

Thoughts?


Sounds reasonable. I just think before any code changes, we should also 
add a section to the portal on how to use virtualenv/tox with the old 
versions. The workaround is not pretty, but people shouldn't need to 
rediscover it.



___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Heads Up: python2 is marked as deprecated

[Petr Viktorin]

On 09/11/18 11:52, Tomasz Kłoczko wrote:
On Tue, 11 Sep 2018 at 16:48, Petr Viktorin <mailto:pvikt...@redhat.com>> wrote:

[..]

These numbers hide an important fact: many things currently come with
RPMs for both python2 and python3.

More detailed statistics (based on SRPMs, not binary RPMs):
http://fedora.portingdb.xyz/
Graph with historical data: http://fedora.portingdb.xyz/history/

 > In other words trying currently announcing python2 as depricated
is at
 > least a bit .. odd.

Do you know a better way to make the python2 numbers go down?

Would *you* be interested in maintaining python2 past 2020, with no
upstream support and 3415 dependent packages?


No and no .. of course :)
I've been only trying to say that with current numbers about balance 
between python 2 and 3 packages are making announcement about 
deprecation a bit to early. Only this and nothing more :P


When will it not be too early?


I fully understand effort to migrate ASAP to python 3.
IMO it should be announced only kind of call to migrate as much as 
possible with completing set of advises abut typical porting issues.


Python 3 is now almost 10 years old. The "Python 3 as Default" change 
was for Fedora 21. Python's documentation has a section on porting. 
There are printed books about porting. There's a comprehensive guide at 
portingguide.readthedocs.io. Fedora's Python packaging guidelines say 
"If it supports only python2 then [...] upstream SHOULD be contacted and 
encouraged to rectify this issue." -- have you done that?


We're honestly tired of making calls to migrate. What would another one 
of those accomplish?


Forming ad hoc team people which could help porting code to python 3 may 
IMO be useful.


Try the Python SIG!

I have in my set of packages one of those which will require migration 
to python 3 as well.


Why have you waited so long?


It is mc package which has amazon s3 vfs backend as and
I'm thinking about create mc-s3 (or mc-vfs-s3) subpackage to separate 
python2 dependent parts.
Theoretically as now aws-cli AFAIK is already ported to python 3 it 
should be easy however I haven't now access to AWS systems to test s3 
bucket access over mc.
Other possible option may be rewrite s3 vfs backent to use some other 
(non python based script .. and backend tool).


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Heads Up: python2 is marked as deprecated

[Petr Viktorin]

On 09/10/18 11:26, Tomasz Kłoczko wrote:

Just FTR current numbers about packages which requires python3 vs python2:

# dnf repoquery -C --whatrequires python3| grep -cv i686; dnf repoquery 
-C --whatrequires python2| grep -cv i686

Last metadata expiration check: 0:59:03 ago on Mon 10 Sep 2018 18:23:45 BST.
3034
Last metadata expiration check: 0:59:09 ago on Mon 10 Sep 2018 18:23:45 BST.
3415


These numbers hide an important fact: many things currently come with 
RPMs for both python2 and python3.


More detailed statistics (based on SRPMs, not binary RPMs): 
http://fedora.portingdb.xyz/

Graph with historical data: http://fedora.portingdb.xyz/history/

In other words trying currently announcing python2 as depricated is at 
least a bit .. odd.

kloczek


Do you know a better way to make the python2 numbers go down?

Would *you* be interested in maintaining python2 past 2020, with no 
upstream support and 3415 dependent packages?

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 30 Self-Contained Change proposal: Make ambiguous python shebangs error

[Petr Viktorin]

On 08/31/18 13:41, Miro Hrončok wrote:

On 31.8.2018 13:30, Miro Hrončok wrote:

On 31.8.2018 09:05, Igor Gnatenko wrote:
On Wed, Aug 22, 2018 at 10:02 PM Ben Cotton > wrote:


https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error 



    == Summary ==
    The /usr/lib/rpm/redhat/brp-mangle-shebangs buildroot
    policy script will be changed to make the build fail when it sees an
    ambiguous python shebang, such as #!/usr/bin/python or
    #!/usr/bin/env python. (The script has been warning in
    these cases for 2 Fedora releases already, saying ''This will become
    an ERROR''.)


Why can't we add something like 
`MANGLE_UNVERSIONED_PYTHON_TO=%{__python3}`?



What about the following:

If %_mangle_python_shebnags_to is set, the script will mangle to that 
(unless it's unversioned python again or otherwise invalid).


If %_mangle_python_shebnags_to is not set (the default), the script 
will error (as currently proposed).


Hmm... Given that upstream PEP 394 allows unversioned "python" for 
py2/py3 straddling code, maybe the name could be even shorter?
Maybe what we need is just a way to say the package(r) is aware of 
python3 and the latest (as of now) upstream recommendations. With that 
info, mangling "/usr/bin/env python" to "/usr/bin/python3" is the right 
thing to do.


Probably %_mangle_unversioned_python_shebnags_to becasue that's the only 

   ^^
BTW, if you copy/paste this, be aware there^^ is a typo :)


thing that would be changed.



___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Why are we shipping debug builds of pythons?

[Petr Viktorin]

On 08/05/18 14:01, Jan Kratochvil wrote:

On Sun, 05 Aug 2018 13:39:58 +0200, Charalampos Stratakis wrote:

Here we are referring to the python2-debug or python3-debug builds, which
are just extra python builds that are compiled with the --with-pydebug flag


Not just --with-pydebug:

--
%if %{with debug_build}
BuildPython debug \
   "--without-ensurepip --with-pydebug" \
   "-O0"
%endif # with debug_build

BuildPython optimized \
   "--without-ensurepip %{optimizations_flag}" \
   ""
--

Many packages have their *-debug variants. And then some packages (like
kernel) switch even their standard builds to debug mode but only in Rawhide.
Despite all that the *-debug package one occasionally needs is then missing
and one still has to build it locally.

That is all together messy. This is why Microsoft has their debug build of
their whole OS - Windows - called Checked Build:

https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/checked-and-free-build-differences


The main issue with --with-pydebug is that it changes the ABI, i.e. all 
extensions need to be re-built specifically for it (and debug extensions 
outside the standard library aren't usually packaged in Fedora). That 
makes it much less useful than if it just used less optimizations.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/4E32NGEBSVXDBVFP5UOMSKADV3EU2GAU/

Re: Why are we shipping debug builds of pythons?

[Petr Viktorin]

On 08/05/18 14:01, Jan Kratochvil wrote:

On Sun, 05 Aug 2018 13:39:58 +0200, Charalampos Stratakis wrote:

Here we are referring to the python2-debug or python3-debug builds, which
are just extra python builds that are compiled with the --with-pydebug flag


Not just --with-pydebug:

--
%if %{with debug_build}
BuildPython debug \
   "--without-ensurepip --with-pydebug" \
   "-O0"
%endif # with debug_build

BuildPython optimized \
   "--without-ensurepip %{optimizations_flag}" \
   ""
--

Many packages have their *-debug variants. And then some packages (like
kernel) switch even their standard builds to debug mode but only in Rawhide.
Despite all that the *-debug package one occasionally needs is then missing
and one still has to build it locally.

That is all together messy. This is why Microsoft has their debug build of
their whole OS - Windows - called Checked Build:

https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/checked-and-free-build-differences


The main issue with --with-pydebug is that it changes the ABI, i.e. all 
extensions need to be re-built specifically for it (and debug extensions 
outside the standard library aren't usually packaged in Fedora). That 
makes it much less useful than if it just used less optimizations.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4E32NGEBSVXDBVFP5UOMSKADV3EU2GAU/

Re: Python Bundled Wheels package

[Petr Viktorin]

On 07/22/18 11:49, Miro Hrončok wrote:

Hi Pythonistas and Fedora packagers.

Recently I've realized we bundle too much wheels with our pythons + 
virtualenv package. That is unfortunate:


  * we don't build those. stricly seeking, it's just a zip with python 
files in it, yet this is not permitted in Fedora
  * we only sometimes list it as Provides: bundled(...) and when we do, 
it is tedious

  * we never list bundled deps in those bundled wheels (pip bundles a lot)
  * we never adapt the license tag to include license of bundled wheels 
(and bundled deps in those) - it would be even more tedious, as pip 
License tag can be very complicated

  * we duplicate those across packages

I went ahead and prepared a concept in:
https://bugzilla.redhat.com/show_bug.cgi?id=1605156

This is one package that builds all the required wheels. It might be a 
bit weird that it's only one package, but I think it can lower the 
maintenance burden. Also, we won't update any wheel package, we only add 
or remove them, so there is no "life cycle". Later we can decide that 
there are simply too many thing sin one package and split it).


This package makes sure the license tag is right and all the virtual 
bundled provides are in place.


Even as one package, I think it's a big improvement comparing to current 
state of things.


Could you please review the decisions made in the spec? Namely:

  * naming (main package, subpackages)
  * virtual provides
  * that the spec is generated by a script and how that script works
  * the method of usage described in the package review request

I've also decided not to run tests, as for them to mean something, we 
would need to run them against all relevant Python versions. Also, it 
would complicate the package a lot.


The package is approved thanks to Robert-André Mauchin, yet I won't 
request the repo until it's settled that we want this.


Also, once we start using this, maybe we can stop doing rewheel and just 
use wheels from here in the python3 package as well?


Let me ask: do we actually need all those different versions of wheels? 
After all, the first thing I do after creating a venv is usually `pip 
install -U pip`.
Can ensurepip be made to work with "the latest available" pip wheel, 
instead of a hardcoded one?


If that's true only the latest version is needed, we could build the pip 
wheel in the pip package instead.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/3LKTYWCVPEWUPEUWYLVQYULOT4P5ZLCE/

Re: F29 System Wide Change: Move /usr/bin/python into a separate package

[Petr Viktorin]

On 05/24/18 18:53, Randy Barlow wrote:

On 05/24/2018 12:45 PM, Jason L Tibbitts III wrote:

Could you fill us in a bit?  The only thing I know about resultsdb is
that there is a thing called resultsdb.  I went to
https://taskotron.fedoraproject.org/resultsdb/results but I have no idea
how to find, say, all packages which call /usr/bin/python at build time.
A naïve search for /usr/bin/python didn't turn anything up.


Well I didn't say it was easy to do, but there is a test that tasktotron
runs on our packages called dist.python-versions.executables that I
*think* is the one responsible for failing packages that still use
/usr/bin/python. This URL can show results for that particular test:

https://taskotron.fedoraproject.org/resultsdb/results?testcases=dist.python-versions.executables

Another test seems to notice if the package uses /usr/bin/python during
build:

https://taskotron.fedoraproject.org/resultsdb/results?testcases=dist.python-versions.python_usage

So we could look through these to find packages that fail. Of course, a
downside to this approach is that there will only be data for packages
that have been built since these tests existed, so unmaintained packages
might not appear in these results even though they could have problems.
So maybe it's not actually a great way to know the answer to the
question now that I think about it more. Well, except that we did try to
build all packages during the mass rebuild, but then again, many of them
failed to build and still haven't been rebuilt...


That's actually our earlier attempt at solving this problem! :)
It's moderately useful, but it does have its shortcomings:
- It works by grepping the logs, so anything that hides stderr isn't 
detected
- It keeps /usr/bin/python available, so config tools that check whether 
/usr/bin/python *exists* get the wrong view of their environment.
- It's not easy to get a package list out of resultsdb (which is 
probably just a solvable UI problem)

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/BPK7SG666PQLURJCQ2JEC4ZG3ZRQCZVR/

Re: F29 System Wide Change: Move /usr/bin/python into a separate package

[Petr Viktorin]

On 05/24/18 01:01, Miro Hrončok wrote:

On 24.5.2018 00:56, Adam Williamson wrote:

The thing you're missing - which the Change doesn't explicitly state,
so it's understandable - is the effect of having the python2 package
Recommend the package with /usr/bin/python in it.

For *user* systems that will mean that package always gets installed
alongside python2, including that it should get installed on
update/upgrade. They could then manually remove it, but this is the
only way they could get into trouble - by default, both existing and
new installs should have /usr/bin/python. This is because dnf's default
behaviour is to install Recommends/Supplements.

However, for *package builds*, Recommends/Supplements are not
installed. So *package builds* which only BuildRequire python2 or
whatever would not get /usr/bin/python , and would thus fail. The
packager could always just add BuildRequires python-unversioned-command
to "fix" the problem, which is probably the fix we don't want them to
do, but at least then we have a handy way to identify recalcitrant
packages, and their maintainers can't pretend they weren't aware of the
problem and didn't actively decide to take the lazy way out.


I guess this needs to be added to the change proposal. I won't have time 
to do this now, so cc'ing Petr.


I'm not sure where to add this information. I see the Change as full of 
clear explanations of the above, but since I helped write it and know 
the technical details, it's hard for me to communicate this to people 
(especially those biased towards the view that we're breaking things 
without cause).


Could you help me out with a more concrete proposal on what to change in 
the text?




For reference:

# Abstract:

In Fedora's build system, only packages explicitly buildrequiring /usr/bin/python will get it. 


# Detailed description:

Packages that need /usr/bin/python to build will need to BuildRequire it. Packages that need /usr/bin/python to be used by users will need to Require it. In both cases, the packager should avoid the need and only fallback to (Build)Requiring /usr/bin/python as a temporary workaround. 


The new package will virtually provide python, ensuring that dnf install python will make the python command available. 


# Don't (Build)Require python-unversioned-command, but /usr/bin/python

If this change ever gets obsoleted, or for older Fedoras support, we ask the maintainers who fallback to (Build)Requiring /usr/bin/python, not to use the package name for the dependency, but the executable path instead. 


# Scope:


Maintainers of packages that use /usr/bin/python need to switch to using 
/usr/bin/python3 or /usr/bin/python2 explicitly (with help from Proposal owners 
if needed). [further instructions/recommendations follow]


# Upgrade/compatibility impact:


All packages that rely on /usr/bin/python need to change to /usr/bin/python2, 
or add explicit (Build)Requires.

Advanced users that don't install recommended packages by default will either have to install /usr/bin/python manually or live without it. Generally, the users should not be affected by this change. 


User Experience

Advanced users might uninstall the /usr/bin/python link if they want to. Others don't need to deal with this change. 



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/AL33T23VH6KNAL4WVFZLOJBCZJ56H73A/