date:20210614

On ma, 14 kesä 2021, Adam Williamson wrote:

On Mon, 2021-06-14 at 15:40 +0300, Alexander Bokovoy wrote:

>
> The fact that this blocks FreeIPA was indeed only discovered by chance
> while the side tag rebuild was already in progress (and about to be
> merged). I wonder haw can we improve the process to ensure problems
> like this are known to FreeIPA maintainers since the beginning and
> prioritized accordingly. (Ideally, the process would not only be
> improved for FreeIPA but the entire distro.)

Well, this is a dependency problem in the first place. When an ABI
change happens, like a Python ABI change with 3.10 mass-rebuild, it
should be assumed and expected that until all previously successfully
built packages were to be rebuilt, there will be broken dependencies.
Perhaps, we could extend our existing checks for a broken compose to be
done on a side-tag on demand? This way mass-rebuilders could ask for
such a run one they consider to be ready to merge and see how that
side-tag merge would affect the distribution. I don't think we have a
better way to detect it before the merge.

Note, we did do a limited version of this; I built a Rawhide installer
image with Python 3.10 and ran a subset of openQA tests on it. We did
not include the FreeIPA tests in that, which was a bit of an oversight;
however, FreeIPA tests failed for an unrelated reason in the compose
before the merge, so it wouldn't necessarily have turned up the issue
anyway.

I think in this particular case even getting a compose closure would
have been enough to get us on track to mod_wsgi problem.

We do have the ODCS - https://pagure.io/odcs ,
https://odcs.fedoraproject.org/composes/ - which we might be able to
use to do this in a more comprehensive and organized way, but I haven't
checked in on exactly what its capabilities are lately. I don't know if
it's possible to request a compose "like Rawhide, but with this side
tag" from it. We might also be able to get releng to hand-roll one, I
guess.

Thanks, this would be a great contribution to side-tag rebuilds.
We are using a similar tooling in RHEL *a lot* (in fact, for IDM CI
testing we do it with every package update).

--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag

2021-06-14 Thread Adam Williamson

On Mon, 2021-06-14 at 15:40 +0300, Alexander Bokovoy wrote:
> > 
> > The fact that this blocks FreeIPA was indeed only discovered by chance 
> > while the side tag rebuild was already in progress (and about to be 
> > merged). I wonder haw can we improve the process to ensure problems 
> > like this are known to FreeIPA maintainers since the beginning and 
> > prioritized accordingly. (Ideally, the process would not only be 
> > improved for FreeIPA but the entire distro.)
> 
> Well, this is a dependency problem in the first place. When an ABI
> change happens, like a Python ABI change with 3.10 mass-rebuild, it
> should be assumed and expected that until all previously successfully
> built packages were to be rebuilt, there will be broken dependencies.
> Perhaps, we could extend our existing checks for a broken compose to be
> done on a side-tag on demand? This way mass-rebuilders could ask for
> such a run one they consider to be ready to merge and see how that
> side-tag merge would affect the distribution. I don't think we have a
> better way to detect it before the merge.

Note, we did do a limited version of this; I built a Rawhide installer
image with Python 3.10 and ran a subset of openQA tests on it. We did
not include the FreeIPA tests in that, which was a bit of an oversight;
however, FreeIPA tests failed for an unrelated reason in the compose
before the merge, so it wouldn't necessarily have turned up the issue
anyway.

We do have the ODCS - https://pagure.io/odcs , 
https://odcs.fedoraproject.org/composes/ - which we might be able to
use to do this in a more comprehensive and organized way, but I haven't
checked in on exactly what its capabilities are lately. I don't know if
it's possible to request a compose "like Rawhide, but with this side
tag" from it. We might also be able to get releng to hand-roll one, I
guess.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Gordon Messmer


On 6/14/21 6:07 PM, Neal Gompa wrote:

That only worked because public was default, not private.
That's not how it works for Fedora packages at all.



You're much more experienced in this area than I am, so I'm inclined to 
think that you're right and I'm missing something. The guidelines do 
make sense to me though, as a packager, because if a package in Fedora 
provides python3dist(samba), then I infer that other packages record a 
dependency on a python package named "samba".  And as long as those 
packages are *only* built as RPMs and installed in Fedora, everything is 
fine.  But if one of those packages is also published on PyPI, and 
records a dependency on "samba", then anyone who installs that package 
through "pip" is going to pull in a bad dependency, and we should be 
trying to avoid that situation, typically by either publishing packages 
in PyPI or at least reserving the names.


It's unfortunate that there is a "samba" module on PyPI.  I don't know 
if we could negotiate with its owner to transfer that name. The 
"homepage" links to an absent github project, and the module does not 
appear to be actively developed.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: openvdb 8.1.0 update

2021-06-14 Thread Richard Shaw

On Mon, Jun 14, 2021 at 9:56 PM Luya Tshimbalanga 
wrote:

> Openvdb 8.1.0 is built on rawhide with so version updated to 8.1.
> Dependent packages like OpenImageIO
> needs updates to resolves their dependencies.
>
> See https://bodhi.fedoraproject.org/updates/FEDORA-2021-485cc4387a


Did I miss the announcement? soname bumps should be announced ahead of
time.

Regardless, a side tag could have been used to prevent breaking Rawhide.
Hopefully OIIO rebuilds without any patching needed.

Thanks,
Richard
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

openvdb 8.1.0 update

2021-06-14 Thread Luya Tshimbalanga

Openvdb 8.1.0 is built on rawhide with so version updated to 8.1. 
Dependent packages like OpenImageIO

needs updates to resolves their dependencies.

See https://bodhi.fedoraproject.org/updates/FEDORA-2021-485cc4387a

--
Luya Tshimbalanga
Fedora Design Team
Fedora Design Suite maintainer
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 8:35 PM Gordon Messmer  wrote:
>
> On 6/14/21 5:11 PM, Neal Gompa wrote:
> > On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  
> > wrote:
> >> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
> > It's not terribly different from how organizations may have private
> > Python package indexes that may use whatever names they want for
> > Python software they build and release.
>
>
> Yes, that was my point.  That's exactly how Alex Birsan was able to
> infiltrate and exploit "dozens" of tech companies.

That only worked because public was default, not private.
That's not how it works for Fedora packages at all.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Ways to contact wakko666 / Brett Lentz


Hello Fedorans.

I wanted to NEEDINFO Brett, but Bugzilla says:

You can't ask Brett Lentz  because that account is disabled.

Do you know any alternate ways to contact them?

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Gordon Messmer


On 6/14/21 5:11 PM, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.



Yes, that was my point.  That's exactly how Alex Birsan was able to 
infiltrate and exploit "dozens" of tech companies.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer  wrote:
>
> On 6/14/21 1:53 PM, Neal Gompa wrote:
> > This is completely unreasonable. The dist-info/egg-info data of a
> > Python module is for generating dependencies, not for forcing people
> > to deal with PyPI.
>
>
> I don't think we can reasonably separate the two.
>
> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
>
> (I believe this is the "namespace" issue mentioned in the proposed change.)

You can. When it comes to distribution packages, we can maintain
consistency of the package names referenced within the distribution.
The point of this was to be able to use the names Python packages call
themselves for dependencies. PyPI is merely a distribution center for
such things. Fedora is another distribution center for such things.

It's not terribly different from how organizations may have private
Python package indexes that may use whatever names they want for
Python software they build and release.

-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Gordon Messmer


On 6/14/21 1:53 PM, Neal Gompa wrote:

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.



I don't think we can reasonably separate the two.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

(I believe this is the "namespace" issue mentioned in the proposed change.)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 3:05 PM Miro Hrončok  wrote:
>
> On 14. 06. 21 20:09, Alexander Bokovoy wrote:
> >>  PyPI Parity 
> >>
> >> Machine-readable metadata (''distribution'' names in
> >> dist-info directories on disk and the corresponding
> >> python3.Xdist(foo) RPM provides) will match the Python
> >> Package Index (PyPI).
> >>
> >> This solves a ''namespace'' issue. Python packaging tools use a flat 
> >> namespace,
> >> and PyPI is ''the'' place where open-source Python packages are generally
> >> published, so users and tools assume a package called requests
> >> is whatever requests means on PyPI.
> >> While this is not ideal (especially for private packages), it makes sense
> >> for Fedora to align with the PyPI namespace.
> >>
> >> Note that Fedora package names are not affected – just the Python packaging
> >> metadata on disk and virtual RPM Provides generated from it.
> >>
> >> The new guidelines cover what to do for packages that cannot be registered
> >> on PyPI. The Change owner is prepared to help with PyPI registration.
> >>
> >> Note that names found in Fedora but not on PyPI
> >> [https://github.com/pypa/pypi-support/issues/355 have been reserved on 
> >> PyPI]
> >> to avoid being taken by unrelated projects.
> >
> > samba has extensive Python C bindings but does not use PyPI at all. We
> > don't want to, we don't need to, it is technically not possible without
> > building Samba from scratch and it would not make it usable for PIP
> > install without a stricter coordination of the non-Python dependencies
> > -- that's what Linux distributions do.
> >
> > In addition, 'samba' name is taken by an unrelated package on PyPI which
> > was not updated since 2019. For us this namespacing enforcement would
> > only be a problem.
>
> As long as the RPM package doesn't contain dist-info/egg-info that says "this
> is a Python package called samba" (and hence doesn't provide e.g.
> python3dist(samba)), this rule does not apply to samba's Python bindings.

This is completely unreasonable. The dist-info/egg-info data of a
Python module is for generating dependencies, not for forcing people
to deal with PyPI.

I reiterate that I did not *ever* envision the Python SIG using the
dependency generator I introduced as a means to act as a gatekeeper
for Python software packaging in Fedora.



--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Dan Čermák

Miro Hrončok  writes:

> On 14. 06. 21 19:35, Benjamin Beasley wrote:
>> I’m still in favor of running every test that is even vaguely practical in 
>> %check, but upstream Python packaging practices are wildly diverse 
>> (arguably, a mess) and it seems like a strongly worded SHOULD with a 
>> fallback of “trust the packager” would be a better approach than forcing 
>> packagers to build complicated CI configurations and go to great lengths to 
>> implement and debug network-connected tests they cannot reproduce locally.
>
> I don't disagree with you.
>
> However I think we should at least strictly require a smoke test (such as 
> %python3 -c "import foo, foo.bar") in such cases, for reasons described 
> below...

I would then suggest to change the wording from "Running upstream tests
is mandatory." to "Upstream tests SHOULD be run unless there are
compelling reasons. In that case basic smoke tests MUST be added to
%check".

We could consider suggesting Fedora CI for tests that require network
access, but given the voiced concerns its unlikely to be a viable
alternative for both package maintainers and the Python SIG conducting
Python version bump rebuilds.


Cheers,

Dan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: go-rpm-integration and symlinks

2021-06-14 Thread Robert-André Mauchin


On 6/14/21 7:22 PM, Link Dupont wrote:

I'm attempting to package github.com/rjeczalik/notify[1] as an RPM. This 
package includes a utility function[2] that determines whether a path is a 
symlink, and resolves it to an absolute path. This function clearly has merit, 
as the package is an abstraction wrapper around inotify. There are tests around 
this function that set up symlinks and test whether or not the function 
correctly resolves the symlink redirections. These test functions fail when I 
build the package[3]. It turns out the abstractions created by %goprep that set 
up the _build/src directory hierarchy (including the final symlink in the tree) 
are tripping up the tests that are attempting to resolve symlinks. I replaced 
the symlink at _build/src/github.com/rjeczalik/notify with a properly unpacked 
source directory and ran go-rpm-integration check by hand, and the tests pass. 
If _build/src/github.com/rjeczalik/notify is a symlink, however, the tests are 
failing.

Is there a proper way to manipulate the %goprep macro or tell it to not create 
symlinks? Or, what's the appropriate Fedora RPM go macro way of handling cases 
like this?




You can redefine the goprep macro in your spec by replacing the linking with 
copying:

%define gomkdir(z:i:b:s:kv) %{expand:
%goenv %{-z} %{?-i} %{?-v}
%define mybuilddir  %{?-b*}%{!-b:%{gobuilddir}}
%define mygoipath   %{?-i*}%{!-i:%{currentgoipath}}
%define mysourcedir %{?-s*}%{!-s:%{currentgosourcedir}}
%{!?-k:rm -fr "%{mysourcedir}/vendor"}
if [[ ! -e"%{mybuilddir}/bin" ]] ; then
  install -m 0755 -vd "%{mybuilddir}/bin"
  export GOPATH="%{mybuilddir}:${GOPATH:+${GOPATH}:}%{?gopath}"
fi
if [[ ! -e  "%{mybuilddir}/src/%{mygoipath}" ]] ; then
  install -m 0755 -vd   "%{mybuilddir}/src/%{mygoipath}"
  find "%{mysourcedir}" -not \\( -path "%{mysourcedir}" \\) -and -not \\( -path 
"%{mysourcedir}/_build" -prune \\) -exec cp -a "{}" "%{mybuilddir}/src/%{mygoipath}/" \\;
fi
cd  "%{mybuilddir}/src/%{mygoipath}"
}

Add a comment explaining why you need to do that though.


You will also find that the test "TestRecreated" is flaky, it is sometimes 
successful, sometimes it timeouts:

Testingin: 
/builddir/build/BUILD/notify-135d4685afb9d0fb32e23a065b95cd797cb8a4ee/_build/src
 PATH: 
/builddir/build/BUILD/notify-135d4685afb9d0fb32e23a065b95cd797cb8a4ee/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
   GOPATH: 
/builddir/build/BUILD/notify-135d4685afb9d0fb32e23a065b95cd797cb8a4ee/_build:/usr/share/gocode
  GO111MODULE: off
  command: go test -buildmode pie -compiler gc -ldflags " -X github.com/rjeczalik/notify/version.commit=135d4685afb9d0fb32e23a065b95cd797cb8a4ee -X github.com/rjeczalik/notify/version=0.9.2 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now 
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld  '"

  testing: github.com/rjeczalik/notify
github.com/rjeczalik/notify
--- FAIL: TestRecreated (5.00s)
notify_test.go:192:  First 
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Create
notify_test.go:179: /tmp/notify_test-447166328/folder/file notify.Write
notify_test.go:179: /tmp/notify_test-447166328/folder/file notify.Create
notify_test.go:199:  Second 
notify_test.go:179: /tmp/notify_test-447166328/folder/file notify.Remove
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Remove
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Remove
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Create
notify_test.go:179: /tmp/notify_test-447166328/folder/file notify.Write
notify_test.go:179: /tmp/notify_test-447166328/folder/file notify.Create
notify_test.go:207:  Third 
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Create
notify_test.go:179: /tmp/notify_test-447166328/folder/file notify.Remove
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Remove
notify_test.go:179: /tmp/notify_test-447166328/folder notify.Remove
notify_test.go:184: timed out before receiving event
FAIL
exit status 1
FAILgithub.com/rjeczalik/notify 21.392s


Report it upstream and use this workaround in the meantime:

%if %{with check}
%check
for test in "TestRecreated" \
; do
awk -i inplace '/^func.*'"$test"'\(/ { print; print "\tt.Skip(\"disabled failing 
test\")"; next}1' $(grep -rl $test)
done
%gocheck
%endif


Best regards,

Robert-André
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report i

Re: Package maintainer docs: Package Retirement: `git rm` all files in the other branches

2021-06-14 Thread Otto Urpelainen


Miro Hrončok kirjoitti 14.6.2021 klo 22.05:

On 14. 06. 21 19:50, Kevin Fenzi wrote:

It might be somehow 'fedpkg retire' doesn't handle this case?


While the wiki probably predates this feature, `fedpkg retire` will 
refuse to retire a package on a stable Fedora branch and it does not 
appear to have a --force switch.


Thank you for input everybody. I added a separate section "Completely 
Removing a Package" on the same page, using 'git rm'and also adding 
dead.package and updating fedora-obsolete-packages.


https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life#Completely_Removing_a_Package

Otto
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Improving Fedora sponsors discoverability

2021-06-14 Thread Miroslav Suchý


Dne 11. 06. 21 v 17:03 Jakub Kadlcik napsal(a):

Do you have any idea if there is a way to generate such areas and tie
people to them based on some already existing information within the
Fedora infrastructure?


What about adopting the approach we use for Fedora Planet:

  https://fedoraproject.org/wiki/Planet#How_to_join_the_Planet

I.e. fetch this information from special file in $HOME at fedorapeople.org?

Miroslav
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Package maintainer docs: Package Retirement: `git rm` all files in the other branches


On 14. 06. 21 19:50, Kevin Fenzi wrote:

It might be somehow 'fedpkg retire' doesn't handle this case?


While the wiki probably predates this feature, `fedpkg retire` will refuse to 
retire a package on a stable Fedora branch and it does not appear to have a 
--force switch.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 20:09, Alexander Bokovoy wrote:

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat namespace,
and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called requests
is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python packaging
metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be registered
on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI]
to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


As long as the RPM package doesn't contain dist-info/egg-info that says "this 
is a Python package called samba" (and hence doesn't provide e.g. 
python3dist(samba)), this rule does not apply to samba's Python bindings.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 20:28, Ben Beasley wrote:



On Mon, Jun 14, 2021, at 2:07 PM, Miro Hrončok wrote:

On 14. 06. 21 19:35, Benjamin Beasley wrote:
However I think we should at least strictly require a smoke test (such as
%python3 -c "import foo, foo.bar") in such cases, for reasons described below...
[…]

It’s also not clear to me why the Python guidelines should be so much stricter 
than the overall Fedora guidelines 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in 
this area.


As written elsewhere in this thread, because we regularly got bitten by pure
Python packages without tests.
[…]


I tentatively agree with the idea of requiring an “import foo.bar” smoke test 
in cases where the upstream tests cannot be used, especially since 
pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add 
runtime dependencies as BR’s. (It may not always be practical to explicitly 
import all subpackages/modules in a complicated package, but even importing the 
top-level package is a good start). It would catch a large portion of the FTI 
bugs that appear in practice.


We could very well add a macro helper that imports modules from the 
%{buildroot} and:


 - when given positional arguments, import the given names
 - when no arguments given, imports all modules saved by %pyproject_save_files


The NodeJS packaging guidelines have a similar requirement 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_build_testing_in_check).


Awesome requirement. Will discuss this with Petr.


Even just mandating that the runtime requirements must be BR’s 
(%pyproject_buildrequires -r or “better”) would catch most such issues at build 
time. I did this in pipx for exactly that reason.


Recently I was considering if -r shouldn't be the default (with -R or similar 
for opt-out). That should catch most of the fails to install issues.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Ben Beasley

On Mon, Jun 14, 2021, at 2:07 PM, Miro Hrončok wrote:
> On 14. 06. 21 19:35, Benjamin Beasley wrote:
> However I think we should at least strictly require a smoke test (such as 
> %python3 -c "import foo, foo.bar") in such cases, for reasons described 
> below...
> […]
> > It’s also not clear to me why the Python guidelines should be so much 
> > stricter than the overall Fedora guidelines 
> > (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) 
> > in this area.
> 
> As written elsewhere in this thread, because we regularly got bitten by pure 
> Python packages without tests.
> […]
> 
> -- 
> Miro Hrončok
> -- 
> Phone: +420777974800
> IRC: mhroncok

I tentatively agree with the idea of requiring an “import foo.bar” smoke test 
in cases where the upstream tests cannot be used, especially since 
pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add 
runtime dependencies as BR’s. (It may not always be practical to explicitly 
import all subpackages/modules in a complicated package, but even importing the 
top-level package is a good start). It would catch a large portion of the FTI 
bugs that appear in practice.

The NodeJS packaging guidelines have a similar requirement 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_build_testing_in_check).

Even just mandating that the runtime requirements must be BR’s 
(%pyproject_buildrequires -r or “better”) would catch most such issues at build 
time. I did this in pipx for exactly that reason.

– Ben Beasley
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On ma, 14 kesä 2021, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x

== Summary ==
The Python Packaging guidelines will be rewritten, with the major changes being
'''PyPI parity''' and usage of '''upstream metadata'''.

A new set of macros, {{package|pyproject-rpm-macros}}, written in mind with
the new guidelines and with upstream best practices, are documented in
the new guidelines as practical examples.

The older (a.k.a. "201x-era") Python Packaging guidelines will remain in effect
as an option (until retired by another Fedora Change).

== Owner ==
* Name: [[User:pviktori| Petr Viktorin]]
** Email: pvikt...@redhat.com

* Name: [[SIGs/Python| Python SIG]]
** Email: python-de...@lists.fedoraproject.org


== Detailed Description ==

The proposed new Python packaging guidelines are available at this
Pull request: https://pagure.io/packaging-committee/pull-request/1072

A rendered preview (which might be deleted by the time you read this)
is at: 
http://100.26.217.43/packaging-committee-pr1072/packaging-guidelines/Python/

=== Not removing older guidelines ===

The current ("201x-era") guidelines will stay in effect as an option for
packages that haven't migrated yet or those that cannot follow the new
guidelines for whatever unforeseen reason.

They will be retired in another Fedora Change,
some time after the vast majority of Python packages follow the new guidelines
and there are no known blockers for the remaining ones.
There's no rush; it might well take a decade.

=== Guideline Changes ===

See [https://hackmd.io/@python-maint/rJmQQc4DP an external document]
for a detailed list of changes to '''MUST'''/'''SHOULD''' rules.
The major ones are listed here:

 Requiring python3-devel 

All packages that use Python at run- or build-time will need to
BuildRequire {{package|python3-devel}}.
This package brings the necessary macros and settings,
and it will enable automated or manual checks.
(For example, Python maintainers use this requirement to list packages
that use Python in some way and might be affected by planned changes.)

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat namespace,
and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called requests
is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python packaging
metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be registered
on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI]
to avoid being taken by unrelated projects.


samba has extensive Python C bindings but does not use PyPI at all. We
don't want to, we don't need to, it is technically not possible without
building Samba from scratch and it would not make it usable for PIP
install without a stricter coordination of the non-Python dependencies
-- that's what Linux distributions do.

In addition, 'samba' name is taken by an unrelated package on PyPI which
was not updated since 2019. For us this namespacing enforcement would
only be a problem.


=== Burdening packagers ===

[https://lists.fedoraproject.org/archives/list/python-de...@lists.fedoraproject.org/message/IYBF5GX6HVQYPC5EJ6HVK7GJHAIKHVBK/
Neal]
does
“not necessarily disagree that PyPI and Fedora pythonXdist() names
should match, but [he disagrees] on burdening packagers with this.”

However, actually pushing a package to PyPI (and maintaining it there)
is not necessary.
The minimal-effort solution is to ''reserve'' the name on PyPI
so that a conflicting package can not be created there.

The author of this change is willing to do this work for all
Python packages whose upstream is not on PyPI;
packagers need to send a request to the Python SIG list.
If something happens to me, Red Hat's python-maint team is also ready
to do this, and in the event of catastrophic management change,
PyPI maintainers can be e-mailed directly.

This is all described in the new guidelines (see the ''PyPI parity'' section).

The new guidelines discourage the minimal-effort solution because it is
suboptimal ''for users'' (and for the wider Python ecosystem).


In case of Samba I would argue that going with anything but
minimal-effort is actually suboptimal to users.


(It was further suggested that the registration process should be ''automatic''
to minimize burden on packagers. It will be automated if it's too much for
humans to handle, but we expec

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 19:35, Benjamin Beasley wrote:

I’m still in favor of running every test that is even vaguely practical in 
%check, but upstream Python packaging practices are wildly diverse (arguably, a 
mess) and it seems like a strongly worded SHOULD with a fallback of “trust the 
packager” would be a better approach than forcing packagers to build 
complicated CI configurations and go to great lengths to implement and debug 
network-connected tests they cannot reproduce locally.


I don't disagree with you.

However I think we should at least strictly require a smoke test (such as 
%python3 -c "import foo, foo.bar") in such cases, for reasons described below...



It’s also not clear to me why the Python guidelines should be so much stricter 
than the overall Fedora guidelines 
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in 
this area.


As written elsewhere in this thread, because we regularly got bitten by pure 
Python packages without tests.


With Python packages, when there is no (useful) %check, the build basically 
just copy-pastes some files around and bytecompiles them. If there are no 
syntax errors, the bytecompilation always succeeds.


Python packages without tests always cause trouble when we measure impact of 
our changes. E.g. when we continuously rebuild packages with the next Python 
release, pure Python packages with incompatibilities but without %check always 
succeed the build, but they break other packages.


Example Python 3.10 failures that only manifest themselves in the depending 
packages because %check is non-existent or commented out:


https://bugzilla.redhat.com/show_bug.cgi?id=1968737
https://bugzilla.redhat.com/show_bug.cgi?id=1969311
https://bugzilla.redhat.com/show_bug.cgi?id=1919789
https://bugzilla.redhat.com/show_bug.cgi?id=1926112

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On ma, 14 kesä 2021, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok  wrote:

On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
> On 14.06.2021 15:32, Ben Cotton wrote:
>> Running upstream tests is mandatory.
>
> What about tests that require network access?

 From the proposed guidelines:

> If a test suite exists upstream, it MUST be run in the %check section and/or
> in Fedora CI. You MAY exclude specific failing tests.
>
> You MUST NOT disable the entire testsuite or ignore the result to solve a
> build failure.
>
> As an exception, you MAY disable tests with an appropriate %if conditional
> (e.g. bcond) when bootstrapping.

Assuming that *all the tests* require network access, you should run them on
Fedora CI instead.

Assuming only some of the tests require network access, you should exclude them
in %check (and possibly run them on Fedora CI instead).

This is unreasonable, since Fedora CI isn't usable across all
architectures. Unless Fedora CI is at platform parity, then we have to
have an allowance for not running tests.

More to that, for packages like FreeIPA, running a FreeIPA installation
-- which is what required to have a reasonable test run -- is simply not
possible in %check. We could run a subset of unit tests but that is not
going to produce a meaningful result.

Current Fedora CI capabilities also pretty limited with regards to
multi-server deployments. We rely on OpenQA testing at the Bodhi update
time for a reason.

--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Package maintainer docs: Package Retirement: `git rm` all files in the other branches

2021-06-14 Thread Kevin Fenzi

On Mon, Jun 14, 2021 at 10:07:27AM +0300, Otto Urpelainen wrote:
> Hello,
> 
> As part of the effort to move the Package Maintainer docs to
> docs.fedoraproject.org [1], I have been reviewing the existing wiki
> documentation. I would like to ask for insight on the following item:
> 
> Page _How to remove a package at end of life_ [2] first says that retirement
> is done by running 'fedpkg retire', then later alludes to a special case:
> 
> > 'git rm' all files in the other branches *only if* there are special
> factors at work, like licensing issues, or package being removed completely
> from Fedora.
> 
> I understand this to mean that in special conditions, the package cannot
> remain behind even in stable or end-of-life Fedora releases. However, I do
> not understand why 'git rm' is mandated instead of 'fedpkg retire' that is
> used in the normal case — is the 'dead.package' tombstone somehow unwanted
> in this case? I cannot understand why it would be.

It might be somehow 'fedpkg retire' doesn't handle this case? 
But I agree if it does it should be used instead of 'git rm'.
We _do_ want a dead.package there.

> Also, if the intent is to get rid of the package completely, should not
> adding it to fedora-obsolete-packages be required as well? And would that
> even work for end-of-life releases?

I would say no, but I guess we do this now all the time to normal actual
obsolete packages. :( 

> Lastly, even if this complete removal case is rare, it seems to be important
> enough to have its own process description. Should it be given its own page,
> "Package Withdrawal Process" or "Package Removal Process" or such, with
> content like "1) retire the package 2) do these additional steps"?

It is very very rare (like 3-4 times ever). I would think a seperate
section there would be fine. Or perhaps it's so rare we don't need to
document it? but that seems like it might be bad someday... 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Preventing supply chain attacks via rekor

2021-06-14 Thread Kevin Fenzi

On Sat, Jun 12, 2021 at 10:29:50AM +0200, Marius Schwarz wrote:
> Am 12.06.21 um 02:51 schrieb Kevin Fenzi:
> > 
> > > Also, not having it available has made it *very* hard to prioritize
> > > getting the issues fixed in DNF. So being able to improve this is
> > > predicated on the existence of signed metadata.
> > This seems odd to me. I mean, it can't be hard to setup a test repo, is
> > it? I suspect we could even ask QE folks to do some testing and map out
> > the issues they find. I don't think it's nice/ethical to break users
> > just as a means to make bugs we want to have fixed higher priority.
> > 
> > Anyhow, we are pretty off topic for this thread, so I'll try and stop...
> > 
> > kevin
> > 
> 
> Does it really hurt someone, if the repos get signed and clients just do not
> check this by default?

Nope, but it's still not technically possible. There needs to be work in
bodhi and robosignatory at least.
> 
> Of course, the signing should not break the unchecked repo in any form,
> which needs a small testcase ;)

Sure, but if all thats needed is a test repo, I can setup one right now
on my fedorapeople space... :) 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Benjamin Beasley

I agree that running the tests wherever practical is the best practice. I do
put my time where my mouth is—I maintain 17 packages that now use the
pyproject-rpm-macros, and you’ll find that in general I’ve added a lot of
%check sections to packages that were previously lacking them.

However, let me point to pipx as an example where I recently gave up on
maintaining an ever-shifting list of 50 tests that require interaction with
PyPI
(https://src.fedoraproject.org/rpms/pipx/c/55c6c9debf3039747327686a2f07dc31cd1e966b?branch=rawhide).
It just wasn’t worth it to keep doing this in order to run the small minority
of tests that can work offline. Even worse, upstream recently added an offline
bundle of a huge number of PyPI wheels to make these tests reproducible
offline, but this bundle is arch-specific with compiled binary wheels (a
non-starter for including as a Source# file), and it would now require a
nontrivial patch to run ANY of the tests without either the wheel bundle or a
network connection.

Now, running these fussy tests in CI would be great! I would accept a
reasonable PR to do so. But I’m not excited about learning Yet Another
YAML-Based Programming-As-Configuration Language being an absolute requirement
in order to maintain packages that have impractical upstream tests like these.
There are an increasing number of upstreams with test suites that are really
only designed to run in upstream’s own fickle CI, and reproducing these
environments could be quite a burden. Realistically, I picked up pipx and
cleaned it up after it was orphaned, and I’m likely just to file a Bugzilla and
orphan it again if CI is mandated. Unfortunately, I suspect there are many
packagers who would choose quiet noncompliance instead.

Another example is python-strictyaml. It builds its documentation using an
idiosyncratic build system (see https://hitchdev.com/) that is hopelessly
entangled with the idea of downloading dependencies from PyPI. An offline
documentation build is therefore nearly impossible. Currently, it has no tests
upstream, but if it ever adds them, they will be run via the same system and
will also be effectively impossible to run offline.

A third example is python-pgspecial. Most of the tests require a postgresql
database, which I cannot set up in %check because this requires root
privileges. This package actually seems like it would be compliant today,
because it has a few tests that work without the database connection, and
automatically skips the rest—but I am sure there are others where every single
test requires a privileged operation. That would be the case for
python-databases if it did not support SQLite.

A fourth example is python-absl-py. There are a couple of “smoke tests” that I
can and do run, but the core test suite requires Bazel—which, as has been
discussed before, will probably never be successfully packaged for Fedora.

I’m still in favor of running every test that is even vaguely practical in
%check, but upstream Python packaging practices are wildly diverse (arguably, a
mess) and it seems like a strongly worded SHOULD with a fallback of “trust the
packager” would be a better approach than forcing packagers to build
complicated CI configurations and go to great lengths to implement and debug
network-connected tests they cannot reproduce locally.

It’s also not clear to me why the Python guidelines should be so much stricter
than the overall Fedora guidelines
(https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in
this area.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Vitaly Zaitsev via devel


On 14.06.2021 19:16, Miro Hrončok wrote:
That is exactly the thing we need to avoid. Python packages without 
tests always cause trouble when we measure impact of our changes. E.g. 
when we continuously rebuild packages with the next Python release, pure 
Python packages with incompatibilities but without %check always succeed 
the build, but they break other packages.


OK, I will orphan all my Python packages than. I don't want to deal with 
tests at all.


--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

go-rpm-integration and symlinks

2021-06-14 Thread Link Dupont

I'm attempting to package github.com/rjeczalik/notify[1] as an RPM. This 
package includes a utility function[2] that determines whether a path is a 
symlink, and resolves it to an absolute path. This function clearly has merit, 
as the package is an abstraction wrapper around inotify. There are tests around 
this function that set up symlinks and test whether or not the function 
correctly resolves the symlink redirections. These test functions fail when I 
build the package[3]. It turns out the abstractions created by %goprep that set 
up the _build/src directory hierarchy (including the final symlink in the tree) 
are tripping up the tests that are attempting to resolve symlinks. I replaced 
the symlink at _build/src/github.com/rjeczalik/notify with a properly unpacked 
source directory and ran go-rpm-integration check by hand, and the tests pass. 
If _build/src/github.com/rjeczalik/notify is a symlink, however, the tests are 
failing.

Is there a proper way to manipulate the %goprep macro or tell it to not create 
symlinks? Or, what's the appropriate Fedora RPM go macro way of handling cases 
like this?

1: https://github.com/rjeczalik/notify
2: 
https://github.com/rjeczalik/notify/blob/135d4685afb9d0fb32e23a065b95cd797cb8a4ee/util.go#L67-L99
3:
+ go-rpm-integration check -i github.com/rjeczalik/notify -b 
/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/bin
 -s 
/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build -V 
0.9.2-1.20210611gite2a77dc.fc35 -C e2a77dcc14cf6732bfa4c361554f27dc696d5d79 -p 
/builddir/build/BUILDROOT/golang-github-rjeczalik-notify-0.9.2-1.20210611gite2a77dc.fc35.x86_64
 -g /usr/share/gocode -r '.*example.*'
Testingin: 
/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src
 PATH: 
/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/bin:/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
   GOPATH: 
/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build:/usr/share/gocode
  GO111MODULE: off
  command: go test -buildmode pie -compiler gc -ldflags " -X 
github.com/rjeczalik/notify/version.commit=e2a77dcc14cf6732bfa4c361554f27dc696d5d79
 -X github.com/rjeczalik/notify/version=0.9.2 -extldflags '-Wl,-z,relro 
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  '"
  testing: github.com/rjeczalik/notify
github.com/rjeczalik/notify
--- FAIL: TestCanonicalNoSymlink (0.00s)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify";
 got "/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79" 
(i=0)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify/testdata";
 got 
"/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/testdata"
 (i=1)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify";
 got "/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79" 
(i=2)
--- FAIL: TestCanonical (0.01s)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify";
 got "/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79" 
(i=0)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify/testdata";
 got 
"/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/testdata"
 (i=1)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify/notify.go";
 got 
"/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/notify.go"
 (i=2)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify/testdata/vfs.txt";
 got 
"/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/testdata/vfs.txt"
 (i=3)
util_test.go:25: want 
full="/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify/testdata/vfs.txt";
 got 
"/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/testdata/vfs.txt"
 (i=4)
--- FAIL: TestCanonical_RelativeSymlink (0.00s)
util_unix_test.go:125: want 
canonical()=/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/_build/src/github.com/rjeczalik/notify/005632054/a/b/x/y/z/d/e/f;
 got 
/builddir/build/BUILD/notify-e2a77dcc14cf6732bfa4c361554f27dc696d5d79/005632054/a/b/x/y/z/d/e/f
FAIL
exit status 1
FAILgithub.com/rjeczalik/notify 17.491s
___
devel mailing list -- devel@lists.fe

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 18:32, Vitaly Zaitsev via devel wrote:

On 14.06.2021 17:56, Miro Hrončok wrote:
Assuming that *all the tests* require network access, you should run them on 
Fedora CI instead.


And if I don't want to patch tests in downstream or not interested in runing 
tests at all?


For example my package wloc[1] has upstream tests, but they cannot run on 
Fedora due to the lack of a test suite file and secret keys.


Another example is a package that requires special equipment to run tests.

I'm strongly against this part of proposal. The package maintainers should 
decide whether they need tests or not.


There are two situations described here:

1) Running tests in %check is impossible, unreasonably hard or useless

In that case, I am inclined to agree running them should not be hard 
requirement, the packagers should instead be required to document the reasoning 
in the spec file.



2) Packager not interested in running tests at all

That is exactly the thing we need to avoid. Python packages without tests 
always cause trouble when we measure impact of our changes. E.g. when we 
continuously rebuild packages with the next Python release, pure Python 
packages with incompatibilities but without %check always succeed the build, 
but they break other packages.


Example Python 3.10 failures that only manifest themselves in the depending 
packages because %check is non-existent or commented out:


https://bugzilla.redhat.com/show_bug.cgi?id=1968737
https://bugzilla.redhat.com/show_bug.cgi?id=1969311
https://bugzilla.redhat.com/show_bug.cgi?id=1919789
https://bugzilla.redhat.com/show_bug.cgi?id=1926112

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 18:35, Neal Gompa wrote:

On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok  wrote:


On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:

On 14.06.2021 15:32, Ben Cotton wrote:

Running upstream tests is mandatory.


What about tests that require network access?


  From the proposed guidelines:


If a test suite exists upstream, it MUST be run in the %check section and/or
in Fedora CI. You MAY exclude specific failing tests.

You MUST NOT disable the entire testsuite or ignore the result to solve a
build failure.

As an exception, you MAY disable tests with an appropriate %if conditional
(e.g. bcond) when bootstrapping.



Assuming that *all the tests* require network access, you should run them on
Fedora CI instead.

Assuming only some of the tests require network access, you should exclude them
in %check (and possibly run them on Fedora CI instead).


This is unreasonable, since Fedora CI isn't usable across all
architectures. Unless Fedora CI is at platform parity, then we have to
have an allowance for not running tests.


Yeah, the situation wrt. platform parity is not good. However I still think 
that running the network-needing tests on x86_64 only is a magnitude better 
than "%%check requires internet, commented out".


That said, requiring to run tests on the Fedora CI if they cannot run in %check 
is a new thing and I would not like the new Python packaging guidelines to be 
blocked exactly on this. I don't event think this was the intention.


However, I have different reasons than platform parity: I don't want Python 
packagers to be blocked by figuring out how to do Fedora CI, since it is 
non-trivial.


Personally, I still prefer tests in %check because they are easily verifiable 
e.g. in Copr when we try to measure the impact of some of our changes. Tests in 
the CI are unusable for this use-case, because there is no tooling around it.


I suppose allowing to not run tests (at all) if they (all of them) require 
resources not available in Koji during rpmbuild (such as internet access or 
secret keys) is a reasonable compromise, however I'd like to give Petr a chance 
to pitch in as well.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok  wrote:
>
> On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
> > On 14.06.2021 15:32, Ben Cotton wrote:
> >> Running upstream tests is mandatory.
> >
> > What about tests that require network access?
>
>  From the proposed guidelines:
>
> > If a test suite exists upstream, it MUST be run in the %check section and/or
> > in Fedora CI. You MAY exclude specific failing tests.
> >
> > You MUST NOT disable the entire testsuite or ignore the result to solve a
> > build failure.
> >
> > As an exception, you MAY disable tests with an appropriate %if conditional
> > (e.g. bcond) when bootstrapping.
>
>
> Assuming that *all the tests* require network access, you should run them on
> Fedora CI instead.
>
> Assuming only some of the tests require network access, you should exclude 
> them
> in %check (and possibly run them on Fedora CI instead).

This is unreasonable, since Fedora CI isn't usable across all
architectures. Unless Fedora CI is at platform parity, then we have to
have an allowance for not running tests.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Vitaly Zaitsev via devel


On 14.06.2021 17:56, Miro Hrončok wrote:
Assuming that *all the tests* require network access, you should run 
them on Fedora CI instead.


And if I don't want to patch tests in downstream or not interested in 
runing tests at all?


For example my package wloc[1] has upstream tests, but they cannot run 
on Fedora due to the lack of a test suite file and secret keys.


Another example is a package that requires special equipment to run tests.

I'm strongly against this part of proposal. The package maintainers 
should decide whether they need tests or not.


[1]: https://github.com/xvitaly/wloc

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)


On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:

On 14.06.2021 15:32, Ben Cotton wrote:

Running upstream tests is mandatory.


What about tests that require network access?


From the proposed guidelines:


If a test suite exists upstream, it MUST be run in the %check section and/or
in Fedora CI. You MAY exclude specific failing tests.

You MUST NOT disable the entire testsuite or ignore the result to solve a
build failure.

As an exception, you MAY disable tests with an appropriate %if conditional
(e.g. bcond) when bootstrapping.



Assuming that *all the tests* require network access, you should run them on 
Fedora CI instead.


Assuming only some of the tests require network access, you should exclude them 
in %check (and possibly run them on Fedora CI instead).


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Vitaly Zaitsev via devel


On 14.06.2021 15:32, Ben Cotton wrote:

Running upstream tests is mandatory.


What about tests that require network access?

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Richard W.M. Jones

Questions, questions ...

These new guidelines seem to be fine for pure Python packages, but I'm
maintaining a couple of packages where Python bindings are built as
subpackages of existing C libraries:

https://src.fedoraproject.org/rpms/libnbd
https://src.fedoraproject.org/rpms/libguestfs

[Yes there are good reasons for this.  No we are not going to decouple
them and package the Python stuff separately.]

Running %pyproject_install etc is not really an option for us.
However I guess some of these macros could make sense
(%pyproject_save_files?  %pyproject_files?).  Also I don't think these
projects generate the extra *.egg-info files or how to create them.

> === Not removing older guidelines ===
> There's no rush; it might well take a decade.

I guess we've got a while ...

While I've got your attention, one other package is interesting:

https://src.fedoraproject.org/rpms/nbdkit

This has Python 3 bindings but they work in completely the opposite
way to normal bindings, namely a Python interpreter is embedded in a
C program.  As far as I know none of the Python guidelines in Fedora
address this scenario directly, and indeed we don't currently use any
of the special %python* macros.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Self Introduction: Odilon Junior

2021-06-14 Thread Ewoud Kohl van Wijngaarden


On Fri, Jun 11, 2021 at 07:00:52PM -0300, Odilon Junior wrote:

My name is Odilon Junior,


Hi :)


I'm also introducing myself because I want to become a maintainer of a new
package in Fedora/EPEL, the new package is called obal[2], there's already
one version of the package in copr. This package is heavily used by the
Foreman/Katello team, which I joined last month, we use it everyday in our
workstations and also in our tooling, obal is a python wrapper around
ansible, obal is the tool that we use to create/update/release packages for
Foreman/Katello/Satellite.

I already created one build on koji to check if the build would pass for
F34, obal[4] and python-obsah[5]. I'm uploading the 2 spec files to one
repo on github[6].

2 - https://github.com/theforeman/obal
3 - https://copr.fedorainfracloud.org/coprs/ekohl/obal/
4 - https://koji.fedoraproject.org/koji/taskinfo?taskID=69896768
5 - https://koji.fedoraproject.org/koji/taskinfo?taskID=69897019
6 - https://github.com/Odilhao/obal-specs


As the current maintainer of the copr, I happily support adding this to 
Fedora. However, I'm not sure I'm the right person to review given I 
authored the specs.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: What are https://src.fedoraproject.org/container ?

2021-06-14 Thread Richard W.M. Jones

On Sat, Jun 12, 2021 at 08:00:15AM -0400, Neal Gompa wrote:
> At build-time, libguestfs copies the content of the kernel package
> along with binaries of various filesystem tools into itself to run a
> custom appliance for manipulating VMs. The Fedora-based libguestfs
> package can handle Btrfs even on RHEL because it relies on the
> binaries of the Fedora kernel and filesystem utilities instead of the
> RHEL ones. It will run QEMU and boot up *its* VM to manipulate VM
> stuff. That's even how guestfish works for mounting VM disks on the host.

So it's a bit more subtle than this.

Normally libguestfs will build the appliance at runtime, using files
from the host.  It is then run it using qemu + the latest kernel image
found in /boot or /lib/modules (so not necessarily the host kernel,
but it might be).  The process is described here:

  https://rwmj.wordpress.com/2014/03/08/supermin-version-5/

This has the advantage that you don't have to ship the appliance at
all (which is usually about ~300 MB, so quite a saving), and security
updates are handled automatically.  Instead we ship only this in the
RPM:

  $ du -sh /usr/lib64/guestfs/supermin.d/
  2.3M /usr/lib64/guestfs/supermin.d/

However ... containers.  There's all kinds of weirdness / brokenness
with containers (and especially when you combine them with Kubernetes)
which makes this harder to do:

 - usually limited or unpredictable space on /var/tmp so we have
   nowhere to build and cache the appliance (but shlepping around
   hundreds of megabytes of the same appliance in the container? totes fine!
   go figure ...)

 - "bazel" doesn't build an RPM database or run %post scripts, so it
   makes something that looks a bit like a container running Fedora,
   but is quite broken, in particular supermin can't work

 - missing/broken kernel packages

My colleague is currently building a container-based version of
libguestfs which does indeed work a lot more like Neal describes
above.  There will be a pre-built appliance, updated every so often
(hopefully often enough that security issues won't be too much of a
problem).  It'll get downloaded -- all hundreds of megabytes --
through the usual container distribution channels.

This is actually why I was interested in the question originally since
I was wondering if there was duplicated effort going on.

I should stress this is only for containers.  supermin will continue
to be used in regular Fedora.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Azure CLI + SDK, side tags, and metapackages


On 14. 06. 21 15:36, Major Hayden wrote:

  2) How do I convert the python-azure-sdk package into a no-files
 metapackage that brings in all of the SDK components?


I.   Bump the metadata to a newer version-release (artificial?).
II.  Remove all sources and contents of %prep/%build/%install.
III. Remove all listed %files but keep the %files section.
IV.  Adapt other metdata like description, summary etc.
V.   Add the requirements that brings in all of the SDK components.
VI.  Use `%py_provides python3-azure-sdk` in the python3-azure-sdk package, 
as this is required for Python meta-packages without files.


This makes sense. Can I ping you when/if I have a PR ready in pagure to ensure 
I'm making good choices? 😉


Absolutely.

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Azure CLI + SDK, side tags, and metapackages

2021-06-14 Thread Major Hayden


On 6/14/21 8:30 AM, Miro Hrončok wrote:
Not sure what you mean by "supersede" in this context, but I assume it 
wouldn't.


Let me break it down:

Would the python3-azure-common package automatically replace 
python3-azure-sdk during upgrades? No.


Would the python3-azure-common package implicitly conflict with 
python3-azure-sdk? Possibly, if they install to overlapping locations on 
the filesystem.


If a package (build)requires python3dist(azure-common) (or 
python3.Xdist(azure-common)), would it get python3-azure-common instead 
of python3-azure-sdk? Possibly, if python3-azure-sdk isn't pulled in by 
another requirement. The fact that "common" sorts earlier than "sdk" 
alphabetically plays in your favor :D


Oh wow, that's a lot to consider. Thanks for helping me understand that.


Here are my main questions:

  1) How do I bring in these smaller packages carefully to
 avoid disruption?


First of all, use a side tag:

https://fedoraproject.org/wiki/Package_update_HOWTO#Multiple_Packages


I'll read up on that. Thank you.


Also, make sure to test the upgrade path. E.g. by:

I.   Installing the existing package versions on Fedora rawhide.
II.  Adding your copr repo.
III. Testing `dnf upgrade` and also testing `dnf upgrade 
`


I have a small shell script that has been doing this for me in a rawhide 
container with podman.


When in trouble, add explicit Conflicts and/or Obsoletes as needed. If 
you share the error you get in (III.), I can help with this step.



  2) How do I convert the python-azure-sdk package into a no-files
 metapackage that brings in all of the SDK components?


I.   Bump the metadata to a newer version-release (artificial?).
II.  Remove all sources and contents of %prep/%build/%install.
III. Remove all listed %files but keep the %files section.
IV.  Adapt other metdata like description, summary etc.
V.   Add the requirements that brings in all of the SDK components.
VI.  Use `%py_provides python3-azure-sdk` in the python3-azure-sdk 
package, as this is required for Python meta-packages without files.


This makes sense. Can I ping you when/if I have a PR ready in pagure to 
ensure I'm making good choices? 😉



 (Is this even a good idea?)


That depends. Would the users still want to `dnf install 
python3-azure-sdk`? If no, I would rather retire and obsolete the 
package from the relevant replacements. If you Obsolete a package from N 
other packages, they replace the obsoleted package during upgrade (all 
of them).


There's one other package that depends on the python3-azure-sdk package, 
but that's it. I'll go review its dependencies to see exactly what it is 
pulling in.



  3) How should I go about getting these smaller packages reviewed?
 The majority of them are almost identical. 


I'd proceed normally, unless it's hundreds or thousands. How many 
reviews do you need? We can organize a sprint :)


There are about 80-90 packages last time I looked. Some have 
dependencies, so I've been getting the initial base ones reviewed first.


Thank you!

--
Major Hayden


OpenPGP_0x737051E0C1011FB1.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)

2021-06-14 Thread Ben Cotton

https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x

== Summary ==
The Python Packaging guidelines will be rewritten, with the major changes being
'''PyPI parity''' and usage of '''upstream metadata'''.

A new set of macros, {{package|pyproject-rpm-macros}}, written in mind with
the new guidelines and with upstream best practices, are documented in
the new guidelines as practical examples.

The older (a.k.a. "201x-era") Python Packaging guidelines will remain in effect
as an option (until retired by another Fedora Change).

== Owner ==
* Name: [[User:pviktori| Petr Viktorin]]
** Email: pvikt...@redhat.com

* Name: [[SIGs/Python| Python SIG]]
** Email: python-de...@lists.fedoraproject.org


== Detailed Description ==

The proposed new Python packaging guidelines are available at this
Pull request: https://pagure.io/packaging-committee/pull-request/1072

A rendered preview (which might be deleted by the time you read this)
is at: 
http://100.26.217.43/packaging-committee-pr1072/packaging-guidelines/Python/

=== Not removing older guidelines ===

The current ("201x-era") guidelines will stay in effect as an option for
packages that haven't migrated yet or those that cannot follow the new
guidelines for whatever unforeseen reason.

They will be retired in another Fedora Change,
some time after the vast majority of Python packages follow the new guidelines
and there are no known blockers for the remaining ones.
There's no rush; it might well take a decade.

=== Guideline Changes ===

See [https://hackmd.io/@python-maint/rJmQQc4DP an external document]
for a detailed list of changes to '''MUST'''/'''SHOULD''' rules.
The major ones are listed here:

 Requiring python3-devel 

All packages that use Python at run- or build-time will need to
BuildRequire {{package|python3-devel}}.
This package brings the necessary macros and settings,
and it will enable automated or manual checks.
(For example, Python maintainers use this requirement to list packages
that use Python in some way and might be affected by planned changes.)

 PyPI Parity 

Machine-readable metadata (''distribution'' names in
dist-info directories on disk and the corresponding
python3.Xdist(foo) RPM provides) will match the Python
Package Index (PyPI).

This solves a ''namespace'' issue. Python packaging tools use a flat namespace,
and PyPI is ''the'' place where open-source Python packages are generally
published, so users and tools assume a package called requests
is whatever requests means on PyPI.
While this is not ideal (especially for private packages), it makes sense
for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python packaging
metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be registered
on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI
[https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI]
to avoid being taken by unrelated projects.


 Upstream metadata 

Upstream “dist-info” metadata for Python packages is now standardized and
shareable with other distributors (be it Linux distros or others).
As much as possible, RPM metadata should be automatically taken from there,
rather be duplicated in the spec file (and likely diverge over time).
This includes run-time and build-time requirements for Python packages, tests
and test requirements for packages that use the tox tool,
and ''Extras'' (optional features).

Packagers are expected to treat metadata bugs as any other bugs.
(That is, ideally: patch them and present the patches upstream).

 Naming 

The new guidelines explain the various names a Python package can have
(Fedora package name, project name, import name)
and begs developers/maintainers to keep them synchronized if possible.

 Python 3 

The new guidelines only cover Python 3.

Python 2 does not need its own guidelines.
{{package|python2.7}} is deprecated and as of this writing,
only about 35 packages use it, usually under individual FESCo exceptions.

Python 4 is not planned upstream.

 Tests, linters and coverage 

Running upstream tests is mandatory.
Running linters (e.g. {{package|pylint}}, {{package|pyflakes}},
{{package|python-black}})
or test coverage measurements (e.g. {{package|python-coverage}}) is discouraged.

=== Non-Guideline Changes ===

The {{package|python3-devel}} package, a mandatory build-time requirement
for all packages that use Python,
will require {{package|pyproject-rpm-macros}},
helpers designed for the new guidelines.

While this small package is not always necessary,
it will make life easier for most Python packagers.

{{package|pyproject-rpm-macros}} is currently
[https://src.fedoraproject.org/rpms/pyproject-rpm-macros documented as
''provisional''];
that label will be removed and a 1.0 release will be made.

=== Example spe

Re: Azure CLI + SDK, side tags, and metapackages


On 14. 06. 21 14:40, Major Hayden wrote:
My python-azure-common package that just finished a review[3] is version 
1.1.27, so I would assume that the python-azure-common package would supersede 
the python-azure-sdk package since it provides a newer version of the 
azure-common module. (Is this true?)


Not sure what you mean by "supersede" in this context, but I assume it wouldn't.

Let me break it down:

Would the python3-azure-common package automatically replace python3-azure-sdk 
during upgrades? No.


Would the python3-azure-common package implicitly conflict with 
python3-azure-sdk? Possibly, if they install to overlapping locations on the 
filesystem.


If a package (build)requires python3dist(azure-common) (or 
python3.Xdist(azure-common)), would it get python3-azure-common instead of 
python3-azure-sdk? Possibly, if python3-azure-sdk isn't pulled in by another 
requirement. The fact that "common" sorts earlier than "sdk" alphabetically 
plays in your favor :D



Here are my main questions:

  1) How do I bring in these smaller packages carefully to
 avoid disruption?


First of all, use a side tag:

https://fedoraproject.org/wiki/Package_update_HOWTO#Multiple_Packages

Also, make sure to test the upgrade path. E.g. by:

I.   Installing the existing package versions on Fedora rawhide.
II.  Adding your copr repo.
III. Testing `dnf upgrade` and also testing `dnf upgrade `

When in trouble, add explicit Conflicts and/or Obsoletes as needed. If you 
share the error you get in (III.), I can help with this step.



  2) How do I convert the python-azure-sdk package into a no-files
 metapackage that brings in all of the SDK components?


I.   Bump the metadata to a newer version-release (artificial?).
II.  Remove all sources and contents of %prep/%build/%install.
III. Remove all listed %files but keep the %files section.
IV.  Adapt other metdata like description, summary etc.
V.   Add the requirements that brings in all of the SDK components.
VI.  Use `%py_provides python3-azure-sdk` in the python3-azure-sdk package, as 
this is required for Python meta-packages without files.



 (Is this even a good idea?)


That depends. Would the users still want to `dnf install python3-azure-sdk`? If 
no, I would rather retire and obsolete the package from the relevant 
replacements. If you Obsolete a package from N other packages, they replace the 
obsoleted package during upgrade (all of them).



  3) How should I go about getting these smaller packages reviewed?
 The majority of them are almost identical. 


I'd proceed normally, unless it's hundreds or thousands. How many reviews do 
you need? We can organize a sprint :)


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On 14. 06. 21 14:40, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Miro Hrončok wrote:

On 14. 06. 21 13:40, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Victor Stinner wrote:

Congrats, that's amazing! :-) Let's fix remaining broken packages!


Right now the biggest broken package to us (FreeIPA) is mod_wsgi which
relied on Python C API which was removed in Python 3.10. The same API
was used by uWSGI as well, so it would really be great if Python core
developers and Python maintainers team could have helped mod_wsgi/uWSGI
upstreams to migrate from the old API.

Sadly, this was only found after the fact when a side-tag was already
merged, so FreeIPA right now is broken in Fedora Rawhide.


The problem in mod_wsgi was actually found and reported in November 2020 [1], 
which is exactly why we test with each pre-release of Python: To report 
problems *early* and give maintainers time to fix them, so we can land the 
new version with limited negative impact.


Unfortunately, the mod_wsgi maintainer did not respond to the problem :( 
While we try really hard, we cannot report each failure to upstreams directly.


The fact that this blocks FreeIPA was indeed only discovered by chance while 
the side tag rebuild was already in progress (and about to be merged). I 
wonder haw can we improve the process to ensure problems like this are known 
to FreeIPA maintainers since the beginning and prioritized accordingly. 
(Ideally, the process would not only be improved for FreeIPA but the entire 
distro.)


Well, this is a dependency problem in the first place. When an ABI
change happens, like a Python ABI change with 3.10 mass-rebuild, it
should be assumed and expected that until all previously successfully
built packages were to be rebuilt, there will be broken dependencies.


Yes, however getting "all previously successfully built packages rebuilt" is an 
impossible goal. Instead we strive to get "most of them rebuilt" and "the 
important ones rebuilt". Getting most of them rebuilt is not an easy task but 
at least is is quite easy to quantify:  are rebuilt, 293 to go, that is 
~91%. OTOH knowing what is "important" is hard. There is the "critical path" 
thing (but mod_wsgi is not in it and the list seems heavily outdated).



Perhaps, we could extend our existing checks for a broken compose to be
done on a side-tag on demand? This way mass-rebuilders could ask for
such a run one they consider to be ready to merge and see how that
side-tag merge would affect the distribution. I don't think we have a
better way to detect it before the merge.


We try hard not to break the compose. It is not fully automated and having a 
way to run a test compose would be awesome.


However, this particular problem does not break the compose, the compose is 
done now even when mod_wsgi is not installable.


We have also validated the following groups are installable before merging the 
side tag:


  @anaconda-tools
  @base-x
  @cloud-server-environment
  @container-management
  @core
  @domain-client
  @firefox
  @fonts
  @guest-agents
  @headless-management
  @input-methods
  @kde-apps
  @kde-desktop-environment
  @kde-media
  @libreoffice
  @multimedia
  @networkmanager-submodules
  @printing
  @server-product
  @standard
  @workstation-product-environment

Because we parsed them from the hopefully compsoe-blocking media kickstarts.


What I do not see as acceptable is what Python core team did in November
2020 when this issue was made clear in the original PR to remove the
'old' parser that broke mod_wsgi/uWSGI/unbound and some more packages.
Instead of following up in providing a supported API function, the
comment[1] from Victor was practically ignored.


I am not happy either that Python upstream keeps breaking things and often the 
migration path is "go figure". Whenever and wherever I can, I argue against that :(


I assume that opening bugzillas for all the dependent packages for each 
failure would be considered as spam by many.


It depends on what that bug opening would mean. If you are doing it
early together with an upstream that breaks its ABI/API, helping them to
see the issues early is one of your goals -- as you stated above.
Realizing through reverse dependencies a damage such breakage could
cause is something you (as Python maintainers in Fedora) could
definitely do and raise the issue earlier also with the package
maintainers/upstreams. In this case FreeIPA is affected but we missed it
completely in November 2020 as nobody told us mod_wsgi would not work
anymore.


This is a chicken-egg problem: Until we measure the impact, we don't know it. 
And reporting many dependent-packages bugzillas early seems like a waste of 
effort if the bug is getting fixed soon. What we need to detect (and we don't 
know how to automate that yet) is:


 - a bugzilla for a problem is stalled AND
 - the package is dependent on by "important stuff" or by a large stack

Even when not automated, we manually rise priorities for bugzill

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag

2021-06-14 Thread Fabio Valentini

On Mon, Jun 14, 2021 at 2:54 PM Miro Hrončok  wrote:
>
> On 14. 06. 21 14:17, Miro Hrončok wrote:
> >> I see one problem with the Rust bindings for CPython / libpython,
> >> where the test suite now fails with Python 3.10:
> >> https://koschei.fedoraproject.org/package/rust-cpython?
> >>
> >> Looking at the build log, the new test failures seem to be caused by
> >> either API removals or subtle behaviour changes:
> >>
> >> - TypeError: 'float' object cannot be interpreted as an integer
> >
> > Indeed. wrap it in int() or math.floor()/ceil() as needed.
> >
> > I suspect this is related to:
> >
> > https://docs.python.org/3.10/whatsnew/3.10.html#other-language-changes
> >
> > """
> > Builtin and extension functions that take integer arguments no longer accept
> > Decimals, Fractions and other objects that can be converted to integers only
> > with a loss (e.g. that have the __int__() method but do not have the
> > __index__() method). https://bugs.python.org/issue37999
> > """
>
> Can I get the Python traceback somehow? The rust thing (object?) seem to have:
>
> PyErr { ptype: , pvalue: Some("'float' object cannot be
> interpreted as an integer"), ptraceback: None }

Not sure ... looking at the sources for the tests:
https://github.com/dgrunwald/rust-cpython/blob/master/src/objects/num.rs#L340
It seems that this just will not work as expected for some float ->
int -> float roundtrip conversions any longer, due to the changes in
python 3.10.

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On 14. 06. 21 14:17, Miro Hrončok wrote:

I see one problem with the Rust bindings for CPython / libpython,
where the test suite now fails with Python 3.10:
https://koschei.fedoraproject.org/package/rust-cpython?

Looking at the build log, the new test failures seem to be caused by
either API removals or subtle behaviour changes:

- TypeError: 'float' object cannot be interpreted as an integer


Indeed. wrap it in int() or math.floor()/ceil() as needed.

I suspect this is related to:

https://docs.python.org/3.10/whatsnew/3.10.html#other-language-changes

"""
Builtin and extension functions that take integer arguments no longer accept 
Decimals, Fractions and other objects that can be converted to integers only 
with a loss (e.g. that have the __int__() method but do not have the 
__index__() method). https://bugs.python.org/issue37999

"""


Can I get the Python traceback somehow? The rust thing (object?) seem to have:

PyErr { ptype: , pvalue: Some("'float' object cannot be 
interpreted as an integer"), ptraceback: None }



--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Azure CLI + SDK, side tags, and metapackages

2021-06-14 Thread Major Hayden


Hey there,

I'd like to provide an update on my packaging work for the Azure SDK and 
CLI[0] and ask for some help.


Current status:

  * All non-Azure deps are packaged in rawhide 🎉
  * SDK + CLI components now co-exist in a COPR repository[1]
  * Base SDK components (azure-core, azure-common) are reviewed
and ready to add to Fedora[2][3]

The challenge now is the python-azure-sdk package. It's packaged as a 
monolith package that took the final composite release of 
azure-sdk-for-python[4] version 5.0.0. Azure does not release the SDK 
components as a monolith any longer -- each SDK component is released 
independently with its own versioning scheme.


It took me a while to understand how the automatic python provides 
works, but it seems that the python-azure-sdk package has lots of 
automatically generated python provides for the individual components 
inside of it:



➜ LANG=C dnf repoquery --provides python3-azure-sdk | grep azure-common
Last metadata expiration check: 0:00:15 ago on Mon Jun 14 07:35:11 2021.
python3.9dist(azure-common) = 1.1.26
python3dist(azure-common) = 1.1.26


My python-azure-common package that just finished a review[3] is version 
1.1.27, so I would assume that the python-azure-common package would 
supersede the python-azure-sdk package since it provides a newer version 
of the azure-common module. (Is this true?)


Ben suggested[5] assembling a COPR with everything in it so we could get 
an idea of how everything would look in Fedora. I've done that[1] but 
I'm not sure where to go from here.


Here are my main questions:

  1) How do I bring in these smaller packages carefully to
 avoid disruption?

  2) How do I convert the python-azure-sdk package into a no-files
 metapackage that brings in all of the SDK components?
 (Is this even a good idea?)

  3) How should I go about getting these smaller packages reviewed?
 The majority of them are almost identical.

Thanks for your help! 🤗

[0] 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/64Q5UZHNFSF6U3CZWKJ72PCQ7NJMCQSZ/

[1] https://copr.fedorainfracloud.org/coprs/mhayden/azure-cli/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1970073
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1970619
[4] https://github.com/azure/azure-sdk-for-python
[5] https://bugzilla.redhat.com/show_bug.cgi?id=1970619#c8

--
Major Hayden


OpenPGP_0x737051E0C1011FB1.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On ma, 14 kesä 2021, Miro Hrončok wrote:

On 14. 06. 21 13:40, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Victor Stinner wrote:

Congrats, that's amazing! :-) Let's fix remaining broken packages!


Right now the biggest broken package to us (FreeIPA) is mod_wsgi which
relied on Python C API which was removed in Python 3.10. The same API
was used by uWSGI as well, so it would really be great if Python core
developers and Python maintainers team could have helped mod_wsgi/uWSGI
upstreams to migrate from the old API.

Sadly, this was only found after the fact when a side-tag was already
merged, so FreeIPA right now is broken in Fedora Rawhide.


The problem in mod_wsgi was actually found and reported in November 
2020 [1], which is exactly why we test with each pre-release of 
Python: To report problems *early* and give maintainers time to fix 
them, so we can land the new version with limited negative impact.


Unfortunately, the mod_wsgi maintainer did not respond to the problem 
:( While we try really hard, we cannot report each failure to 
upstreams directly.


The fact that this blocks FreeIPA was indeed only discovered by chance 
while the side tag rebuild was already in progress (and about to be 
merged). I wonder haw can we improve the process to ensure problems 
like this are known to FreeIPA maintainers since the beginning and 
prioritized accordingly. (Ideally, the process would not only be 
improved for FreeIPA but the entire distro.)


Well, this is a dependency problem in the first place. When an ABI
change happens, like a Python ABI change with 3.10 mass-rebuild, it
should be assumed and expected that until all previously successfully
built packages were to be rebuilt, there will be broken dependencies.
Perhaps, we could extend our existing checks for a broken compose to be
done on a side-tag on demand? This way mass-rebuilders could ask for
such a run one they consider to be ready to merge and see how that
side-tag merge would affect the distribution. I don't think we have a
better way to detect it before the merge.

What I do not see as acceptable is what Python core team did in November
2020 when this issue was made clear in the original PR to remove the
'old' parser that broke mod_wsgi/uWSGI/unbound and some more packages.
Instead of following up in providing a supported API function, the
comment[1] from Victor was practically ignored.

I assume that opening bugzillas for all the dependent packages for 
each failure would be considered as spam by many.


It depends on what that bug opening would mean. If you are doing it
early together with an upstream that breaks its ABI/API, helping them to
see the issues early is one of your goals -- as you stated above.
Realizing through reverse dependencies a damage such breakage could
cause is something you (as Python maintainers in Fedora) could
definitely do and raise the issue earlier also with the package
maintainers/upstreams. In this case FreeIPA is affected but we missed it
completely in November 2020 as nobody told us mod_wsgi would not work
anymore.

In the meantime, I've marked mod_wsgi as "blocker" for the Python 3.11 
rebuild next year, but that is not a systematic solution.



I saw that there is some discussion and an effort to help mod_wsgi
upstream already, thank you!


There is a proposed fix [2] but we don't know enough about Apache to 
say if it is ideal. In case this blocks you very much, I suggest you 
work with the Fedora's mod_wsgi maintainer and backport this patch to 
rawhide, keeping an eye on it in case it breaks your use cases, and 
report problems to us, so we can adapt it. At this point of the Fedora 
35 development schedule, I think this patch should be good enough for 
it.


I already nominated mod_wsgi bug as Fedora 35 beta release blocker
because it violates Fedora Server release criteria[2]. I am not a
maintainer for mod_wsgi myself and have no rights to that package.

[1] https://bugs.python.org/issue40939#msg381227
[2] 
https://fedoraproject.org/wiki/Basic_Release_Criteria#FreeIPA_server_requirements

--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag

2021-06-14 Thread Fabio Valentini

On Mon, Jun 14, 2021 at 2:21 PM Miro Hrončok  wrote:
>
> On 14. 06. 21 14:17, Miro Hrončok wrote:
> >> - AttributeError: module 'unicodedata' has no attribute 'ucnhash_CAPI'
> >
> > https://docs.python.org/3.10/whatsnew/3.10.html#removed
> >
> > """
> > Removed the unicodedata.ucnhash_CAPI attribute which was an internal 
> > PyCapsule
> > object. The related private _PyUnicode_Name_CAPI structure was moved to the
> > internal C API. https://bugs.python.org/issue42157
> > """
> >
> > The relevant issue makes it sound like the unicodedata.ucnhash_CAPI 
> > attribute
> > was not usable at all but in case you've used it, I gues it was. There was 
> > no
> > deprecation period for this and maybe it should not have been removed?
>
> I've reported https://bugs.python.org/issue44418

Thanks, and thanks for your help with deciphering the exception messages.
At least I know what caused them now, which should help with fixing
them for good :)

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On 14. 06. 21 14:17, Miro Hrončok wrote:

- AttributeError: module 'unicodedata' has no attribute 'ucnhash_CAPI'


https://docs.python.org/3.10/whatsnew/3.10.html#removed

"""
Removed the unicodedata.ucnhash_CAPI attribute which was an internal PyCapsule 
object. The related private _PyUnicode_Name_CAPI structure was moved to the 
internal C API. https://bugs.python.org/issue42157

"""

The relevant issue makes it sound like the unicodedata.ucnhash_CAPI attribute 
was not usable at all but in case you've used it, I gues it was. There was no 
deprecation period for this and maybe it should not have been removed?


I've reported https://bugs.python.org/issue44418

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On 14. 06. 21 13:53, Fabio Valentini wrote:

On Sun, Jun 13, 2021 at 9:52 PM Miro Hrončok  wrote:


And we have a successful compose, so Python 3.10 is now the main Python on 
Rawhide.


Awesome, thanks for working on this, as always.
Python version updates are always scary, but you guys handle them so well.

I see one problem with the Rust bindings for CPython / libpython,
where the test suite now fails with Python 3.10:
https://koschei.fedoraproject.org/package/rust-cpython?

Looking at the build log, the new test failures seem to be caused by
either API removals or subtle behaviour changes:

- TypeError: 'float' object cannot be interpreted as an integer


Indeed. wrap it in int() or math.floor()/ceil() as needed.

I suspect this is related to:

https://docs.python.org/3.10/whatsnew/3.10.html#other-language-changes

"""
Builtin and extension functions that take integer arguments no longer accept 
Decimals, Fractions and other objects that can be converted to integers only 
with a loss (e.g. that have the __int__() method but do not have the 
__index__() method). https://bugs.python.org/issue37999

"""



- AttributeError: module 'unicodedata' has no attribute 'ucnhash_CAPI'


https://docs.python.org/3.10/whatsnew/3.10.html#removed

"""
Removed the unicodedata.ucnhash_CAPI attribute which was an internal PyCapsule 
object. The related private _PyUnicode_Name_CAPI structure was moved to the 
internal C API. https://bugs.python.org/issue42157

"""

The relevant issue makes it sound like the unicodedata.ucnhash_CAPI attribute 
was not usable at all but in case you've used it, I gues it was. There was no 
deprecation period for this and maybe it should not have been removed?


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On 14. 06. 21 13:40, Alexander Bokovoy wrote:

On ma, 14 kesä 2021, Victor Stinner wrote:

Congrats, that's amazing! :-) Let's fix remaining broken packages!


Right now the biggest broken package to us (FreeIPA) is mod_wsgi which
relied on Python C API which was removed in Python 3.10. The same API
was used by uWSGI as well, so it would really be great if Python core
developers and Python maintainers team could have helped mod_wsgi/uWSGI
upstreams to migrate from the old API.

Sadly, this was only found after the fact when a side-tag was already
merged, so FreeIPA right now is broken in Fedora Rawhide.


The problem in mod_wsgi was actually found and reported in November 2020 [1], 
which is exactly why we test with each pre-release of Python: To report 
problems *early* and give maintainers time to fix them, so we can land the new 
version with limited negative impact.


Unfortunately, the mod_wsgi maintainer did not respond to the problem :( While 
we try really hard, we cannot report each failure to upstreams directly.


The fact that this blocks FreeIPA was indeed only discovered by chance while 
the side tag rebuild was already in progress (and about to be merged). I wonder 
haw can we improve the process to ensure problems like this are known to 
FreeIPA maintainers since the beginning and prioritized accordingly. (Ideally, 
the process would not only be improved for FreeIPA but the entire distro.)


I assume that opening bugzillas for all the dependent packages for each failure 
would be considered as spam by many.


In the meantime, I've marked mod_wsgi as "blocker" for the Python 3.11 rebuild 
next year, but that is not a systematic solution.



I saw that there is some discussion and an effort to help mod_wsgi
upstream already, thank you!


There is a proposed fix [2] but we don't know enough about Apache to say if it 
is ideal. In case this blocks you very much, I suggest you work with the 
Fedora's mod_wsgi maintainer and backport this patch to rawhide, keeping an eye 
on it in case it breaks your use cases, and report problems to us, so we can 
adapt it. At this point of the Fedora 35 development schedule, I think this 
patch should be good enough for it.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1898158#c0
[2] https://github.com/GrahamDumpleton/mod_wsgi/pull/688

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag

2021-06-14 Thread Fabio Valentini

On Sun, Jun 13, 2021 at 9:52 PM Miro Hrončok  wrote:
>
> And we have a successful compose, so Python 3.10 is now the main Python on 
> Rawhide.

Awesome, thanks for working on this, as always.
Python version updates are always scary, but you guys handle them so well.

I see one problem with the Rust bindings for CPython / libpython,
where the test suite now fails with Python 3.10:
https://koschei.fedoraproject.org/package/rust-cpython?

Looking at the build log, the new test failures seem to be caused by
either API removals or subtle behaviour changes:

- TypeError: 'float' object cannot be interpreted as an integer
- AttributeError: module 'unicodedata' has no attribute 'ucnhash_CAPI'

I have no knowledge of the libpython API, so help with debugging and
fixing those failures would be very much appreciated.

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag


On ma, 14 kesä 2021, Victor Stinner wrote:

Congrats, that's amazing! :-) Let's fix remaining broken packages!


Right now the biggest broken package to us (FreeIPA) is mod_wsgi which
relied on Python C API which was removed in Python 3.10. The same API
was used by uWSGI as well, so it would really be great if Python core
developers and Python maintainers team could have helped mod_wsgi/uWSGI
upstreams to migrate from the old API.

Sadly, this was only found after the fact when a side-tag was already
merged, so FreeIPA right now is broken in Fedora Rawhide.

I saw that there is some discussion and an effort to help mod_wsgi
upstream already, thank you!



Victor

On Sun, Jun 13, 2021 at 9:52 PM Miro Hrončok  wrote:


On 08. 06. 21 1:01, Miro Hrončok wrote:
> On 02. 06. 21 10:02, Tomas Hrnciar wrote:
>> Hello, in order to deliver Python 3.10, we are running a coordinated rebuild
>> in a side tag.
>>
>> https://fedoraproject.org/wiki/Changes/Python3.10
>> 
>>
>> If you see a "Rebuilt for Python 3.10" (or similar) commit in your package,
>> please don't rebuild it in regular rawhide.
>> If you need to, please let usknow, so we can coordinate.
>>
>> If you'd like to build the package, you should be able to build it in the
>> side tag via:
>>
>> on branch rawhide:
>> $ fedpkg build --target=f35-python
>> $ koji wait-repo f35-python --build 
>>
>> Note that it will take a while before all the essential packages are rebuilt,
>> so don't expect all your dependencies to be available right away. When in
>> trouble, ask here or on IRC (#fedora-python on Libera.Chat).
>>
>> Builds:
>> 
https://koji.fedoraproject.org/koji/builds?latest=0&tagID=f35-python&order=-build_id&inherited=0
>> 

>>
>>
>> Please avoid any potentially disturbing changes in Python packages until the
>> rebuild is over.
>>
>> Thanks. Let us know if you have any questions.
>
> The side tag has been merged into Rawhide.

And we have a successful compose, so Python 3.10 is now the main Python on 
Rawhide.

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
python-devel mailing list -- python-de...@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-de...@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure




--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag

2021-06-14 Thread Victor Stinner

Congrats, that's amazing! :-) Let's fix remaining broken packages!

Victor

On Sun, Jun 13, 2021 at 9:52 PM Miro Hrončok  wrote:
>
> On 08. 06. 21 1:01, Miro Hrončok wrote:
> > On 02. 06. 21 10:02, Tomas Hrnciar wrote:
> >> Hello, in order to deliver Python 3.10, we are running a coordinated 
> >> rebuild
> >> in a side tag.
> >>
> >> https://fedoraproject.org/wiki/Changes/Python3.10
> >> 
> >>
> >> If you see a "Rebuilt for Python 3.10" (or similar) commit in your package,
> >> please don't rebuild it in regular rawhide.
> >> If you need to, please let usknow, so we can coordinate.
> >>
> >> If you'd like to build the package, you should be able to build it in the
> >> side tag via:
> >>
> >> on branch rawhide:
> >> $ fedpkg build --target=f35-python
> >> $ koji wait-repo f35-python --build 
> >>
> >> Note that it will take a while before all the essential packages are 
> >> rebuilt,
> >> so don't expect all your dependencies to be available right away. When in
> >> trouble, ask here or on IRC (#fedora-python on Libera.Chat).
> >>
> >> Builds:
> >> https://koji.fedoraproject.org/koji/builds?latest=0&tagID=f35-python&order=-build_id&inherited=0
> >> 
> >>
> >>
> >> Please avoid any potentially disturbing changes in Python packages until 
> >> the
> >> rebuild is over.
> >>
> >> Thanks. Let us know if you have any questions.
> >
> > The side tag has been merged into Rawhide.
>
> And we have a successful compose, so Python 3.10 is now the main Python on 
> Rawhide.
>
> --
> Miro Hrončok
> --
> Phone: +420777974800
> IRC: mhroncok
> ___
> python-devel mailing list -- python-de...@lists.fedoraproject.org
> To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/python-de...@lists.fedoraproject.org
> Do not reply to spam on the list, report it: 
> https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Fedora-Cloud-34-20210614.0 compose check report

2021-06-14 Thread Fedora compose checker

No missing expected images.

Failed openQA tests: 8/8 (x86_64)

Old failures (same test failed in Fedora-Cloud-34-20210613.0):

ID: 907440  Test: x86_64 Cloud_Base-qcow2-qcow2 base_package_install_remove
URL: https://openqa.fedoraproject.org/tests/907440
ID: 907441  Test: x86_64 Cloud_Base-qcow2-qcow2 base_service_manipulation
URL: https://openqa.fedoraproject.org/tests/907441
ID: 907442  Test: x86_64 Cloud_Base-qcow2-qcow2 base_reboot_unmount
URL: https://openqa.fedoraproject.org/tests/907442
ID: 907443  Test: x86_64 Cloud_Base-qcow2-qcow2 base_services_start
URL: https://openqa.fedoraproject.org/tests/907443
ID: 907444  Test: x86_64 Cloud_Base-qcow2-qcow2 base_selinux
URL: https://openqa.fedoraproject.org/tests/907444
ID: 907445  Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud
URL: https://openqa.fedoraproject.org/tests/907445
ID: 907446  Test: x86_64 Cloud_Base-qcow2-qcow2 base_system_logging
URL: https://openqa.fedoraproject.org/tests/907446
ID: 907447  Test: x86_64 Cloud_Base-qcow2-qcow2 base_update_cli
URL: https://openqa.fedoraproject.org/tests/907447

Soft failed openQA tests: 1/8 (aarch64)
(Tests completed, but using a workaround for a known bug)

Old soft failures (same test soft failed in Fedora-Cloud-34-20210613.0):

ID: 907452  Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi
URL: https://openqa.fedoraproject.org/tests/907452

Passed openQA tests: 7/8 (aarch64)
-- 
Mail generated by check-compose:
https://pagure.io/fedora-qa/check-compose
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Package maintainer docs: Package Retirement: `git rm` all files in the other branches

2021-06-14 Thread Vít Ondruch



Dne 14. 06. 21 v 9:07 Otto Urpelainen napsal(a):

Hello,

As part of the effort to move the Package Maintainer docs to 
docs.fedoraproject.org [1], I have been reviewing the existing wiki 
documentation. I would like to ask for insight on the following item:


Page _How to remove a package at end of life_ [2] first says that 
retirement is done by running 'fedpkg retire', then later alludes to a 
special case:


> 'git rm' all files in the other branches *only if* there are special 
factors at work, like licensing issues, or package being removed 
completely from Fedora.


I understand this to mean that in special conditions, the package 
cannot remain behind even in stable or end-of-life Fedora releases. 
However, I do not understand why 'git rm' is mandated instead of 
'fedpkg retire' that is used in the normal case — is the 
'dead.package' tombstone somehow unwanted in this case? I cannot 
understand why it would be.



We were using `git rm` for everything prior `fedpkg retire` was 
introduced. Isn't it just some residue? The Wiki history could help.






Also, if the intent is to get rid of the package completely, should 
not adding it to fedora-obsolete-packages be required as well? And 
would that even work for end-of-life releases?



fedora-obsolete-packages is rather recent development, so probably 
nobody thought of it yet.



Vít





Lastly, even if this complete removal case is rare, it seems to be 
important enough to have its own process description. Should it be 
given its own page, "Package Withdrawal Process" or "Package Removal 
Process" or such, with content like "1) retire the package 2) do these 
additional steps"?


Thank you,
Otto

[1]: 
https://pagure.io/fork/oturpe/fedora-docs/package-maintainer-docs/tree/wiki-import
[2]: 
https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: [HEADS UP] Fedora 35 Python 3.10 rebuilds have started in a side tag