Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 31. 03. 21 21:52, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/RPM-4.17 == Summary == Update RPM to the [https://rpm.org/wiki/Releases/4.17.0 4.17] release. == Owner == * Name: [[User:pmatilai|Panu Matilainen]] * Email: [pmati...@redhat.com] == Detailed Description == RPM 4.17 contains numerous improvements over previous versions * More robust install failure handling * Many macro improvements, in particular much improved Lua integration * Strict checking for unpackaged content in builds * Libraries no longer need executable permission for dependency generation and is automatically removed for non-executable libraries * Long needed transaction APIs enhancements * Improved documentation Panu, in case this was lost in the enthusiastic discussion about the behavior of %exclude, I just wanted to say that I am very exited about all the other new stuff in this release, particularly the Lua stuff. Thanks! -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/8/21 10:17 AM, Panu Matilainen wrote: On 4/7/21 4:16 PM, Panu Matilainen wrote: On 4/7/21 2:21 PM, Tomas Orsava wrote: On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. Nod. I'll need to sleep on this all a bit. There was good feedback and constructive ideas in the PR. I don't have the energy or the cycles to deal with all this now so I reverted the %exclude-changes upstream, to be revisited at a better time, along with some better options for %check. Thanks everybody for keeping this civil and constructive. Thank you too, Panu, for taking our feedback into account! Best of luck to you and our beloved RPM! Tomas - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 4:16 PM, Panu Matilainen wrote: On 4/7/21 2:21 PM, Tomas Orsava wrote: On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. Nod. I'll need to sleep on this all a bit. There was good feedback and constructive ideas in the PR. I don't have the energy or the cycles to deal with all this now so I reverted the %exclude-changes upstream, to be revisited at a better time, along with some better options for %check. Thanks everybody for keeping this civil and constructive. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 5:14 PM, Vít Ondruch wrote:
Dne 07. 04. 21 v 11:05 Panu Matilainen napsal(a):
On 4/7/21 11:45 AM, Panu Matilainen wrote:
On 4/6/21 7:40 PM, Vít Ondruch wrote:
Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a):
On 4/6/21 1:36 PM, Miro Hrončok wrote:
For example, what is common for Python "namepsace" packages, e.g.
pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set
PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure
the files
we actually ship are working and the installed file set is
complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the
tests.
3) The files from (2) must be excluded from the package because
*pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
This seems like a fairly exotic case to me - okay, a
Python-peculiar problem. And a problem of stepping (not saying
crossing, because I'm not sure it is) on the boundaries of the
%check use-case I suppose.
%ghost'ing the files might be one option - I don't know about the
Ruby cache case beyond that there is one.
There is only `%{gem_cache}` (I assume it was mentioned in the
context of `%exclude`, because that has nothing to do with testing).
Not shipping this file is enough and we don't ship it just because
we don't want to ship file which looks like upstream file but it is
not the original upstream file. Moreover we don't really need it for
the purposes it is used by upstream, which is restoring the original
state of the library.
Ah, okay, it's an entirely different case then. Does this file ever
get created when running the installed software (by root)? If so,
then %ghost'ing it would seem to be the right thing.
The chances to have this file created are pretty slim end even if it was
created, it would not be trustworthy.
If they can get created (even if rare) then you'd want it to be removed
along with the package. Which having them %ghost'ed would achieve.
If it's just something that needs to be scrubbed on each and every
ruby package ever built, we could always add a buildroot policy for
it. Just like we could automatically clean .la files rather than
manually clean them in thousands of packages...
Is it correct assumption that any "random" packages can ship BRP?
Interesting idea, I have to give it thought. I guess the concern would
be that the packages for the most recent Fedora/RPM won't be compatible
with older releases without the BRP. But it should be possible to
backport it I guess.
Anybody can ship BRP but there's no drop-in mechanism to get then to
execute atm, you have to override %__os_install_post for that. So in
practise BRPs need to be enabled via central macros in
redhat-rpm-config, but there's no reason domain-specific BRPs could not
be enabled from there (even if the actual BRP lives in another package).
- Panu -
Vít
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Panu Matilainen wrote: > I'm starting to think the right thing to do is to move %check to run > after %build rather than %install. Here's one thing that would be affected by such a change: https://docs.fedoraproject.org/en-US/packaging-guidelines/Ada/#_runpaths Our Ada packaging policy has a requirement to run check-rpaths at the end of %check to guard against runpaths slipping into packaged files. It was added on the FPC's request. They wanted it to run as late as possible. Here's the discussion for reference: https://meetbot.fedoraproject.org/teams/fpc/fpc.2013-12-19-17.02.log.html check-rpaths checks files in RPM_BUILD_ROOT, so it would have to be moved to the end of %install if %check would run before %install. I'm aware that there's a proposal to run check-rpaths automatically on all packages: https://pagure.io/packaging-committee/issue/886 I think that's a good idea. If it gets implemented, then we can remove check-rpaths from the Ada spec files – but there might be other similar usecases where something runs in %check to check files in the buildroot, which would break if %check would be moved before %install. Björn Persson pgpfJE5qbRvw7.pgp Description: OpenPGP digital signatur ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 07. 04. 21 v 11:05 Panu Matilainen napsal(a):
On 4/7/21 11:45 AM, Panu Matilainen wrote:
On 4/6/21 7:40 PM, Vít Ondruch wrote:
Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a):
On 4/6/21 1:36 PM, Miro Hrončok wrote:
For example, what is common for Python "namepsace" packages, e.g.
pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set
PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure
the files
we actually ship are working and the installed file set is
complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the
tests.
3) The files from (2) must be excluded from the package because
*pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
This seems like a fairly exotic case to me - okay, a
Python-peculiar problem. And a problem of stepping (not saying
crossing, because I'm not sure it is) on the boundaries of the
%check use-case I suppose.
%ghost'ing the files might be one option - I don't know about the
Ruby cache case beyond that there is one.
There is only `%{gem_cache}` (I assume it was mentioned in the
context of `%exclude`, because that has nothing to do with testing).
Not shipping this file is enough and we don't ship it just because
we don't want to ship file which looks like upstream file but it is
not the original upstream file. Moreover we don't really need it for
the purposes it is used by upstream, which is restoring the original
state of the library.
Ah, okay, it's an entirely different case then. Does this file ever
get created when running the installed software (by root)? If so,
then %ghost'ing it would seem to be the right thing.
The chances to have this file created are pretty slim end even if it was
created, it would not be trustworthy.
If it's just something that needs to be scrubbed on each and every
ruby package ever built, we could always add a buildroot policy for
it. Just like we could automatically clean .la files rather than
manually clean them in thousands of packages...
Is it correct assumption that any "random" packages can ship BRP?
Interesting idea, I have to give it thought. I guess the concern would
be that the packages for the most recent Fedora/RPM won't be compatible
with older releases without the BRP. But it should be possible to
backport it I guess.
Vít
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 2:21 PM, Tomas Orsava wrote: On 4/7/21 12:45 PM, Panu Matilainen wrote: On 4/7/21 1:12 PM, Tomas Orsava wrote: On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Isn't that exactly what we're doing here? The %check PR may or may not go in, it was linked here so people have a chance to discuss it. Because there's this just discovered linkage between %exclude and %check uses, it probably makes sense for them to go in together. Which could mean doing both, or neither, in 4.17. As in, reverting the %exclude change in 4.17 is entirely possible if it makes sense in the grand scheme things. We cannot possibly know what all the tens of thousands of packages out there are doing, and people will only ever wake up when the change is about to hit them. We've done a dozens of similar changes not because it's fun or because I enjoy getting yelled at for breaking their stuff but because there's no other way to fix ambiguity than making it unambiguous. You're right, knowing the scope of the breakage in advance is just not possible. We have similar problems with Python, but given the nature of RPM, you've got it even tougher. I really apologize if it felt like yelling, that was not my intention. No worries. Didn't mean to imply *you* were yelling, this has been entirely civil as far as I'm concerned. I was just referring to years of battling these issues: for every little loophole in rpm we close in order to make it saner we get a backslash, and not always so civilized. It just gets tiresome sometimes. I'm sure you know the feeling. I myself was kicking and screaming through most of the Python 3 transition, so been on the other side of it too :) However, I hope you'll understand why I felt a certain urgency when writing my response. From my own packaging experience the change would break a lot of usage, and other people on this thread only reinforced that this would be a major problem in many areas. And so when I saw that the follow up to this discussion was a PR that would break even more use cases, I grew worried. Yes, message heard. This has been an important discussion to have because it revealed various things I had no idea about, in particular the %exclude linkage to %check. On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. Nod. I'll need to sleep on this all a bit. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 12:45 PM, Panu Matilainen wrote: On 4/7/21 1:12 PM, Tomas Orsava wrote: On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Isn't that exactly what we're doing here? The %check PR may or may not go in, it was linked here so people have a chance to discuss it. Because there's this just discovered linkage between %exclude and %check uses, it probably makes sense for them to go in together. Which could mean doing both, or neither, in 4.17. As in, reverting the %exclude change in 4.17 is entirely possible if it makes sense in the grand scheme things. We cannot possibly know what all the tens of thousands of packages out there are doing, and people will only ever wake up when the change is about to hit them. We've done a dozens of similar changes not because it's fun or because I enjoy getting yelled at for breaking their stuff but because there's no other way to fix ambiguity than making it unambiguous. You're right, knowing the scope of the breakage in advance is just not possible. We have similar problems with Python, but given the nature of RPM, you've got it even tougher. I really apologize if it felt like yelling, that was not my intention. However, I hope you'll understand why I felt a certain urgency when writing my response. From my own packaging experience the change would break a lot of usage, and other people on this thread only reinforced that this would be a major problem in many areas. And so when I saw that the follow up to this discussion was a PR that would break even more use cases, I grew worried. On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. All the best, Tomas - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 1:12 PM, Tomas Orsava wrote: On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Isn't that exactly what we're doing here? The %check PR may or may not go in, it was linked here so people have a chance to discuss it. Because there's this just discovered linkage between %exclude and %check uses, it probably makes sense for them to go in together. Which could mean doing both, or neither, in 4.17. As in, reverting the %exclude change in 4.17 is entirely possible if it makes sense in the grand scheme things. We cannot possibly know what all the tens of thousands of packages out there are doing, and people will only ever wake up when the change is about to hit them. We've done a dozens of similar changes not because it's fun or because I enjoy getting yelled at for breaking their stuff but because there's no other way to fix ambiguity than making it unambiguous. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Tomas - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 11:45 AM, Panu Matilainen wrote:
On 4/6/21 7:40 PM, Vít Ondruch wrote:
Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a):
On 4/6/21 1:36 PM, Miro Hrončok wrote:
For example, what is common for Python "namepsace" packages, e.g.
pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set
PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure
the files
we actually ship are working and the installed file set is
complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the
tests.
3) The files from (2) must be excluded from the package because
*pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
This seems like a fairly exotic case to me - okay, a Python-peculiar
problem. And a problem of stepping (not saying crossing, because I'm
not sure it is) on the boundaries of the %check use-case I suppose.
%ghost'ing the files might be one option - I don't know about the
Ruby cache case beyond that there is one.
There is only `%{gem_cache}` (I assume it was mentioned in the context
of `%exclude`, because that has nothing to do with testing). Not
shipping this file is enough and we don't ship it just because we
don't want to ship file which looks like upstream file but it is not
the original upstream file. Moreover we don't really need it for the
purposes it is used by upstream, which is restoring the original state
of the library.
Ah, okay, it's an entirely different case then. Does this file ever get
created when running the installed software (by root)? If so, then
%ghost'ing it would seem to be the right thing.
If it's just something that needs to be scrubbed on each and every ruby
package ever built, we could always add a buildroot policy for it. Just
like we could automatically clean .la files rather than manually clean
them in thousands of packages...
But since Ruby was mentioned there, we generally run test suite in
`%{_builddir}` (and there are (unfortunately) two possible location,
while only one is preferred). Generally, it could be run in
`%{buildroot}` with similar results, but it should be discouraged due
to possible `%{buildroot}` pollution.
Ok, good. Indeed, buildroot should not be used by %check because it can
and often does have side-effects.
I'm starting to think the right thing to do is to move %check to run
after %build rather than %install. That would completely eliminate
arguments over what is proper use and should this or that be done etc.
https://github.com/rpm-software-management/rpm/pull/1618
- Panu -
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/6/21 7:40 PM, Vít Ondruch wrote:
Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a):
On 4/6/21 1:36 PM, Miro Hrončok wrote:
For example, what is common for Python "namepsace" packages, e.g.
pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set
PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure the
files
we actually ship are working and the installed file set is complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the tests.
3) The files from (2) must be excluded from the package because
*pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
This seems like a fairly exotic case to me - okay, a Python-peculiar
problem. And a problem of stepping (not saying crossing, because I'm
not sure it is) on the boundaries of the %check use-case I suppose.
%ghost'ing the files might be one option - I don't know about the Ruby
cache case beyond that there is one.
There is only `%{gem_cache}` (I assume it was mentioned in the context
of `%exclude`, because that has nothing to do with testing). Not
shipping this file is enough and we don't ship it just because we don't
want to ship file which looks like upstream file but it is not the
original upstream file. Moreover we don't really need it for the
purposes it is used by upstream, which is restoring the original state
of the library.
Ah, okay, it's an entirely different case then. Does this file ever get
created when running the installed software (by root)? If so, then
%ghost'ing it would seem to be the right thing.
If it's just something that needs to be scrubbed on each and every ruby
package ever built, we could always add a buildroot policy for it. Just
like we could automatically clean .la files rather than manually clean
them in thousands of packages...
But since Ruby was mentioned there, we generally run test suite in
`%{_builddir}` (and there are (unfortunately) two possible location,
while only one is preferred). Generally, it could be run in
`%{buildroot}` with similar results, but it should be discouraged due to
possible `%{buildroot}` pollution.
Ok, good. Indeed, buildroot should not be used by %check because it can
and often does have side-effects.
I'm starting to think the right thing to do is to move %check to run
after %build rather than %install. That would completely eliminate
arguments over what is proper use and should this or that be done etc.
- Panu -
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a):
On 4/6/21 1:36 PM, Miro Hrončok wrote:
For example, what is common for Python "namepsace" packages, e.g.
pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set
PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure the
files
we actually ship are working and the installed file set is complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the tests.
3) The files from (2) must be excluded from the package because
*pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
This seems like a fairly exotic case to me - okay, a Python-peculiar
problem. And a problem of stepping (not saying crossing, because I'm
not sure it is) on the boundaries of the %check use-case I suppose.
%ghost'ing the files might be one option - I don't know about the Ruby
cache case beyond that there is one.
There is only `%{gem_cache}` (I assume it was mentioned in the context
of `%exclude`, because that has nothing to do with testing). Not
shipping this file is enough and we don't ship it just because we don't
want to ship file which looks like upstream file but it is not the
original upstream file. Moreover we don't really need it for the
purposes it is used by upstream, which is restoring the original state
of the library.
But since Ruby was mentioned there, we generally run test suite in
`%{_builddir}` (and there are (unfortunately) two possible location,
while only one is preferred). Generally, it could be run in
`%{buildroot}` with similar results, but it should be discouraged due to
possible `%{buildroot}` pollution.
Vít
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/6/21 3:01 PM, Tim Landscheidt wrote:
Miro Hrončok wrote:
[…]
2. change %check not to rely on unpackaged files in buildroot
That one is non-trivial and depends on the reason it is needed.
For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure the files
we actually ship are working and the installed file set is complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the tests.
3) The files from (2) must be excluded from the package because *pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
[…]
My understanding of the RPM spec sections was always that:
- "%prep" is for "./configure",
- "%build" is for "make",
- "%install" is for "make install", and
- "%check" is for "make check".
"make check" (usually executed /before/ "make install")
works in and on the working directory
(https://www.gnu.org/prep/standards/html_node/Standard-Targets.html#Standard-Targets).
Exactly. Only rpm got this sequence wrong, because %check is executed
after %install. If it wasn't for that, we wouldn't be having this
conversation at all because then %check would be clearly defined.
For the curious, this is how %check came to existence:
https://bugzilla.redhat.com/show_bug.cgi?id=64137
I'm kinda wondering here if there wouldn't be any solutions to in this
direction - the existing %check is being (ab)used for things its not
well suited to because it's positioned in the way it is, and people be
better off with something different entirely.
To test that the resulting binary package is functional, an-
other venue is needed, and in the case of Fedora, for that
purpose Fedora CI was created
(https://docs.fedoraproject.org/en-US/ci/tests/).
That feels like a much cleaner solution than installing some
files, testing and then not shipping them because the test
environment will be the same as a user's who just installed
the package instead of being in the process of building the
package.
Indeed.
- Panu -
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/6/21 1:36 PM, Miro Hrončok wrote:
On 06. 04. 21 11:16, Panu Matilainen wrote:
On 4/1/21 12:45 AM, Miro Hrončok wrote:
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to
own the files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
This still succeeds:
%files
/
%exclude /foo
%files foo
/foo
Many packages do the former in Fedora for various different reasons,
namely to, well... exclude files from the package (and not ship them
at all). Sometimes a `rm` in %install can be used instead. Sometimes
not, because the files are needed in the %{buildroot} for %check but
not needed to be shipped.
The bottom line is that the buildroot should not contain anything that
is not packaged. This has been the basic premise ever since the check
for unpackaged files was added somewhere around turn of the millenium,
but some loopholes were left around. Yes, %exclude is a loophole.
So 'rm -f' at end of %install for undesired material is the expected fix.
What %check may or may not do has probably never been designed, much
less documented or enforced. It runs after %install yes, so one might
assume it's ok/supposed to use what's in buildroot, but then it runs
from the build directory, not buildroot.
It doesn't matter where does it run *from*, it needs to "see" the files
from %{buildroot} because that is what we want to test: The files we ship.
The way I see it, %check should be able to use/access buildroot
contents, but it certainly should not write there. That might be
something to even enforce one day.
The packages (that I know about) don't need to actually write there in
%check. They just need to to see the files that will be later excluded
(e.g. because they belong to a different package that the
soon-to-be-built package requires on runtime).
And therein lies the problem. Like many things in rpm, %check isn't
actually defined in any meaningful way, but in practise it's kinda
limited to standalone unit/functional testing and this steps on the
border of integration testing.
When this change was introduced upstream in November 2020, I've
analyzed the impact on Fedora packages. Bare in mind that the data is
4+ months old.
https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917
- 1675 packages had %exclude in the spec file
- 261 packages FTBFS for unrelated reason (incl. a very limited
timeout)
- 1414 packages actually tested
- 537 packages built successfully, that is ~38%
- 877 packages failed with unpackaged files, that is ~62%, list
attached
When I extrapolate the numbers to compensate the unrelated FTBFS,
that's likely more than 1000 affected Fedora packages.
OTOH ~500 packages are generated rubygem-* packages, automatically
fixable.
I'd like to know how are the affected packages supposed to migrate to
RPM 4.17 behavior, especially if they cannot remove the files in
%install prior to %check. Are they supposed to remove the files at
the end of %check instead? What if the package is build without %check?
In the order of preference:
1. 'rm -f ' at end of %install
That one is simple. I agree. If it is possible, let's do this. We could
even automate it if needed. (As a side note I'd say writing such
automation is a right thing to do when a change like this is proposed,
but I understand that we cannot have everything.)
2. change %check not to rely on unpackaged files in buildroot
That one is non-trivial and depends on the reason it is needed.
Of course.
For example, what is common for Python "namepsace" packages, e.g.
pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set
PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure the files
we actually ship are working and the installed file set is complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the tests.
3) The files from (2) must be excluded from the package because
*pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Miro Hrončok wrote:
> […]
>> 2. change %check not to rely on unpackaged files in buildroot
> That one is non-trivial and depends on the reason it is needed.
> For example, what is common for Python "namepsace" packages, e.g.
> pkg_name.foo.
> 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH
> to
>%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
>(try hard to) exclude $PWD from it. This is crucial to ensure the files
>we actually ship are working and the installed file set is complete.
>Our macros do this for the packagers.
> 2) The %{python3_site...}/pkg_name/ directory and
>%{python3_site...}/pkg_name/__init__.py and
>%{python3_site...}/pkg_name/__pycache__/ and
>%{python3_site...}/pkg_name/__pycache__/__init__...pyc
>files must be present in %{buildroot} to successfully run the tests.
> 3) The files from (2) must be excluded from the package because *pkg_name*
> owns
>them, not *pkg_name.foo*.
>We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
>The files are not bit-by-bit+metadata identical,
>so both packages cannot ship them.
> […]
My understanding of the RPM spec sections was always that:
- "%prep" is for "./configure",
- "%build" is for "make",
- "%install" is for "make install", and
- "%check" is for "make check".
"make check" (usually executed /before/ "make install")
works in and on the working directory
(https://www.gnu.org/prep/standards/html_node/Standard-Targets.html#Standard-Targets).
To test that the resulting binary package is functional, an-
other venue is needed, and in the case of Fedora, for that
purpose Fedora CI was created
(https://docs.fedoraproject.org/en-US/ci/tests/).
That feels like a much cleaner solution than installing some
files, testing and then not shipping them because the test
environment will be the same as a user's who just installed
the package instead of being in the process of building the
package.
Tim
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 06. 04. 21 11:16, Panu Matilainen wrote:
On 4/1/21 12:45 AM, Miro Hrončok wrote:
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to own the
files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
This still succeeds:
%files
/
%exclude /foo
%files foo
/foo
Many packages do the former in Fedora for various different reasons, namely
to, well... exclude files from the package (and not ship them at all).
Sometimes a `rm` in %install can be used instead. Sometimes not, because the
files are needed in the %{buildroot} for %check but not needed to be shipped.
The bottom line is that the buildroot should not contain anything that is not
packaged. This has been the basic premise ever since the check for unpackaged
files was added somewhere around turn of the millenium, but some loopholes were
left around. Yes, %exclude is a loophole.
So 'rm -f' at end of %install for undesired material is the expected fix.
What %check may or may not do has probably never been designed, much less
documented or enforced. It runs after %install yes, so one might assume it's
ok/supposed to use what's in buildroot, but then it runs from the build
directory, not buildroot.
It doesn't matter where does it run *from*, it needs to "see" the files from
%{buildroot} because that is what we want to test: The files we ship.
The way I see it, %check should be able to use/access buildroot contents, but it
certainly should not write there. That might be something to even enforce one day.
The packages (that I know about) don't need to actually write there in %check.
They just need to to see the files that will be later excluded (e.g. because
they belong to a different package that the soon-to-be-built package requires on
runtime).
When this change was introduced upstream in November 2020, I've analyzed the
impact on Fedora packages. Bare in mind that the data is 4+ months old.
https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917
- 1675 packages had %exclude in the spec file
- 261 packages FTBFS for unrelated reason (incl. a very limited timeout)
- 1414 packages actually tested
- 537 packages built successfully, that is ~38%
- 877 packages failed with unpackaged files, that is ~62%, list attached
When I extrapolate the numbers to compensate the unrelated FTBFS, that's
likely more than 1000 affected Fedora packages.
OTOH ~500 packages are generated rubygem-* packages, automatically fixable.
I'd like to know how are the affected packages supposed to migrate to RPM 4.17
behavior, especially if they cannot remove the files in %install prior to
%check. Are they supposed to remove the files at the end of %check instead?
What if the package is build without %check?
In the order of preference:
1. 'rm -f ' at end of %install
That one is simple. I agree. If it is possible, let's do this. We could even
automate it if needed. (As a side note I'd say writing such automation is a
right thing to do when a change like this is proposed, but I understand that we
cannot have everything.)
2. change %check not to rely on unpackaged files in buildroot
That one is non-trivial and depends on the reason it is needed.
For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo.
1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to
%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we
(try hard to) exclude $PWD from it. This is crucial to ensure the files
we actually ship are working and the installed file set is complete.
Our macros do this for the packagers.
2) The %{python3_site...}/pkg_name/ directory and
%{python3_site...}/pkg_name/__init__.py and
%{python3_site...}/pkg_name/__pycache__/ and
%{python3_site...}/pkg_name/__pycache__/__init__...pyc
files must be present in %{buildroot} to successfully run the tests.
3) The files from (2) must be excluded from the package because *pkg_name* owns
them, not *pkg_name.foo*.
We Require the "toplevel" *pkg_name* package from *pkg_name.foo*.
The files are not bit-by-bit+metadata identical,
so both packages cannot ship them.
Or, let's assume I want to package libfoo-devel for EPEL 9 because RHEL 9 only
ships libfoo. And I want to run %check, but only ship the headers. There are
plenty situations like this.
The case-by-case fix is also impossible to do at scale without a huge heroic
effort.
3. short-term,
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 02. 04. 21 v 12:07 Zbigniew Jędrzejewski-Szmek napsal(a): On Thu, Apr 01, 2021 at 12:33:48AM +0200, Kalev Lember wrote: On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek < zbys...@in.waw.pl> wrote: On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. Thank you for the explanation. I believe the motivation for that change is brp scripts that would still see the files that are %excluded in files and possibly do wrong things. Using rm in install doesn't have that problem. That sounds like a plausible concern. But is this something that actually caused real problems, or just a theoretical issue? This is one of the real issues: https://bugzilla.redhat.com/show_bug.cgi?id=878863 Vít For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). If the system was designed like this initially, that'd be fine. But this pattern is widely used and up-to-now there was no indication in the packaging guidelines or rpm output that this is not the recommended pattern. In fact, I know some people preferred the (declarative) %exclude over the (imperative) rm. And Miro raises another issue upthread: there might be packages which require files in %check, and %exclude them. This change would require removing those files at the end of %check, which is rather ugly. Right we have ~1000 packages which will break. There is also an unknown number of non-distro packages which may be affected. I'm not happy about how rpm is changing in a backwards incompatible way. E.g. this means that suddenly ~5% of F33 will not rebuild on F35. I think we should have a strong justification for such a change. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/1/21 1:33 AM, Kalev Lember wrote: On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek mailto:zbys...@in.waw.pl>> wrote: On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > On 31. 03. 21 21:52, Ben Cotton wrote: > >* Strict checking for unpackaged content in builds > > ... > >* Many existing packages will fail to build due to the stricter > >buildroot content checking. Fixing this in the packaging is always > >backwards compatible. We could temporarily set > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > >impact if necessary. > > This is my main concern with this update. > > tl;dr If you %exclude something and there is no other subpackage to > own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. Indeed. For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). I believe the motivation for that change is brp scripts that would still see the files that are %excluded in files and possibly do wrong things. Using rm in install doesn't have that problem. More generally: what is not there can not cause unwanted side-effects. File-level exclusion is impossible to meaningfully communicate to externally executed scripts, including but not limited to those shipping with rpm itself. There are other benefits further down the road, such as automatic sub-packaging, enforcing %check will not muck with buildroot contents etc. - Panu - -- Kalev ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/1/21 12:45 AM, Miro Hrončok wrote:
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to own
the files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
This still succeeds:
%files
/
%exclude /foo
%files foo
/foo
Many packages do the former in Fedora for various different reasons,
namely to, well... exclude files from the package (and not ship them at
all). Sometimes a `rm` in %install can be used instead. Sometimes not,
because the files are needed in the %{buildroot} for %check but not
needed to be shipped.
The bottom line is that the buildroot should not contain anything that
is not packaged. This has been the basic premise ever since the check
for unpackaged files was added somewhere around turn of the millenium,
but some loopholes were left around. Yes, %exclude is a loophole.
So 'rm -f' at end of %install for undesired material is the expected fix.
What %check may or may not do has probably never been designed, much
less documented or enforced. It runs after %install yes, so one might
assume it's ok/supposed to use what's in buildroot, but then it runs
from the build directory, not buildroot.
The way I see it, %check should be able to use/access buildroot
contents, but it certainly should not write there. That might be
something to even enforce one day.
When this change was introduced upstream in November 2020, I've analyzed
the impact on Fedora packages. Bare in mind that the data is 4+ months old.
https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917
- 1675 packages had %exclude in the spec file
- 261 packages FTBFS for unrelated reason (incl. a very limited timeout)
- 1414 packages actually tested
- 537 packages built successfully, that is ~38%
- 877 packages failed with unpackaged files, that is ~62%, list attached
When I extrapolate the numbers to compensate the unrelated FTBFS, that's
likely more than 1000 affected Fedora packages.
OTOH ~500 packages are generated rubygem-* packages, automatically fixable.
I'd like to know how are the affected packages supposed to migrate to
RPM 4.17 behavior, especially if they cannot remove the files in
%install prior to %check. Are they supposed to remove the files at the
end of %check instead? What if the package is build without %check?
In the order of preference:
1. 'rm -f ' at end of %install
2. change %check not to rely on unpackaged files in buildroot
3. short-term, "%_unpackaged_files_terminate_build 0" in the spec with
explanation (similar to eg unsupported architectures)
2. would be case-by-case of course, but an "universal" solution is to
create a private install root inside %check, eg
---
%check
export CHECKROOT=${PWD}/_check
%make_install DESTDIR=${CHECKROOT}
# ...do what you normally do, just replace BUILDROOT with CHECKROOT
---
This way %check will not be messing with the precious to-be-packaged
contents.
More light-weight options will exist on case-by-case basis, eg typically
a cache can be diverted to some other location with environment
variables or such.
Using `s` in entire rawhide just to
compensate this:
- is dangerous for other implications of the setting
- only postpones the problem to a later time (when we will face the
same issue)
It's hardly "dangerous", because the only content that will "leak"
because of it is buildid links, which is what happens now. It doesn't
affect content inclusion, just whether the message is a warning or error.
And for a more specific problem, around ~100 Python packages were
affected when tested, many of them crucial (e.g. dnf), so this problem
will block the upgrade to Python 3.10 if the change lands in Rawhide
before we upgrade Python (which is the current plan) until we fix all
the affected packages (by at least adding `%global
_unpackaged_files_terminate_build 0` to them, which is a tad big hammer,
but it will be our last-resort option).
Which is why suggested the global "%_unpackaged_files_terminate_build 0"
in rawhide: just to move the impact to a less inconvenient time. No
kittens will be killed by doing so.
- Panu -
List of affected Python packages:
https://pagure.io/releng/issue/10072#comment-724315
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Fri, Apr 2, 2021 at 3:34 AM Richard W.M. Jones wrote: > On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: > > * Libraries no longer need executable permission for dependency > > generation and is automatically removed for non-executable libraries > > That's interesting - something which affected OCaml packaging: > > https://github.com/ocaml/dune/issues/4148 In a twist of irony, the version of dune that fixes your bug report is now available in Rawhide and F34, so dune no longer removes the executable bits that rpm no longer requires to be present. :-) -- Jerry James http://www.jamezone.org/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Thu, Apr 01, 2021 at 12:33:48AM +0200, Kalev Lember wrote: > On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek < > zbys...@in.waw.pl> wrote: > > > On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > > > On 31. 03. 21 21:52, Ben Cotton wrote: > > > >* Strict checking for unpackaged content in builds > > > > ... > > > >* Many existing packages will fail to build due to the stricter > > > >buildroot content checking. Fixing this in the packaging is always > > > >backwards compatible. We could temporarily set > > > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > > > >impact if necessary. > > > > > > This is my main concern with this update. > > > > > > tl;dr If you %exclude something and there is no other subpackage to > > > own the files, the build fails: > > > > Whaaat? What is the point of %exclude if not to exclude files from the > > list? Why would rpm upstream want to break this? Seems like a completely > > backwards change that will make packaging harder instead of easier. > > > > %exclude can be used for splitting up packages, so you can do > > %files foo > %exclude bar.so > *.so > > %files bar > bar.so > > > If my understanding is right, the above is what rpm upstream considers > correct use for %exclude. Thank you for the explanation. > I believe the motivation for that change is brp scripts that would still > see the files that are %excluded in files and possibly do wrong things. > Using rm in install doesn't have that problem. That sounds like a plausible concern. But is this something that actually caused real problems, or just a theoretical issue? > For just not packaging some files, rm at the end of %install usually works > just fine (but people have also been using %exclude for that and this > change would break a bunch of packages that do this. I'm unsure if it's a > good thing or not). If the system was designed like this initially, that'd be fine. But this pattern is widely used and up-to-now there was no indication in the packaging guidelines or rpm output that this is not the recommended pattern. In fact, I know some people preferred the (declarative) %exclude over the (imperative) rm. And Miro raises another issue upthread: there might be packages which require files in %check, and %exclude them. This change would require removing those files at the end of %check, which is rather ugly. Right we have ~1000 packages which will break. There is also an unknown number of non-distro packages which may be affected. I'm not happy about how rpm is changing in a backwards incompatible way. E.g. this means that suddenly ~5% of F33 will not rebuild on F35. I think we should have a strong justification for such a change. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Thu, Apr 01, 2021 at 11:26:04AM +0200, Miro Hrončok wrote:
> On 01. 04. 21 10:47, Vít Ondruch wrote:
> >
> >Dne 01. 04. 21 v 0:54 Mamoru TASAKA napsal(a):
> >>Hello:
> >>
> >>Miro Hrončok wrote on 2021/04/01 6:45:
> >>>On 31. 03. 21 21:52, Ben Cotton wrote:
> * Strict checking for unpackaged content in builds
> >>> > ...
> * Many existing packages will fail to build due to the stricter
> buildroot content checking. Fixing this in the packaging is always
> backwards compatible. We could temporarily set
> `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
> impact if necessary.
> >>>
> >>>This is my main concern with this update.
> >>>
> >>>tl;dr If you %exclude something and there is no other
> >>>subpackage to own the files, the build fails:
> >>>
> >>>
> >>>This fails:
> >>>
> >>> %install
> >>> ...
> >>> touch %{buildroot}/foo %{buildroot}/bar
> >>>
> >>> %files
> >>> /
> >>> %exclude /foo
> >>
> >>As the files Miro has attached shows, this affects not a few rubygems
> >>related
> >>packages. Many rubygems related packages has: %exclude %gem_cache .
> >
> >
> >Just FTR, as a Ruby maintainer and gem2rpm maintainer, I am well
> >aware of this change and believe me or not, I support the
> >intention, mainly because it avoids unintentional side-effects.
> >
> >However, so far I have not figured alternative (should be probably
> >read as elegant) way to do this. Maybe we should generate some
> >file lists for the packages and remove the selected files from the
> >FS as well as from the file list. Dunno.
>
> Yeah, I have no problem with "using %exclude like this is wrong and
> it was never intended to be abused in this way" but I miss the "this
> is how to do it properly" migration guide.
Wouldn't the sensible thing be to introduce another keyword to mean
that files should not be packaged, eg. %ignore ?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: > * Libraries no longer need executable permission for dependency > generation and is automatically removed for non-executable libraries That's interesting - something which affected OCaml packaging: https://github.com/ocaml/dune/issues/4148 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/1/21 12:33 AM, Kalev Lember wrote: On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek mailto:zbys...@in.waw.pl>> wrote: On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > On 31. 03. 21 21:52, Ben Cotton wrote: > >* Strict checking for unpackaged content in builds > > ... > >* Many existing packages will fail to build due to the stricter > >buildroot content checking. Fixing this in the packaging is always > >backwards compatible. We could temporarily set > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > >impact if necessary. > > This is my main concern with this update. > > tl;dr If you %exclude something and there is no other subpackage to > own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). This is what I have enforced when reviewing packages, except for Rubygems where the problem is endemic. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 01. 04. 21 10:47, Vít Ondruch wrote:
Dne 01. 04. 21 v 0:54 Mamoru TASAKA napsal(a):
Hello:
Miro Hrončok wrote on 2021/04/01 6:45:
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to own the
files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
As the files Miro has attached shows, this affects not a few rubygems related
packages. Many rubygems related packages has: %exclude %gem_cache .
Just FTR, as a Ruby maintainer and gem2rpm maintainer, I am well aware of this
change and believe me or not, I support the intention, mainly because it avoids
unintentional side-effects.
However, so far I have not figured alternative (should be probably read as
elegant) way to do this. Maybe we should generate some file lists for the
packages and remove the selected files from the FS as well as from the file
list. Dunno.
Yeah, I have no problem with "using %exclude like this is wrong and it was never
intended to be abused in this way" but I miss the "this is how to do it
properly" migration guide.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 01. 04. 21 v 0:54 Mamoru TASAKA napsal(a):
Hello:
Miro Hrončok wrote on 2021/04/01 6:45:
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to
own the files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
As the files Miro has attached shows, this affects not a few rubygems
related
packages. Many rubygems related packages has: %exclude %gem_cache .
Just FTR, as a Ruby maintainer and gem2rpm maintainer, I am well aware
of this change and believe me or not, I support the intention, mainly
because it avoids unintentional side-effects.
However, so far I have not figured alternative (should be probably read
as elegant) way to do this. Maybe we should generate some file lists for
the packages and remove the selected files from the FS as well as from
the file list. Dunno.
Vít
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Can't wait. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Hello:
Miro Hrončok wrote on 2021/04/01 6:45:
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to own the
files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
As the files Miro has attached shows, this affects not a few rubygems related
packages. Many rubygems related packages has: %exclude %gem_cache .
Regards,
Mamoru
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek < zbys...@in.waw.pl> wrote: > On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > > On 31. 03. 21 21:52, Ben Cotton wrote: > > >* Strict checking for unpackaged content in builds > > > ... > > >* Many existing packages will fail to build due to the stricter > > >buildroot content checking. Fixing this in the packaging is always > > >backwards compatible. We could temporarily set > > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > > >impact if necessary. > > > > This is my main concern with this update. > > > > tl;dr If you %exclude something and there is no other subpackage to > > own the files, the build fails: > > Whaaat? What is the point of %exclude if not to exclude files from the > list? Why would rpm upstream want to break this? Seems like a completely > backwards change that will make packaging harder instead of easier. > %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). I believe the motivation for that change is brp scripts that would still see the files that are %excluded in files and possibly do wrong things. Using rm in install doesn't have that problem. -- Kalev ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote:
> On 31. 03. 21 21:52, Ben Cotton wrote:
> >* Strict checking for unpackaged content in builds
> > ...
> >* Many existing packages will fail to build due to the stricter
> >buildroot content checking. Fixing this in the packaging is always
> >backwards compatible. We could temporarily set
> >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
> >impact if necessary.
>
> This is my main concern with this update.
>
> tl;dr If you %exclude something and there is no other subpackage to
> own the files, the build fails:
Whaaat? What is the point of %exclude if not to exclude files from the
list? Why would rpm upstream want to break this? Seems like a completely
backwards change that will make packaging harder instead of easier.
Zbyszek
> This fails:
>
> %install
> ...
> touch %{buildroot}/foo %{buildroot}/bar
>
> %files
> /
> %exclude /foo
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 31. 03. 21 21:52, Ben Cotton wrote:
* Strict checking for unpackaged content in builds
> ...
* Many existing packages will fail to build due to the stricter
buildroot content checking. Fixing this in the packaging is always
backwards compatible. We could temporarily set
`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial
impact if necessary.
This is my main concern with this update.
tl;dr If you %exclude something and there is no other subpackage to own the
files, the build fails:
This fails:
%install
...
touch %{buildroot}/foo %{buildroot}/bar
%files
/
%exclude /foo
This still succeeds:
%files
/
%exclude /foo
%files foo
/foo
Many packages do the former in Fedora for various different reasons, namely to,
well... exclude files from the package (and not ship them at all). Sometimes a
`rm` in %install can be used instead. Sometimes not, because the files are
needed in the %{buildroot} for %check but not needed to be shipped.
When this change was introduced upstream in November 2020, I've analyzed the
impact on Fedora packages. Bare in mind that the data is 4+ months old.
https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917
- 1675 packages had %exclude in the spec file
- 261 packages FTBFS for unrelated reason (incl. a very limited timeout)
- 1414 packages actually tested
- 537 packages built successfully, that is ~38%
- 877 packages failed with unpackaged files, that is ~62%, list attached
When I extrapolate the numbers to compensate the unrelated FTBFS, that's likely
more than 1000 affected Fedora packages.
OTOH ~500 packages are generated rubygem-* packages, automatically fixable.
I'd like to know how are the affected packages supposed to migrate to RPM 4.17
behavior, especially if they cannot remove the files in %install prior to
%check. Are they supposed to remove the files at the end of %check instead? What
if the package is build without %check?
Using `%_unpackaged_files_terminate_build 0` in entire rawhide just to
compensate this:
- is dangerous for other implications of the setting
- only postpones the problem to a later time (when we will face the same issue)
And for a more specific problem, around ~100 Python packages were affected when
tested, many of them crucial (e.g. dnf), so this problem will block the upgrade
to Python 3.10 if the change lands in Rawhide before we upgrade Python (which is
the current plan) until we fix all the affected packages (by at least adding
`%global _unpackaged_files_terminate_build 0` to them, which is a tad big
hammer, but it will be our last-resort option).
List of affected Python packages:
https://pagure.io/releng/issue/10072#comment-724315
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
afpfs-ng
akmods
annobin
antimicrox
antimicroX
appstream
argyllcms
armacycles-ad
arpwatch
asciidoc
audacious-plugin-fc
audiocd-kio
autoconf
awesome
aws
a52dec
bacula
bakefile
bamf
berusky
bless
blitz
blosc
bogl
booth
botan
calls
cantoolz
CCfits
CGAL
cloog
cockatrice
colin
compizconfig-python
conda
coturn
cowsay
cpl
crrcsim-addon-models
debian-keyring
djvulibre
dnf
docbook5-style-xsl
drupal7-active_tags
drupal7-boxes
drupal7-calendar
drupal7-cck
drupal7-cs_adaptive_image
drupal7-domain_views
drupal7-email
drupal7-file_entity_inline
drupal7-flexifilter
drupal7-i18n_boxes
drupal7-i18nviews
drupal7-languageicons
drupal7-language_switcher
drupal7-locale_auto_import
drupal7-locale_cookie
drupal7-l10n_client
drupal7-l10n_pconfig
drupal7-l10n_server
drupal7-stringoverrides
drupal7-strongarm
drupal7-theme-ninesixty
drupal7-translation_helpers
drupal7-translation_table
drupal7-transliteration
drupal7-workbench
drupal8
drush
dvdbackup
erfa
etckeeper
fbzx
fedora-packager
fence-agents
flare
flaw
fontopia
fwknop
gap-pkg-gbnp
geany
general-purpose-preprocessor
ginga
glabels
glances
glpk
glusterfs
gnome-js-common
gnote
gpaw
gpsim
grads
gsim85
gtk+extra
gutenprint
hatari
HdrHistogram_c
hexter-dssi
chipmunk
icc-profiles-openicc
icu
ikiwiki
ipython
iwd
i3
jack-audio-connection-kit
kbd
knights
kqoauth-qt5
libast
libcaca
libcdaudio
libclaw
libcsv
libdc1394
libdnet
libdrm-armada
libdsk
libdwarf
libfc14audiodecoder
libfilezilla
libgtop2
liblas
libmemcached
libnss-mysql
libnss-pgsql
liboping
liborigin
libpari23
libqmi
LibRaw
libsigrok
libspectrum
libstorj
liburing
libuser
libvdpau-va-gl
libvorbis
libyubikey
libzdb
lighttpd
lxcfs
mhash
milia
mingw-cppunit
mingw-dirac
mingw-fltk
mingw-gsm
mingw-python-lxml
mingw-python-markupsafe
mingw-qt5-qtgraphicaleffects
mingw-qt5-qtserialport
mISDN
moarvm
module-build-service
mom
mon
monitor-edid
moodle
mrtg
nbd-runner
ncmpc
nemiver
netbsd-iscsi
netmask
NetworkManager-strongswan
nfs-ganesha
nickle
nightview
numactl
numpy
ocaml-findlib
odcs
omniORBpy
openal-soft
open-vm-tools
openvpn
openwsman
orafce
osm-gps-map
ots
pacemaker
pacman
pam_mount
parallel
passenger
pcb-rnd
pcs
pen
perl-Algorithm-FastPermute
perl-Apache-Session-NoSQL
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Zbigniew Jędrzejewski-Szmek writes: > On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: >> ** Dynamic spec generation > > Details? My guess would be that this one is meant: https://github.com/rpm-software-management/rpm/pull/1485 signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: > ** Dynamic spec generation Details? Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
https://fedoraproject.org/wiki/Changes/RPM-4.17 == Summary == Update RPM to the [https://rpm.org/wiki/Releases/4.17.0 4.17] release. == Owner == * Name: [[User:pmatilai|Panu Matilainen]] * Email: [pmati...@redhat.com] == Detailed Description == RPM 4.17 contains numerous improvements over previous versions * More robust install failure handling * Many macro improvements, in particular much improved Lua integration * Strict checking for unpackaged content in builds * Libraries no longer need executable permission for dependency generation and is automatically removed for non-executable libraries * Long needed transaction APIs enhancements * Improved documentation * Tentative (planned but not committed as of this writing) ** Split debugedit to its own project and package ** Split language-specific packaging aids to separate projects (Python, Perl, Ocaml...) ** Dynamic spec generation The plan is to get 4.17-alpha into rawhide as early as possible (during April) to sort out any initial rough edges long before the general feature deadline rush. Final version is expected to be released well in time before F35 beta. == Benefit to Fedora == See description for overall benefits, but in particular: * All users benefit from the more robust installation * Packaging sanity wrt libraries * Macro authors will have a much saner experience creating complex macros in Lua * DNF for the enhanced transaction APIs == Scope == * Proposal owners: ** Rebase RPM ** Assist with dealing with incompatibilities * Other developers: ** Test new release, report issues and bugs ** Adjust packaging to adhere to the strict buildroot content checking * Release engineering: [https://pagure.io/releng/issue/10072 #10072] * Policies and guidelines: ** Guidelines have nothing on unpackaged contents in buildroot, so don't necessarily need updating. Many packages will fail to build because of the stricter checking though: with rpm >= 4.17 unpackaged content is not permitted in the buildroot at all. ** Libraries no longer need to be executable for dependency generation, and executable bit will in fact be removed if invalidly set on a library. Guidelines only have a vague "executable if appropriate" mention so it does not *need* changing but could now be clarified/tightened if desired. * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: no relation to current objectives == Upgrade/compatibility impact == * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. * Rpm no longer implicitly creates databases on read-only access, this may require changes to existing scripts/tooling. Ensuring mock/dnf works is a pre-requisite to landing this change into rawhide, and will be handled, one way or the other, by the rpm maintainers. == How To Test == Rpm receives a thorough and constant testing via every single package build, system installs and updates. New features can be tested specifically as per their documentation. == User Experience == The user-experience remains largely as-is, but install failures are handled more gracefully. == Dependencies == * dnf and/or mock will likely need some adjusting for the lack of implicit database creation. If necessary, rpm maintainers will provide patches prior to landing this change. * soname bump is not expected so rebuilds should not be required == Contingency Plan == * Contingency mechanism: Revert back to RPM 4.16, but the risk of having to do should be negligible * Contingency deadline: Beta freeze * Blocks release? No == Documentation == Work-in-progress release notes at https://rpm.org/wiki/Releases/4.17.0 and reference manual at https://github.com/rpm-software-management/rpm/blob/master/doc/manual/index.md -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
