Re: Frequently broken Rawhide/Branched composes

[Vít Ondruch]

Dne 6.3.2018 v 21:36 Kevin Kofler napsal(a):
> Nicolas Mailhot wrote:
>> The “never go backwards” policy means that as soon something hits devel
>> other packages can rely on your package and start adapting
>> their packages on the basis of your changes. You can not pull the carpet
>> from under their feet just because you changed your mind.
> This is entirely orthogonal to the current monotonic EVR policy because 
> Epoch exists (and is used to revert to older upstream releases in Rawhide at 
> times, which is perfectly compliant with the current policy) and because 
> pure packaging changes are not covered at all (given that you can just bump 
> Release again when reverting them).
>
> The policy you would like to see does not exist at this time and would have 
> to be worded entirely differently.
>
> And additionally, I would argue the opposite: It is just impossible to do 
> development without sometimes trying something that may fail and thus have 
> to be reverted. Banning that would also contradict the Changes process (see 
> the contingency plans).

I think that Nicolas is right. In stable version, you have to, at
minimum, submit build override to propagate your library into buildroot.
But in Rawhide everything goes directly into buildroot and things
immediately starts to depend on it.

Vít
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Vít Ondruch]

Dne 6.3.2018 v 19:26 Kevin Fenzi napsal(a):
> On 03/06/2018 07:47 AM, Pierre-Yves Chibon wrote:
>> On Tue, Mar 06, 2018 at 02:58:45PM +, Stephen Gallagher wrote:
> ...snip...
>>> But that has its own issues.
>> Sorry, just to be clear, what would have its own issues:
>> - asking rawhide users to use distro-sync instead of update?
>> - automatically have dnf detect it's running in rawhide and default to
>>   distro-sync instead of update?
>> - or.. ?
> I'll note that this has come up before in the past after we had
> distro-sync, and I tried here to use distro-sync instead of update on my
> rawhide laptop, and I ran into issues. Unfortunately, it's been a while
> so I can't fully recall what the problem was, but I do remember
> distro-sync failed and I had to use update. I can give it a try again
> and see if things are any better.
>
> Do note that distro-sync can downgrade packages, but can't handle all
> the cases. ie, upgrade postgresql and update all your data you can't
> just downgrade the rpm and be fine. Or any number of other scriptlets
> that do things that cannot easily be reversed.

Never really used distro-sync, but this is the difference between
distro-sync and update:

~~~
$ LANG=C.UTF-8 sudo dnf update
Failed to synchronize cache for repo 'rcm-tools-fedora-rpms', disabling.
Last metadata expiration check: 1:08:01 ago on Thu Mar  8 08:45:26 2018.
Dependencies resolved.

 Problem 1: package rpmfusion-free-release-28-0.4.noarch requires
system-release(28), but none of the providers can be installed
  - cannot install both fedora-release-29-0.1.noarch and
fedora-release-28-0.2.noarch
  - package generic-release-28-0.3.fc28.noarch conflicts with
fedora-release provided by fedora-release-29-0.1.noarch
  - cannot install the best update candidate for package
rpmfusion-free-release-28-0.3.noarch
  - cannot install the best update candidate for package
fedora-release-28-0.2.noarch
 Problem 2: package rpmfusion-nonfree-release-28-0.4.noarch requires
system-release(28), but none of the providers can be installed
  - cannot install both fedora-release-29-0.1.noarch and
fedora-release-28-0.2.noarch
  - package generic-release-28-0.3.fc28.noarch conflicts with
fedora-release provided by fedora-release-29-0.1.noarch
  - package fedora-release-workstation-29-0.1.noarch requires
fedora-release = 29-0.1, but none of the providers can be installed
  - cannot install the best update candidate for package
rpmfusion-nonfree-release-28-0.3.noarch
  - cannot install the best update candidate for package
fedora-release-workstation-28-0.2.noarch
 Problem 3: problem with installed package
rpmfusion-nonfree-release-28-0.3.noarch
  - package rpmfusion-nonfree-release-28-0.3.noarch requires
system-release(28), but none of the providers can be installed
  - package rpmfusion-nonfree-release-28-0.4.noarch requires
system-release(28), but none of the providers can be installed
  - package fedora-release-28-0.2.noarch requires fedora-repos(28), but
none of the providers can be installed
  - package generic-release-28-0.3.fc28.noarch requires
fedora-repos(28), but none of the providers can be installed
  - cannot install both fedora-repos-29-0.1.noarch and
fedora-repos-28-0.3.noarch
  - cannot install the best update candidate for package
fedora-repos-28-0.3.noarch
 Problem 4: problem with installed package
rpmfusion-free-release-28-0.3.noarch
  - package rpmfusion-free-release-28-0.3.noarch requires
system-release(28), but none of the providers can be installed
  - package rpmfusion-free-release-28-0.4.noarch requires
system-release(28), but none of the providers can be installed
  - package fedora-release-28-0.2.noarch requires fedora-repos(28), but
none of the providers can be installed
  - package generic-release-28-0.3.fc28.noarch requires
fedora-repos(28), but none of the providers can be installed
  - package fedora-repos-28-0.3.noarch requires fedora-gpg-keys =
28-0.3, but none of the providers can be installed
  - cannot install both fedora-gpg-keys-29-0.1.noarch and
fedora-gpg-keys-28-0.3.noarch
  - cannot install the best update candidate for package
fedora-gpg-keys-28-0.3.noarch
===
 Package    Arch   Version   
Repository  Size
===
Installing:
 kernel x86_64 4.16.0-0.rc3.git4.1.fc29
 
rawhide 88 k
 kernel-core    x86_64 4.16.0-0.rc3.git4.1.fc29
 
rawhide 25 M
 kernel-modules x86_64 4.16.0-0.rc3.git4.1.fc29
 
rawhide 28 M
 kernel-modules-extra   x86_64 4.16.0-0.rc3.git4.1.fc29
  rawhide   
2.3 M
Upgrading:
 doublecmd-gtk  

Re: Frequently broken Rawhide/Branched composes

[Randy Barlow]

On 03/06/2018 03:37 PM, Kevin Kofler wrote:
> But bumping Epoch, as the policy makes you do in such a case, does 
> absolutely nothing to fix that. So I don't see how the current policy helps.

True, but the epoch thing also offers a bit more encouragement to fix
the package without forcing a downgrade, or at least to think more about
it first. If we just have a policy to use distro-sync I would expect
that we might see downgrades happening more often, which could lead to
the data problems.



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Lars Seipel]

On Tue, Mar 06, 2018 at 04:47:49PM +0100, Pierre-Yves Chibon wrote:
> Sorry, just to be clear, what would have its own issues:
> - asking rawhide users to use distro-sync instead of update?
> - automatically have dnf detect it's running in rawhide and default to
>   distro-sync instead of update?

As distro-sync puts all packages back at the repo version, it can also
revert the installation of local rebuilds or bug fix builds
cherry-picked from Koji.

The latter has become pretty common for me in recent weeks, as the
persistent compose issues mean that packages may take a long time
to hit the repos.

Nothing that can't be worked around by judicious use of excludes, you
just have to be aware of it instead of blindly firing off a distro-sync.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Sérgio Basto]

On Tue, 2018-03-06 at 21:30 +0100, Kevin Kofler wrote:
> Kevin Fenzi wrote:
> > Do note that distro-sync can downgrade packages, but can't handle
> > all
> > the cases. ie, upgrade postgresql and update all your data you
> > can't
> > just downgrade the rpm and be fine. Or any number of other
> > scriptlets
> > that do things that cannot easily be reversed.
> 
> The Epoch hack that the current policy forces us to use does not
> solve any 
> of that though.

Definitively IMHO we should change this policy on rawhide and updates-
testing repos, with these repos we should use (sometimes) : 

dnf distro-sync 

Will allow packager revert one package upgrade without Epoch hack . 

Cheers,
-- 
Sérgio M. B.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Pierre-Yves Chibon]

On Tue, Mar 06, 2018 at 12:56:56PM -0500, Randy Barlow wrote:
> On 03/03/2018 01:34 PM, Kevin Kofler wrote:
> > That is due to the "Rawhide can never go backwards" policy, which I still 
> > do 
> > not understand the point of, especially in the light of "distro-sync" 
> > having 
> > been supported by both the old yum and the new dnf for years.
> 
> Sometimes an updated package makes changes to its data structures. For
> example, consider if the package does some kind of migration to its data
> in such a way that the new version can read it, but the old cannot (e.g.
> a PostgreSQL upgrade, and IIRC the Firefox discussion from a while ago
> also noted that it cannot be downgraded in all cases). Thus, distro-sync
> doesn't work in all cases if the upgraded package has already made
> changes on the user's system.

What this tells me is that not all updates, even if they break things, will be
revertable, and for these, it's status quo with what we have today.
Allowing downgrade of updates that can be reverted would give us a little more
flexibility. We'll just have to keep in mind that not everything will be
revertable, which means we'll likely not be able to automatically rollback.

That doesn't sound too bad though, does it?


Pierre


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Adam Williamson]

On Fri, 2018-03-02 at 04:40 +0100, Kevin Kofler wrote:
> 
> > I disagree entirely with the above. I think the solution is to gate
> > packages coming into rawhide and hold or reject those that break the
> > compose until they are fixed. I think being proactive is VASTLY better
> > than being reactive. Not breaking something in the first place is much
> > easier to deal with than trying to fix it later.
> 
> And how do you propose doing that? The only reliable way to hold packages 
> that break the compose would be to actually try to run the whole compose 
> process after every package build. That just does not scale.

Then we need to compose faster. This is precisely one of the reasons
why there's quite a lot of work going into exactly that.

"Composes take multiple hours" isn't an immutable law of the universe.
It's a limitation of our compose process, and one we really ought to
fix.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Randy Barlow wrote:

> On 03/03/2018 01:34 PM, Kevin Kofler wrote:
>> That is due to the "Rawhide can never go backwards" policy, which I still
>> do not understand the point of, especially in the light of "distro-sync"
>> having been supported by both the old yum and the new dnf for years.
> 
> Sometimes an updated package makes changes to its data structures. For
> example, consider if the package does some kind of migration to its data
> in such a way that the new version can read it, but the old cannot (e.g.
> a PostgreSQL upgrade, and IIRC the Firefox discussion from a while ago
> also noted that it cannot be downgraded in all cases). Thus, distro-sync
> doesn't work in all cases if the upgraded package has already made
> changes on the user's system.

But bumping Epoch, as the policy makes you do in such a case, does 
absolutely nothing to fix that. So I don't see how the current policy helps.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Nicolas Mailhot wrote:
> The “never go backwards” policy means that as soon something hits devel
> other packages can rely on your package and start adapting
> their packages on the basis of your changes. You can not pull the carpet
> from under their feet just because you changed your mind.

This is entirely orthogonal to the current monotonic EVR policy because 
Epoch exists (and is used to revert to older upstream releases in Rawhide at 
times, which is perfectly compliant with the current policy) and because 
pure packaging changes are not covered at all (given that you can just bump 
Release again when reverting them).

The policy you would like to see does not exist at this time and would have 
to be worded entirely differently.

And additionally, I would argue the opposite: It is just impossible to do 
development without sometimes trying something that may fail and thus have 
to be reverted. Banning that would also contradict the Changes process (see 
the contingency plans).

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Kevin Fenzi wrote:
> Do note that distro-sync can downgrade packages, but can't handle all
> the cases. ie, upgrade postgresql and update all your data you can't
> just downgrade the rpm and be fine. Or any number of other scriptlets
> that do things that cannot easily be reversed.

The Epoch hack that the current policy forces us to use does not solve any 
of that though.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Fenzi]

On 03/06/2018 07:47 AM, Pierre-Yves Chibon wrote:
> On Tue, Mar 06, 2018 at 02:58:45PM +, Stephen Gallagher wrote:
...snip...
>> But that has its own issues.
> 
> Sorry, just to be clear, what would have its own issues:
> - asking rawhide users to use distro-sync instead of update?
> - automatically have dnf detect it's running in rawhide and default to
>   distro-sync instead of update?
> - or.. ?

I'll note that this has come up before in the past after we had
distro-sync, and I tried here to use distro-sync instead of update on my
rawhide laptop, and I ran into issues. Unfortunately, it's been a while
so I can't fully recall what the problem was, but I do remember
distro-sync failed and I had to use update. I can give it a try again
and see if things are any better.

Do note that distro-sync can downgrade packages, but can't handle all
the cases. ie, upgrade postgresql and update all your data you can't
just downgrade the rpm and be fine. Or any number of other scriptlets
that do things that cannot easily be reversed.
> 
> Random stupid question, what does update bring in addition to distro-sync?
> Is one more cpu/memory/bandwidth expensive than the other?

No, just distro-sync will downgrade things to match the repo in addition
to upgrading things to match the repo.

kevin




signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Randy Barlow]

On 03/03/2018 01:34 PM, Kevin Kofler wrote:
> That is due to the "Rawhide can never go backwards" policy, which I still do 
> not understand the point of, especially in the light of "distro-sync" having 
> been supported by both the old yum and the new dnf for years.

Sometimes an updated package makes changes to its data structures. For
example, consider if the package does some kind of migration to its data
in such a way that the new version can read it, but the old cannot (e.g.
a PostgreSQL upgrade, and IIRC the Firefox discussion from a while ago
also noted that it cannot be downgraded in all cases). Thus, distro-sync
doesn't work in all cases if the upgraded package has already made
changes on the user's system.



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Nicolas Mailhot]

Le mardi 06 mars 2018 à 15:53 +0100, Pierre-Yves Chibon a écrit :
> On Tue, Mar 06, 2018 at 03:33:44PM +0100, Kevin Kofler wrote:
> > Pierre-Yves Chibon wrote:
> > 
> > > On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
> > > > So please let us just repeal that "Rawhide can never go
> > > > backwards"
> > > > policy.
> > > 
> > > This is actually a fair point, but I wonder what prevents us from
> > > doing it today.
> > 
> > Technically, nothing. This is purely a policy issue.
> 
> I'd be curious if there isn't more than just this, or if someone
> remembers why
> that policy was created.

The “never go backwards” policy means that as soon something hits devel
other packages can rely on your package and start adapting
their packages on the basis of your changes. You can not pull the carpet
from under their feet just because you changed your mind.

Stable releases do not need this policy because devel is used to
coordinate the direction the distro does towards.

That's pretty much the only way to run development over a large pile of
code that involves many people over many organisations, timezones and
countries. There is just not enough bandwidth to communicate changes
better than "here is the repo that makes authority, you can rely on it
always going forward".

Linus has pretty much the same policy when he forces kernel developers
to commit on the API they push to him.

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Pierre-Yves Chibon]

On Tue, Mar 06, 2018 at 02:58:45PM +, Stephen Gallagher wrote:
>On Tue, Mar 6, 2018 at 9:54 AM Pierre-Yves Chibon 
>wrote:
> 
>  On Tue, Mar 06, 2018 at 03:33:44PM +0100, Kevin Kofler wrote:
>  > Pierre-Yves Chibon wrote:
>  >
>  > > On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
>  > >> So please let us just repeal that "Rawhide can never go backwards"
>  > >> policy.
>  > >
>  > > This is actually a fair point, but I wonder what prevents us from
>  doing it
>  > > today.
>  >
>  > Technically, nothing. This is purely a policy issue.
> 
>  I'd be curious if there isn't more than just this, or if someone
>  remembers why
>  that policy was created.
> 
>I *think* that the reason for Rawhide not being able to go backwards is
>simple RPM limitations; if people have updated their system with rawhide
>packages and encounter a serious bug, if we just roll the updates repo
>back to the previous working package, the people who upgraded to it have
>no *automatic* way forwards.
>Though, I suppose we could perhaps implement this policy if we
>special-cased (or simply encouraged) people on Rawhide to use distro-sync
>instead of simple update. 

> But that has its own issues.

Sorry, just to be clear, what would have its own issues:
- asking rawhide users to use distro-sync instead of update?
- automatically have dnf detect it's running in rawhide and default to
  distro-sync instead of update?
- or.. ?

Random stupid question, what does update bring in addition to distro-sync?
Is one more cpu/memory/bandwidth expensive than the other?


Thanks,
Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Stephen Gallagher]

On Tue, Mar 6, 2018 at 9:54 AM Pierre-Yves Chibon 
wrote:

> On Tue, Mar 06, 2018 at 03:33:44PM +0100, Kevin Kofler wrote:
> > Pierre-Yves Chibon wrote:
> >
> > > On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
> > >> So please let us just repeal that "Rawhide can never go backwards"
> > >> policy.
> > >
> > > This is actually a fair point, but I wonder what prevents us from
> doing it
> > > today.
> >
> > Technically, nothing. This is purely a policy issue.
>
> I'd be curious if there isn't more than just this, or if someone remembers
> why
> that policy was created.
>
>
I *think* that the reason for Rawhide not being able to go backwards is
simple RPM limitations; if people have updated their system with rawhide
packages and encounter a serious bug, if we just roll the updates repo back
to the previous working package, the people who upgraded to it have no
*automatic* way forwards.

Though, I suppose we could perhaps implement this policy if we
special-cased (or simply encouraged) people on Rawhide to use distro-sync
instead of simple update. But that has its own issues.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Pierre-Yves Chibon]

On Tue, Mar 06, 2018 at 03:33:44PM +0100, Kevin Kofler wrote:
> Pierre-Yves Chibon wrote:
> 
> > On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
> >> So please let us just repeal that "Rawhide can never go backwards"
> >> policy.
> > 
> > This is actually a fair point, but I wonder what prevents us from doing it
> > today.
> 
> Technically, nothing. This is purely a policy issue.

I'd be curious if there isn't more than just this, or if someone remembers why
that policy was created.

> > The only thing I can think of is that we have no mechanism to choose what
> > goes in rawhide and what does not, from the moment you build it, it will
> > go into rawhide.
> > So maybe gating rawhide, would give us that mechanism to a) prevent
> > package we know are broken from entering rawhide, b) potentially remove
> > from rawhide package we later find are breaking things.
> 
> "b)" can be done without any gating. That's what koji untag-pkg is for.

I suspect there is an ACL question here (which may be one of the reasons why the
policy was put in place).

> > I guess another issue with removing something from rawhide is that
> > something else may have been built on the top of it, thus removing A would
> > imply, automatically rebuilding B, and C, and D...
> 
> That would have to happen anyway, even if you bump Epoch to revert A as is 
> the policy now.

Fair, would be nice to have a way to do that automatically though :)

Maybe, we should only allow removing builds from rawhide for a limited period of
time and when that happens, just rebuild everything that has been built since
the build being removed happened.
There is a need for some tooling there to make it comfortable for all of us, but
we would need to map out carefully the requirements.
I'm not entirely sure I can commit on writing the tool right now, but would you
be willing to help mapping out the requirements this tool would have?

Thanks,
Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Pierre-Yves Chibon wrote:

> On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
>> So please let us just repeal that "Rawhide can never go backwards"
>> policy.
> 
> This is actually a fair point, but I wonder what prevents us from doing it
> today.

Technically, nothing. This is purely a policy issue.

> The only thing I can think of is that we have no mechanism to choose what
> goes in rawhide and what does not, from the moment you build it, it will
> go into rawhide.
> So maybe gating rawhide, would give us that mechanism to a) prevent
> package we know are broken from entering rawhide, b) potentially remove
> from rawhide package we later find are breaking things.

"b)" can be done without any gating. That's what koji untag-pkg is for.

> I guess another issue with removing something from rawhide is that
> something else may have been built on the top of it, thus removing A would
> imply, automatically rebuilding B, and C, and D...

That would have to happen anyway, even if you bump Epoch to revert A as is 
the policy now.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Pierre-Yves Chibon]

On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
> Kevin Fenzi wrote:
> > with using Epoch's much more commonly.
> 
> That is due to the "Rawhide can never go backwards" policy, which I still do 
> not understand the point of, especially in the light of "distro-sync" having 
> been supported by both the old yum and the new dnf for years.
> 
> We allow even updates-testing to go backwards, so why not Rawhide? I think 
> the only place we should ever enforce upgrade paths in is stable releases. 
> Rawhide can go backwards just fine. The fix is simply to use dnf distro-sync 
> instead of dnf update. (Enforcing the upgrade path from stable releases to 
> Rawhide may make sense to prepare for when Rawhide will eventually be 
> branched into a release, but what is the point of enforcing the upgrade path 
> from Rawhide at day d to Rawhide at day d+1? Rawhide users can just be 
> taught to use distro-sync, and users of stable releases will never see this 
> upgrade path "breakage".)
> 
> Red Hat Linux and early Fedora had worked fine for years without that 
> policy, and Epoch was required less often back then.
> 
> So please let us just repeal that "Rawhide can never go backwards" policy.

This is actually a fair point, but I wonder what prevents us from doing it
today. The only thing I can think of is that we have no mechanism to choose what
goes in rawhide and what does not, from the moment you build it, it will go into
rawhide.
So maybe gating rawhide, would give us that mechanism to a) prevent package we
know are broken from entering rawhide, b) potentially remove from rawhide
package we later find are breaking things.

I guess another issue with removing something from rawhide is that something
else may have been built on the top of it, thus removing A would imply,
automatically rebuilding B, and C, and D...
So while gating rawhide would prevent this, acting on it once it landed in
rawhide will be a little trickier.


Am I missing something?

Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Re: Frequently broken Rawhide/Branched composes

[Randy Barlow]

On 03/02/2018 04:04 AM, Nicolas Mailhot wrote:
> Also, how would you handle obsolete forgotten stray packages that linger
> in the repo (because they've not been killed properly, or a tool bug
> resulted in their non removal)? Gating gives any such package the power
> to block all updates in more critical packages

We can probably use it as an opportunity to fix that broken stray
package. Also if we had to, we do have WaiverDB now to just waive the
test failure.



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Matthew Miller]

On Sat, Mar 03, 2018 at 07:35:00PM +0100, Kevin Kofler wrote:
> > * It means things will likely be broken longer as there is less urgency
> > to fix them quickly.
> This means less stress for maintainers who are usually not working full time 
> on Fedora.

Which, let's not forget, is the *vast* majority of maintainers.
Including most people paid by Red Hat, and even most people paid by Red
Hat to work on Fedora full time.

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Kevin Fenzi wrote:
> * It means things will likely be broken longer as there is less urgency
> to fix them quickly.

This means less stress for maintainers who are usually not working full time 
on Fedora.

I don't see why a broken dependency in some leaf application that happens to 
be included on a release-blocking Edition or Spin needs to block the whole 
Rawhide compose. Such breakage is normal in a development version of a 
distribution, it happens even in Debian sid/unstable that probably has more 
daily users than Rawhide.

> * Shipping things out means we can't easily untag or revert packages
> with using Epoch's much more commonly.

That is due to the "Rawhide can never go backwards" policy, which I still do 
not understand the point of, especially in the light of "distro-sync" having 
been supported by both the old yum and the new dnf for years.

We allow even updates-testing to go backwards, so why not Rawhide? I think 
the only place we should ever enforce upgrade paths in is stable releases. 
Rawhide can go backwards just fine. The fix is simply to use dnf distro-sync 
instead of dnf update. (Enforcing the upgrade path from stable releases to 
Rawhide may make sense to prepare for when Rawhide will eventually be 
branched into a release, but what is the point of enforcing the upgrade path 
from Rawhide at day d to Rawhide at day d+1? Rawhide users can just be 
taught to use distro-sync, and users of stable releases will never see this 
upgrade path "breakage".)

Red Hat Linux and early Fedora had worked fine for years without that 
policy, and Epoch was required less often back then.

So please let us just repeal that "Rawhide can never go backwards" policy.

> * It will mean we are not in fact always shipping alpha quality, we
> could be shipping anything.

Even if everything composes, that does not guarantee any level of quality 
when you actually try to boot the composes.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Re: Frequently broken Rawhide/Branched composes

[Kevin Fenzi]

On 03/02/2018 08:42 AM, Nicolas Mailhot wrote:
> Le vendredi 02 mars 2018 à 10:52 +0100, Pierre-Yves Chibon a écrit :

...snip...

>> That'd actually give us the opportunity to clean up more our repos no?
>> :)
> 
> Explain that to someone who had several hours/days of builds rejected
> because of a broken obsolete package somewhere in the repo :(

Well, it wouldn't be rejected, it just would be held.

* Get your side tag
* Do all your bootstrapping builds and rebuilding things that depend on
it in the side tag.
* submit it, and it gets checked

Say there's one old package that doesn't build anymore against the new
stack. You could either wait for it to get fixed or decide the rest of
the builds are too important and wave the test to move them in.
There would still be some breakage there (the one old package), but at
least we would have record of who said that was ok so we could talk to
them, and it's a deliberate decision.

kevin




signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Fenzi]

On 03/01/2018 07:40 PM, Kevin Kofler wrote:

> And how do you propose doing that? The only reliable way to hold packages 
> that break the compose would be to actually try to run the whole compose 
> process after every package build. That just does not scale. The composes 
> are only daily for a reason. Running a compose for every package build would 
> use a lot of resources and would also take very long, delaying the tagging 
> of the package into Rawhide for an insane amount of time (at least hours).

The vast majority of issues are related to a package update causing
broken deps for something needed in something release blocking. So, I
would start there and setup gating to check for broken deps and require
waving that test or fixing the deps.

The next most common issue is core tools changing that we use to compose
(lorax, anaconda, kernel, glibc). For those we are intending to try and
make 'quick composes' that compose something quickly and test it to
catch common/simple issues.

Then we go from there.

> If you propose just doing some fast automated tests catching some issues, 
> that will never reliably catch all issues breaking the composes, and so my 
> proposals to increase the robustness of the compose process will still be 
> relevant. The only way to know for sure whether the compose will succeed is 
> to run it (and even that will not necessarily catch non-deterministic 
> failures).

The problems I see with just shipping whatever we compose:

* It means things will likely be broken longer as there is less urgency
to fix them quickly.
* There's cascading issues where issue A stops composing of items 1, 2,
3, 4 and each of those have further issues, so it means we don't even
know what all is broken.
* Shipping things out means we can't easily untag or revert packages
with using Epoch's much more commonly.
* It will mean we are not in fact always shipping alpha quality, we
could be shipping anything.
* reactive just is a lot harder in the end, IMHO.

> It is just defensive programming to not fail the whole process on any small 
> issue that you cannot avoid (with the resources that are available).

Not if you aren't sure of the thing you are producing.
> 
> By the way, in the distant past, if some (or even all) live image compose(s) 
> failed, the compose would just get published without that/those live 
> image(s) (in the worst case, with an empty Live directory). Not ideal (and 
> keeping the last successful build of each image as I suggest doing would be 
> an improvement on that, Koji should be enough to fulfill the copyleft 
> licenses' source distribution requirements), but at least the package repo 
> went out a lot more often than nowadays.

Well, I think the last few weeks have been particularly bad... it's not
always like this (or even ever).
> 
> 
> I am not asking you to ship bug-free composes. (I know that's impossible. 
> ;-) ) I am just asking you to take less than a week to get a compose with an 
> already built critical fix out. :-)

Well, I am not sure I would call that a critical fix, but to each their
own.

kevin




signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Re: Frequently broken Rawhide/Branched composes

[Nicolas Mailhot]

Le vendredi 02 mars 2018 à 10:52 +0100, Pierre-Yves Chibon a écrit :
> On Fri, Mar 02, 2018 at 10:04:35AM +0100, Nicolas Mailhot wrote:
> > Le jeudi 01 mars 2018 à 12:42 -0800, Kevin Fenzi a écrit :
> > > 
> > > I disagree entirely with the above. I think the solution is to
> > > gate
> > > packages coming into rawhide and hold or reject those that break
> > > the
> > > compose until they are fixed. 
> > 
> > While I don't disagree with the sentiment there needs to be a
> > mechanism
> > to handle bootstraps, where there are known-broken repository
> > statuses
> > between the beginning of the operation, when first bootstapped
> > pavkages
> > are injected, and the end, when several rebuild passes managed to
> > get
> > them all in fully build status.
> 
> Wouldn't side-tags in koji handle this? Imagining there was a tool to
> let packagers create/merge them as they need.

Maybe.
Let's just say, than after working for a few months with go, I've
severely revised my understanding of bootstrapping complexity, both on
the number of packages it may involve and the number of build phases it
may need before succeeding.

> > Also, how would you handle obsolete forgotten stray packages that
> > linger
> > in the repo (because they've not been killed properly, or a tool bug
> > resulted in their non removal)? Gating gives any such package the
> > power
> > to block all updates in more critical packages
> 
> That'd actually give us the opportunity to clean up more our repos no?
> :)

Explain that to someone who had several hours/days of builds rejected
because of a broken obsolete package somewhere in the repo :(

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Re: Frequently broken Rawhide/Branched composes

[Pierre-Yves Chibon]

On Fri, Mar 02, 2018 at 10:04:35AM +0100, Nicolas Mailhot wrote:
> Le jeudi 01 mars 2018 à 12:42 -0800, Kevin Fenzi a écrit :
> > 
> > I disagree entirely with the above. I think the solution is to gate
> > packages coming into rawhide and hold or reject those that break the
> > compose until they are fixed. 
> 
> While I don't disagree with the sentiment there needs to be a mechanism
> to handle bootstraps, where there are known-broken repository statuses
> between the beginning of the operation, when first bootstapped pavkages
> are injected, and the end, when several rebuild passes managed to get
> them all in fully build status.

Wouldn't side-tags in koji handle this? Imagining there was a tool to let
packagers create/merge them as they need.

> Also, how would you handle obsolete forgotten stray packages that linger
> in the repo (because they've not been killed properly, or a tool bug
> resulted in their non removal)? Gating gives any such package the power
> to block all updates in more critical packages

That'd actually give us the opportunity to clean up more our repos no? :)


Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Re: Frequently broken Rawhide/Branched composes

[Nicolas Mailhot]

Le jeudi 01 mars 2018 à 12:42 -0800, Kevin Fenzi a écrit :
> 
> I disagree entirely with the above. I think the solution is to gate
> packages coming into rawhide and hold or reject those that break the
> compose until they are fixed. 

While I don't disagree with the sentiment there needs to be a mechanism
to handle bootstraps, where there are known-broken repository statuses
between the beginning of the operation, when first bootstapped pavkages
are injected, and the end, when several rebuild passes managed to get
them all in fully build status.

Also, how would you handle obsolete forgotten stray packages that linger
in the repo (because they've not been killed properly, or a tool bug
resulted in their non removal)? Gating gives any such package the power
to block all updates in more critical packages

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Pierre-Yves Chibon]

On Fri, Mar 02, 2018 at 04:40:03AM +0100, Kevin Kofler wrote:
> Kevin Fenzi wrote:
> >> Something needs to be done to make the compose process more robust, e.g.:
> >> * running createrepo on a stable release, so that we do not have to be
> >> able
> >>   to init a chroot of the target system to compose a repository. A broken
> >>   dependency, even in systemd or rpm, should NEVER be a reason for the
> >>   repository to fail to compose.
> >> * publishing individual deliverables one at a time, i.e.:
> >>   1. compose the repositories,
> >>   2. sync the repositories out to the mirrors,
> >>   3. build the images (atomic ostrees, live images etc.) one at a time,
> >>   4. sync those images that succeeded out to the mirrors, keep the old
> >>  versions of the other ones (the matching SRPMs are in Koji anyway,
> >>  so it does not matter if the SRPMs in the tree don't match)
> >> etc.
> > 
> > I disagree entirely with the above. I think the solution is to gate
> > packages coming into rawhide and hold or reject those that break the
> > compose until they are fixed. I think being proactive is VASTLY better
> > than being reactive. Not breaking something in the first place is much
> > easier to deal with than trying to fix it later.
> 
> And how do you propose doing that? The only reliable way to hold packages 
> that break the compose would be to actually try to run the whole compose 
> process after every package build. That just does not scale. The composes 
> are only daily for a reason. Running a compose for every package build would 
> use a lot of resources and would also take very long, delaying the tagging 
> of the package into Rawhide for an insane amount of time (at least hours).

Well, do you remember Will Woods' talk at DevConf? They managed to do a compose
in a matter of minutes, so down the line this may be doable.

In the mean while, baby steps. Yes running fast automated tests won't catch all
the issues for every issues it will catch is one that will no need to be dealt
with later.
Once we have the mechanisms in place to gate packages entering rawhide, it will
be easier to improve the gating tools. We could even do as everybody else is
doing, start with some tests, fix a bug that got through, add the corresponding
tests to prevent it from happening again, repeat... :)


Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Kevin Fenzi wrote:
>> Something needs to be done to make the compose process more robust, e.g.:
>> * running createrepo on a stable release, so that we do not have to be
>> able
>>   to init a chroot of the target system to compose a repository. A broken
>>   dependency, even in systemd or rpm, should NEVER be a reason for the
>>   repository to fail to compose.
>> * publishing individual deliverables one at a time, i.e.:
>>   1. compose the repositories,
>>   2. sync the repositories out to the mirrors,
>>   3. build the images (atomic ostrees, live images etc.) one at a time,
>>   4. sync those images that succeeded out to the mirrors, keep the old
>>  versions of the other ones (the matching SRPMs are in Koji anyway,
>>  so it does not matter if the SRPMs in the tree don't match)
>> etc.
> 
> I disagree entirely with the above. I think the solution is to gate
> packages coming into rawhide and hold or reject those that break the
> compose until they are fixed. I think being proactive is VASTLY better
> than being reactive. Not breaking something in the first place is much
> easier to deal with than trying to fix it later.

And how do you propose doing that? The only reliable way to hold packages 
that break the compose would be to actually try to run the whole compose 
process after every package build. That just does not scale. The composes 
are only daily for a reason. Running a compose for every package build would 
use a lot of resources and would also take very long, delaying the tagging 
of the package into Rawhide for an insane amount of time (at least hours).

If you propose just doing some fast automated tests catching some issues, 
that will never reliably catch all issues breaking the composes, and so my 
proposals to increase the robustness of the compose process will still be 
relevant. The only way to know for sure whether the compose will succeed is 
to run it (and even that will not necessarily catch non-deterministic 
failures).

It is just defensive programming to not fail the whole process on any small 
issue that you cannot avoid (with the resources that are available).

By the way, in the distant past, if some (or even all) live image compose(s) 
failed, the compose would just get published without that/those live 
image(s) (in the worst case, with an empty Live directory). Not ideal (and 
keeping the last successful build of each image as I suggest doing would be 
an improvement on that, Koji should be enough to fulfill the copyleft 
licenses' source distribution requirements), but at least the package repo 
went out a lot more often than nowadays.

>> Right now, e.g., we have a known broken GCC (8.0.1-0.14) in the Rawhide
>> and Branched trees (which miscompiles the Chromium/QtWebEngine build tool
>> GN on x86_64), the fix (8.0.1-0.16) has already been in the right Koji
>> tags for days, but any third-party repository (RPM Fusion, Copr, etc.)
>> will still get the broken GCC. This is not acceptable.
> 
> I'm sure there's any number of bugs in any number of collections of
> packages.

I am not asking you to ship bug-free composes. (I know that's impossible. 
;-) ) I am just asking you to take less than a week to get a compose with an 
already built critical fix out. :-)

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Fenzi]

On 02/28/2018 06:19 PM, Kevin Kofler wrote:
> Fabio Valentini wrote:
>> AFAICT, those "broken deps in rawhide" mails are only sent if there is a
>> compose, and during the past weeks, there have been few of those ... so
>> breakage is sometimes allowed to sit unnoticed (and grow increasingly
>> worse) for very long.
> 
> Isn't that the real issue to fix? Failed Rawhide composes used to be a rare 
> event. Now we have both Rawhide and Branched composes broken for days at a 
> time, e.g., currently since February 20. This is just not acceptable.

Some of us have been/are working on fixing it. I agree that its not good
and we should work to fix it.
> 
> Something needs to be done to make the compose process more robust, e.g.:
> * running createrepo on a stable release, so that we do not have to be able
>   to init a chroot of the target system to compose a repository. A broken
>   dependency, even in systemd or rpm, should NEVER be a reason for the
>   repository to fail to compose.
> * publishing individual deliverables one at a time, i.e.:
>   1. compose the repositories,
>   2. sync the repositories out to the mirrors,
>   3. build the images (atomic ostrees, live images etc.) one at a time,
>   4. sync those images that succeeded out to the mirrors, keep the old
>  versions of the other ones (the matching SRPMs are in Koji anyway, so
>  it does not matter if the SRPMs in the tree don't match)
> etc.

I disagree entirely with the above. I think the solution is to gate
packages coming into rawhide and hold or reject those that break the
compose until they are fixed. I think being proactive is VASTLY better
than being reactive. Not breaking something in the first place is much
easier to deal with than trying to fix it later.
> 
> Right now, e.g., we have a known broken GCC (8.0.1-0.14) in the Rawhide and 
> Branched trees (which miscompiles the Chromium/QtWebEngine build tool GN on 
> x86_64), the fix (8.0.1-0.16) has already been in the right Koji tags for 
> days, but any third-party repository (RPM Fusion, Copr, etc.) will still get 
> the broken GCC. This is not acceptable.

I'm sure there's any number of bugs in any number of collections of
packages.

We had a successfull rawhide compose today. Many thanks to Patrick,
Adam, Dusty, Mohan, Dennis, the installer team and many others for
working through all the breakage.

Branched is running now, hopefully it will finish later today.

kevin




signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Dusty Mabe wrote:
> FYI: atomic ostree composes are not release blocking so that is not why a
> compose will fail.

In the past we had composes failing due to something Atomic failing to 
compose, I guess that changed since.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Dusty Mabe]

On 03/01/2018 11:52 AM, Kevin Kofler wrote:
> Vít Ondruch wrote:
>> Speaking of that, it seems that the Rawhide compose failed yesterday due
>> to some KDE/QT soname bump:
> [actually a typo in a Requires, as was already pointed out]
>>
>> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log
>> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log
>> https://kojipkgs.fedoraproject.org//work/tasks/8910/25368910/root.log
> 
> Well, the real issue is that the entire Rawhide compose fails just because 
> one release-blocking deliverable failed to compose. In this case, it was the 
> KDE Spin, but it has been happening with any other release-blocking 
> deliverable, e.g., Atomic ostree composes.

FYI: atomic ostree composes are not release blocking so that is not why a 
compose
will fail.

> 
> Why can we not just deliver the parts that succeeded and keep the last 
> working version of the deliverables that failed to compose this time? In 
> other words, why can we not just upload the 20180228 compose and keep the 
> 20180227 or whatever KDE ISO that last built? Why does it have to be all or 
> nothing?
> 
> Kevin Kofler
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Kevin Kofler]

Vít Ondruch wrote:
> Speaking of that, it seems that the Rawhide compose failed yesterday due
> to some KDE/QT soname bump:
[actually a typo in a Requires, as was already pointed out]
> 
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log
> https://kojipkgs.fedoraproject.org//work/tasks/8910/25368910/root.log

Well, the real issue is that the entire Rawhide compose fails just because 
one release-blocking deliverable failed to compose. In this case, it was the 
KDE Spin, but it has been happening with any other release-blocking 
deliverable, e.g., Atomic ostree composes.

Why can we not just deliver the parts that succeeded and keep the last 
working version of the deliverables that failed to compose this time? In 
other words, why can we not just upload the 20180228 compose and keep the 
20180227 or whatever KDE ISO that last built? Why does it have to be all or 
nothing?

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Vít Ondruch]

Dne 1.3.2018 v 14:47 Mamoru TASAKA napsal(a):
> Vít Ondruch wrote on 03/01/2018 06:44 PM:
>>
>>
>> Dne 1.3.2018 v 03:19 Kevin Kofler napsal(a):
>>> Fabio Valentini wrote:
 AFAICT, those "broken deps in rawhide" mails are only sent if there
 is a
 compose, and during the past weeks, there have been few of those
 ... so
 breakage is sometimes allowed to sit unnoticed (and grow increasingly
 worse) for very long.
>>> Isn't that the real issue to fix? Failed Rawhide composes used to be
>>> a rare
>>> event.
>>
>> Speaking of that, it seems that the Rawhide compose failed yesterday due
>> to some KDE/QT soname bump:
>>
>> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log
>>
>> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log
>>
>> https://kojipkgs.fedoraproject.org//work/tasks/8910/25368910/root.log
>>
>> Were these announced? Are they handled? Why aren't these done in
>> sidetag?
>
> No, the above root.log does not mean some KDE/QT soname bump. You must
> look at dnf complaint from the bottom to the top:
>
> 1. nothing provides gstreamer1-plugins-good5{?_isa} needed by
> phonon-qt5-backend-gstreamer-2:4.9.0-7.fc29.x86_64
> Apparently 5{?_isa} is typo (5 should be %, perhaps)

I might be wrong. Thx for pointing this out and thx to Adam for taking
care about this.

https://src.fedoraproject.org/rpms/phonon/c/8a0f0ad81bce771f85c1b0f4dc8b1f9c8372c17f?branch=master

Vít


>
> 2. So package phonon-qt5-4.10.0-1.fc29.x86_64 requires
> phonon-qt5-backend(x86-64) >= 4.7, but
> (as phonon-qt5-backend(x86-64) is provided by
> phonon-qt5-backend-gstreamer but
>  phonon-qt5-backend-gstreamer cannot be installed because of
> gstreamer1-plugins-good typo)
>    none of the providers can be installed
>
> 3. package kf5-knotifyconfig-5.43.0-2.fc28.x86_64 requires
> libphonon4qt5.so.4()(64bit), but
> (as libphonon4qt5.so.4 is provided by phonon-qt5-4.10.0-1.fc29.x86_64
> but phonon-qt5
>  cannot be installed because .  .) none of the providers
> can be installed
>
> 4. is the same, from the top to bottom.
>
>> V.
>
> Regards,
> Mamoru
>
>>
>>
>>>   Now we have both Rawhide and Branched composes broken for days at a
>>> time, e.g., currently since February 20. This is just not acceptable.
>>>
>>> Something needs to be done to make the compose process more robust,
>>> e.g.:
>>> * running createrepo on a stable release, so that we do not have to
>>> be able
>>>    to init a chroot of the target system to compose a repository. A
>>> broken
>>>    dependency, even in systemd or rpm, should NEVER be a reason for the
>>>    repository to fail to compose.
>>> * publishing individual deliverables one at a time, i.e.:
>>>    1. compose the repositories,
>>>    2. sync the repositories out to the mirrors,
>>>    3. build the images (atomic ostrees, live images etc.) one at a
>>> time,
>>>    4. sync those images that succeeded out to the mirrors, keep the old
>>>   versions of the other ones (the matching SRPMs are in Koji
>>> anyway, so
>>>   it does not matter if the SRPMs in the tree don't match)
>>> etc.
>>>
>>> Right now, e.g., we have a known broken GCC (8.0.1-0.14) in the
>>> Rawhide and
>>> Branched trees (which miscompiles the Chromium/QtWebEngine build
>>> tool GN on
>>> x86_64), the fix (8.0.1-0.16) has already been in the right Koji
>>> tags for
>>> days, but any third-party repository (RPM Fusion, Copr, etc.) will
>>> still get
>>> the broken GCC. This is not acceptable.
>>>
>>>  Kevin Kofler
>>> ___
>>> devel mailing list -- devel@lists.fedoraproject.org
>>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>>
>> ___
>> devel mailing list -- devel@lists.fedoraproject.org
>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>>
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Mamoru TASAKA]

Vít Ondruch wrote on 03/01/2018 06:44 PM:



Dne 1.3.2018 v 03:19 Kevin Kofler napsal(a):

Fabio Valentini wrote:

AFAICT, those "broken deps in rawhide" mails are only sent if there is a
compose, and during the past weeks, there have been few of those ... so
breakage is sometimes allowed to sit unnoticed (and grow increasingly
worse) for very long.

Isn't that the real issue to fix? Failed Rawhide composes used to be a rare
event.


Speaking of that, it seems that the Rawhide compose failed yesterday due
to some KDE/QT soname bump:

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log
https://kojipkgs.fedoraproject.org//work/tasks/8910/25368910/root.log

Were these announced? Are they handled? Why aren't these done in sidetag?


No, the above root.log does not mean some KDE/QT soname bump. You must
look at dnf complaint from the bottom to the top:

1. nothing provides gstreamer1-plugins-good5{?_isa} needed by 
phonon-qt5-backend-gstreamer-2:4.9.0-7.fc29.x86_64
Apparently 5{?_isa} is typo (5 should be %, perhaps)

2. So package phonon-qt5-4.10.0-1.fc29.x86_64 requires phonon-qt5-backend(x86-64) 
>= 4.7, but
(as phonon-qt5-backend(x86-64) is provided by phonon-qt5-backend-gstreamer but
 phonon-qt5-backend-gstreamer cannot be installed because of 
gstreamer1-plugins-good typo)
   none of the providers can be installed

3. package kf5-knotifyconfig-5.43.0-2.fc28.x86_64 requires 
libphonon4qt5.so.4()(64bit), but
(as libphonon4qt5.so.4 is provided by phonon-qt5-4.10.0-1.fc29.x86_64 but 
phonon-qt5
 cannot be installed because .  .) none of the providers can be 
installed

4. is the same, from the top to bottom.


V.


Regards,
Mamoru





  Now we have both Rawhide and Branched composes broken for days at a
time, e.g., currently since February 20. This is just not acceptable.

Something needs to be done to make the compose process more robust, e.g.:
* running createrepo on a stable release, so that we do not have to be able
   to init a chroot of the target system to compose a repository. A broken
   dependency, even in systemd or rpm, should NEVER be a reason for the
   repository to fail to compose.
* publishing individual deliverables one at a time, i.e.:
   1. compose the repositories,
   2. sync the repositories out to the mirrors,
   3. build the images (atomic ostrees, live images etc.) one at a time,
   4. sync those images that succeeded out to the mirrors, keep the old
  versions of the other ones (the matching SRPMs are in Koji anyway, so
  it does not matter if the SRPMs in the tree don't match)
etc.

Right now, e.g., we have a known broken GCC (8.0.1-0.14) in the Rawhide and
Branched trees (which miscompiles the Chromium/QtWebEngine build tool GN on
x86_64), the fix (8.0.1-0.16) has already been in the right Koji tags for
days, but any third-party repository (RPM Fusion, Copr, etc.) will still get
the broken GCC. This is not acceptable.

 Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Frequently broken Rawhide/Branched composes

[Vít Ondruch]

Dne 1.3.2018 v 03:19 Kevin Kofler napsal(a):
> Fabio Valentini wrote:
>> AFAICT, those "broken deps in rawhide" mails are only sent if there is a
>> compose, and during the past weeks, there have been few of those ... so
>> breakage is sometimes allowed to sit unnoticed (and grow increasingly
>> worse) for very long.
> Isn't that the real issue to fix? Failed Rawhide composes used to be a rare 
> event.

Speaking of that, it seems that the Rawhide compose failed yesterday due
to some KDE/QT soname bump:

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log
https://kojipkgs.fedoraproject.org//work/tasks/8910/25368910/root.log

Were these announced? Are they handled? Why aren't these done in sidetag?


V.


>  Now we have both Rawhide and Branched composes broken for days at a 
> time, e.g., currently since February 20. This is just not acceptable.
>
> Something needs to be done to make the compose process more robust, e.g.:
> * running createrepo on a stable release, so that we do not have to be able
>   to init a chroot of the target system to compose a repository. A broken
>   dependency, even in systemd or rpm, should NEVER be a reason for the
>   repository to fail to compose.
> * publishing individual deliverables one at a time, i.e.:
>   1. compose the repositories,
>   2. sync the repositories out to the mirrors,
>   3. build the images (atomic ostrees, live images etc.) one at a time,
>   4. sync those images that succeeded out to the mirrors, keep the old
>  versions of the other ones (the matching SRPMs are in Koji anyway, so
>  it does not matter if the SRPMs in the tree don't match)
> etc.
>
> Right now, e.g., we have a known broken GCC (8.0.1-0.14) in the Rawhide and 
> Branched trees (which miscompiles the Chromium/QtWebEngine build tool GN on 
> x86_64), the fix (8.0.1-0.16) has already been in the right Koji tags for 
> days, but any third-party repository (RPM Fusion, Copr, etc.) will still get 
> the broken GCC. This is not acceptable.
>
> Kevin Kofler
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org