Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Bruno Wolff III]

On Thu, Mar 01, 2018 at 09:19:48 -0600,
 Bruno Wolff III  wrote:


Note that there was a completed compose for March 1st, so you don't 
need this today.


I'm not sure what happened, but it looks like the compose only got copied 
over to the secondary architecure. So i386 has updates from today available 
by rsync, but x86_64 doesn't.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Kevin Fenzi]

On 03/01/2018 07:19 AM, Bruno Wolff III wrote:
> On Thu, Mar 01, 2018 at 10:31:21 +0100,
>  Vít Ondruch  wrote:
>>
>> Actually yes, it would be super useful if I knew where the repos are.
> 
> I use the following to get the rawhide i386 compose repo. I need to
> change the date and sequence number to match the one I want. I use it to
> get update packages and rarely to rollback packages after a bad update.

Note that these repos below only exist for two weeks and then are removed.

> You probably don't want the gpgcheck and sslverify disabled.

Yeah, everything should be signed and you should always use ssl/have
sslverify to true.

> 
> [compose]
> name=Fedora - Compose
> failovermethod=priority
> #baseurl=http://download.fedoraproject.org/pub/fedora/linux/development/$basearch/os/
> 
> #mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=rawhide=$basearch
> 
> #baseurl=http://bruno.wolff.to/fedora/development/rawhide/$basearch/os/
> #baseurl=file:///home/fedora/development/rawhide/Everything/$basearch/os/
> baseurl=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180301.n.0/compose/Everything/i386/os/
> 
> enabled=1
> gpgcheck=0
> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
> sslverify=False
> 
> Note that there was a completed compose for March 1st, so you don't need
> this today.

kevin





signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Bruno Wolff III]

On Thu, Mar 01, 2018 at 10:31:21 +0100,
 Vít Ondruch  wrote:


Actually yes, it would be super useful if I knew where the repos are.


I use the following to get the rawhide i386 compose repo. I need to change the 
date and sequence number to match the one I want. I use it to get update 
packages and rarely to rollback packages after a bad update.

You probably don't want the gpgcheck and sslverify disabled.

[compose]
name=Fedora - Compose
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/development/$basearch/os/
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=rawhide=$basearch
#baseurl=http://bruno.wolff.to/fedora/development/rawhide/$basearch/os/
#baseurl=file:///home/fedora/development/rawhide/Everything/$basearch/os/
baseurl=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180301.n.0/compose/Everything/i386/os/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
sslverify=False

Note that there was a completed compose for March 1st, so you don't need 
this today.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Vít Ondruch]

Dne 28.2.2018 v 18:56 Bruno Wolff III napsal(a):
> On Wed, Feb 28, 2018 at 10:35:11 +0100,
>  Vít Ondruch  wrote:
>>
>> And was the compose successful today or not. Looking at
>>
>> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log
>>
>>
>> it probably was, but it is not obvious from the log.
>
> Depending on why you care, it may be useful to know that in most cases
> when it fails a lot still gets done. Typically you can use the repos
> from failed composes to do updates if you want. This might be easier
> than pulling updates from koji, especially if you can about multiple
> packages.

Actually yes, it would be super useful if I knew where the repos are.

V.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Bruno Wolff III]

On Wed, Feb 28, 2018 at 10:35:11 +0100,
 Vít Ondruch  wrote:


And was the compose successful today or not. Looking at

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log

it probably was, but it is not obvious from the log.


Depending on why you care, it may be useful to know that in most cases when 
it fails a lot still gets done. Typically you can use the repos from 
failed composes to do updates if you want. This might be easier than pulling 
updates from koji, especially if you can about multiple packages.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Dennis Gilmore]

El mié, 28-02-2018 a las 10:35 +0100, Vít Ondruch escribió:
> Can the logs actually contain timestamps with timezone?
> 
> And was the compose successful today or not. Looking at
> 
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-201
> 80228.n.0/logs/global/pungi.global.log
> 
> it probably was, but it is not obvious from the log.
> 
the status of the compose can always be found in the STATUS file of the
root of the compose https://kojipkgs.fedoraproject.org/compose/rawhide/
Fedora-Rawhide-20180228.n.0/STATUS is todays for instance

Dennis

signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Vít Ondruch]

Dne 28.2.2018 v 10:52 Lubomír Sedlář napsal(a):
> Vít Ondruch píše v St 28. 02. 2018 v 10:35 +0100:
>> Can the logs actually contain timestamps with timezone?
> There's a timezone offset at the beginning of the log:
>
> 2018-02-28 00:33:30 [INFO] Current timezone offset: +00:00

A bit hard to find if you don't know. Thx

>
>> And was the compose successful today or not. Looking at
>>
>> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-201
>> 80228.n.0/logs/global/pungi.global.log
>>
>> it probably was, but it is not obvious from the log.
> It did not actually finish, the status is still STARTED:
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180
> 228.n.0/STATUS

Ah, ok. Thx for explanation.


V.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Lubomír Sedlář]

Vít Ondruch píše v St 28. 02. 2018 v 10:35 +0100:
> Can the logs actually contain timestamps with timezone?

There's a timezone offset at the beginning of the log:

2018-02-28 00:33:30 [INFO] Current timezone offset: +00:00

> And was the compose successful today or not. Looking at
> 
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-201
> 80228.n.0/logs/global/pungi.global.log
> 
> it probably was, but it is not obvious from the log.

It did not actually finish, the status is still STARTED:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180
228.n.0/STATUS

If the compose is successful, the log will say so explicitly:

2018-02-27 20:46:57 [INFO] Compose finished:
/mnt/koji/compose/updates/Fedora-Epel-7-updates-testing-20180227.1

Even on failure there should be indication:

2018-02-27 14:34:22 [CRITICAL] Compose failed:
/mnt/koji/compose/rawhide/Fedora-Rawhide-20180227.n.0

Lubomír
> 
> Vít
> 
> 
> 
> Dne 16.2.2018 v 21:14 Adam Williamson napsal(a):
> > On Fri, 2018-02-16 at 14:34 +0100, Vít Ondruch wrote:
> > > I wish the compose process was more transparent. May be it is
> > > just me,
> > > but I don't know where to start looking when I don't get the
> > > daily
> > > compose report.
> > 
> > https://kojipkgs.fedoraproject.org/compose/
> > 
> > All composes initially happen there. Rawhide composes happen in the
> > rawhide/ subdirectory. Branched happen in the branched/
> > subdirectory.
> > Each compose directory contains logs. The usual process for
> > debugging
> > failed composes is to look at the pungi.global.log file, which
> > usually
> > indicates what the failed fatal task was, get the task ID or full
> > task
> > URL from that log, then go to Koji and look at the actual failed
> > task
> > and figure out what went wrong with it.
> > 
> > > And the compose report email does pretty bad job explaining where
> > > the
> > > information comes from etc.
> > 
> > The compose reports are generated by https://pagure.io/compose-util
> > s ,
> > which is called by the scripts that actually run the composes,
> > which
> > live in https://pagure.io/pungi-fedora , along with (most of) the
> > Fedora-specific compose configuration bits.
> > https://pagure.io/pungi-fedora/blob/master/f/nightly.sh is the
> > script
> > that runs the Rawhide composes. Branched composes are run by the
> > copy
> > of the same script on the appropriately-numbered branch.
> > (Personally I
> > hate these scripts and the branching system, FWIW, and would much
> > prefer to rewrite them entirely, but Mohan has said he's going to
> > work
> > on that and it's more his job than it is mine, so I'm deferring to
> > him
> > on that one).
> 
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Vít Ondruch]

Can the logs actually contain timestamps with timezone?

And was the compose successful today or not. Looking at

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180228.n.0/logs/global/pungi.global.log

it probably was, but it is not obvious from the log.



Vít



Dne 16.2.2018 v 21:14 Adam Williamson napsal(a):
> On Fri, 2018-02-16 at 14:34 +0100, Vít Ondruch wrote:
>> I wish the compose process was more transparent. May be it is just me,
>> but I don't know where to start looking when I don't get the daily
>> compose report.
> https://kojipkgs.fedoraproject.org/compose/
>
> All composes initially happen there. Rawhide composes happen in the
> rawhide/ subdirectory. Branched happen in the branched/ subdirectory.
> Each compose directory contains logs. The usual process for debugging
> failed composes is to look at the pungi.global.log file, which usually
> indicates what the failed fatal task was, get the task ID or full task
> URL from that log, then go to Koji and look at the actual failed task
> and figure out what went wrong with it.
>
>> And the compose report email does pretty bad job explaining where the
>> information comes from etc.
> The compose reports are generated by https://pagure.io/compose-utils ,
> which is called by the scripts that actually run the composes, which
> live in https://pagure.io/pungi-fedora , along with (most of) the
> Fedora-specific compose configuration bits.
> https://pagure.io/pungi-fedora/blob/master/f/nightly.sh is the script
> that runs the Rawhide composes. Branched composes are run by the copy
> of the same script on the appropriately-numbered branch. (Personally I
> hate these scripts and the branching system, FWIW, and would much
> prefer to rewrite them entirely, but Mohan has said he's going to work
> on that and it's more his job than it is mine, so I'm deferring to him
> on that one).
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Mon, 2018-02-19 at 14:14 -0500, Matthew Miller wrote:
> On Sun, Feb 18, 2018 at 05:35:12PM -0600, Dennis Gilmore wrote:
> > I would like to have us look at not only the processes for how we build
> > and ship the distribution. I have some ideas for how we can keep the
> > important pieces of how we build and ship things today, but give us the
> > flexibility to have the peices be more independent and controlled. 
> 
> Okay, I'm a believer. :)
> 
> > than in the past, there is a long way to go. We may even need to take a
> > time out in order to focus on making fundamental changes to how we
> > manufacture Fedora.
> 
> We should make a plan for this. It's been almost five years (!) since
> we took the year for F20, and maybe it's time soon to do it again.

A big +1 to Dennis here.

To be clear, I'm not saying everything is fine and we shouldn't change
anything. I was just a bit worried about the way the initial proposal
appeared to be based on a belief that basically no-one cared about
composes right now (as this would be a mistaken starting point that
might lead to the loss or unnecessary laborious recreation of existing
knowledge and practices). I entirely agree with Dennis that another
flaw in the initial proposal is that it's based on trying to smooth out
the operation of the existing process, whereas it would be much better
to focus on making the process better.

So long as we're starting from an accurate assessment of how the
current process works (including the fact that several people follow it
quite closely and attempt to remove the grit in the gears, and have
quite a lot of domain-specific knowledge about doing that), and we
don't just think about how to smooth out that process but think about
making it better, I think we'll get somewhere useful.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Matthew Miller]

On Sun, Feb 18, 2018 at 05:35:12PM -0600, Dennis Gilmore wrote:
> I would like to have us look at not only the processes for how we build
> and ship the distribution. I have some ideas for how we can keep the
> important pieces of how we build and ship things today, but give us the
> flexibility to have the peices be more independent and controlled. 

Okay, I'm a believer. :)

> than in the past, there is a long way to go. We may even need to take a
> time out in order to focus on making fundamental changes to how we
> manufacture Fedora.

We should make a plan for this. It's been almost five years (!) since
we took the year for F20, and maybe it's time soon to do it again.



-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Dennis Gilmore]

El jue, 15-02-2018 a las 17:34 -0500, Matthew Miller escribió:
> As shown at https://www.happyassassin.net/nightlies.html, we haven't
> had a successful compose for almost two weeks. AIUI this is mostly
> worked out now, but it raises the question: who should be keeping
> track
> of this and coordinating fixes? For the releases, the blocker bug
> process basically handles this, so functionally the ownership ends up
> with QA (and the various heroes of QA who then track down all the
> problems). For rawhide, we don't have any such thing. Is it rel-eng?
> Is
> it FESCo? Is it the FPgM? Is it QA after all?
> 
> I suspect that right now, the answer is kind of "It's all of us
> together", which unfortunately practically speaking often comes down
> to
> 0.02% per person and rounds down to 0.
> 
> Or if the answer is: "Matthew, you are the FPL. It's you!" … then
> fine,
> but I'm going to then have to find some way to turn around and
> delegate. :)
> 
> I was chatting with Kevin Fenzi and he suggested naming a release
> manager for each release — someone who would keep track of stuff
> starting at branch, and help coordinate things like this.
> 
> This would help with more than just Rawhide — I'm sure everyone
> involved in the blocker bug process would be grateful for more help.
> At
> least, if it was someone who isn't already deeply involved in that.
> I'm
> thinking probably someone selected from FESCo — but it also could be
> a
> way for people with the particular set of skills required here to get
> involved in a way that's different from FESCo membership.
> 
> What do you think?

I think you are putting the cart before the horse slightly here. We
have a few issues that we need to address and that we have to
fundamentally step back and rethink how we put Fedora together. I feel
that your statement makes an assumption that how we do things today is
okay.

A fundamental issue we have today is that we have no way to test the
changes as they come in. As a result we hit apoint right before
branching where people put in a lot of change all at once, with the
result being it breaks things, we then get into fire fighting mode.
Having a project manager to stay on top of changes and change
management and trying to balance the change and get people to submit it
ealier may help some, but it would not fully address the issue of
making sure that the change is good.

I strongly believe that in order to really address the problem we need
to take a step back and look at the factory we have for making Fedora. 
 Today in order to make and ship Fedora we have a process that starts
by gathering by gathering together all of the rpms we have built,
putting them into a place so that we can feed them into making the
anaconda run time. Once we have the anaconda runtime we make the
distribution repos for the rpms and start the ostree creation. we then
create the Cloud images, containers, install dvd's, arm disk images,
livecd's etc. 

The entire process today is configured as a big blob. It requires that
the release engineer configuring the compose understand the end to end
process on how we build and ship everything. What the inputs for each
piece is, the expected outputs, and what pieces are required to make
other pieces. The way we have set things up means that the amount of
knowledge needed to be effecive is massive.  If you compare it to a
more traditional process, say a restaurant, you would have to go and
pick the fruits and vegetables that have been planted by someone else,
plan a extensive menu, answer the phone and take reservations, start
cooking the meals, greet the guest and seat them, take drink orders, go
make and deliver the drinks, take the meal orders, then go cook and
plate the meals, serve them to the guests and refresh drinks, make sure
that everyone is happy, once they are done eating clean the tables, and
take desert orders, go make plate and deliver desert, make any coffee
or tea thats required, deliver to the table, once the table is done,
you generate the bill, take it to the table, collect payment, then
clean up and do all the dishes after, wash the table linens and get
tings ready for the next meal.  If anything goes wrong during this
process that is run by a single person  you get to clean up and start
over from the start. That is a little long winded but shows how we have
not done the best by ourselves in how we have built and designed the
delivery of things today. Restaurants do not work in the way described
by a single person, neither does a car get built by having someone
design then build the car and get it out the door.  It can work for
special boutique offerings, it does not scale well for wide spread
distribution.

I would like to have us look at not only the processes for how we build
and ship the distribution. I have some ideas for how we can keep the
important pieces of how we build and ship things today, but give us the
flexibility to have the peices be more independent and controlled. 
Some 

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Sat, 2018-02-17 at 23:13 +0100, Lars Seipel wrote:
> On Fri, Feb 16, 2018 at 12:14:39PM -0800, Adam Williamson wrote:
> > On Fri, 2018-02-16 at 14:34 +0100, Vít Ondruch wrote:
> > > I wish the compose process was more transparent. May be it is
> > > just me,
> > > but I don't know where to start looking when I don't get the
> > > daily
> > > compose report.
> > 
> > https://kojipkgs.fedoraproject.org/compose/
> > 
> > All composes initially happen there. Rawhide composes happen in the
> > rawhide/ subdirectory. Branched happen in the branched/
> > subdirectory.
> > Each compose directory contains logs. The usual process for
> > debugging
> > failed composes is to look at the pungi.global.log file, which
> > usually
> > indicates what the failed fatal task was, get the task ID or full
> > task
> > URL from that log, then go to Koji and look at the actual failed
> > task
> > and figure out what went wrong with it.
> 
> Thank you, Adam. That was super-helpful.
> 
> If I'm reading those correctly, all issues are due to live images,
> container images or OSTree-related stuff. The composes seem to result
> in perfectly fine netinstall images and they even appear to work.
> 
> Why do we block the whole Rawhide repo for weeks on that?

We didn't used to. It was changed a couple of years ago. The idea is
basically that a change which causes release-blocking images to fail to
compose is probably not a change we want to publish to the repos.

The most recent issue we fixed that I know of was actually in the KDE
live image, which is release-blocking. qt5-qtwebengine depends on
libevent, which was inadvertently soname-bumped (see devel@), so it
needed a rebuild. However, it failed to build with GCC 8. We had to
work with both Chromium (the issue was ultimately in Chromium -
qtwebengine includes a copy of Chromium) and GCC upstreams to get that
one addressed. So, ultimately what this policy "means" there is: we
don't want to ship the libevent soname bump out until it's at least not
stopping release blocking images from composing any more. That seems
like a fairly reasonable position to me, really - is it really 'better'
if we sync out a Rawhide repo which contains a broken KDE (and,
incidentally, a broken Chromium)?

That one should have been addressed in the 20180217.n.1 compose,
though, and someone's fired a 20180217.n.2 today, so they obviously
found and fixed something else. I've been out all day today so I don't
know specifically what that issue was.

The major improvement we could really do with is blocking problematic
changes *earlier* - before they are tagged into Rawhide at all. That's
something people are actively working on ATM. If we could, for
instance, have blocked the inadvertent libevent soname bump, that would
have saved a lot of trouble.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Lars Seipel]

On Fri, Feb 16, 2018 at 12:14:39PM -0800, Adam Williamson wrote:
> On Fri, 2018-02-16 at 14:34 +0100, Vít Ondruch wrote:
> > I wish the compose process was more transparent. May be it is just me,
> > but I don't know where to start looking when I don't get the daily
> > compose report.
> 
> https://kojipkgs.fedoraproject.org/compose/
> 
> All composes initially happen there. Rawhide composes happen in the
> rawhide/ subdirectory. Branched happen in the branched/ subdirectory.
> Each compose directory contains logs. The usual process for debugging
> failed composes is to look at the pungi.global.log file, which usually
> indicates what the failed fatal task was, get the task ID or full task
> URL from that log, then go to Koji and look at the actual failed task
> and figure out what went wrong with it.

Thank you, Adam. That was super-helpful.

If I'm reading those correctly, all issues are due to live images,
container images or OSTree-related stuff. The composes seem to result
in perfectly fine netinstall images and they even appear to work.

Why do we block the whole Rawhide repo for weeks on that? If building
container images and OSTrees doesn't work in a reliable way, they
probably shouldn't be blocking repo composes. The failing spins
(Astronomy, PyClassroom, Mate, etc.) are already non-blocking, I
presume?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Kevin Fenzi]

On 02/16/2018 12:02 PM, Adam Williamson wrote:
> On Fri, 2018-02-16 at 09:54 +0100, nicolas.mail...@laposte.net wrote:
...snip...
>>
>> Right now rawhide has to be *very* broken for it to get any attention.
> 
> No it does not. This is not true. Rawhide is tested every day and I
> (and several others) pay attention to the results and attempt to get
> bugs fixed as quickly as possible.

Yeah, what Adam said. I run rawhide full time on my laptop and work to
get issues I hit fixed before they even land in rawhide for others.

I admit that when we have a branched and are near beta/final there's
less help for rawhide, but I at least still do care along with a few
others.

I don't think we are going to really get things better than now until we
start gating packages (which is very tricky). I hope we will be doing
that later this year tho.

kevin





signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Jason L Tibbitts III]

> "CW" == Colin Walters  writes:

CW> Meanwhile, there are probably hundreds of people on this -devel list
CW> who are capable of debugging and fixing things - some very
CW> experienced engineers, yet some of them are here busily debating
CW> minor things about spec files.

That's a false dichotomy.

 - J<
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Kevin Fenzi]

On 02/16/2018 05:31 AM, Colin Walters wrote:
> On Thu, Feb 15, 2018, at 5:39 PM, Adam Williamson wrote:
> 
>> In practice it tends to boil down to "me, nirik, and puiterwijk".
> 
> Meanwhile, there are probably hundreds of people on this -devel
> list who are capable of debugging and fixing things - some very
> experienced engineers, yet some of them are here busily debating
> minor things about spec files.
> 
> It doesn't make any sense at all to have hundreds (thousands?)
> of people whose sole power is to increment the versions of
> packages they own, and a tiny subset of people who are capable
> of e.g. reverting changes.
> 
> First step: Close down the #fedora-releng channel and discuss
> problems in #fedora-devel.  Releasing is not a distinct problem
> from development, and people should have experience in *both*
> roles.

I don't care much about this... I think the channel was split off
because there was a lot of other traffic in devel and releng issues
couldn't be discussed.

> 
> Two next steps:
> 
>  - Empower anyone to submit a pull request that can *revert* changes
>(e.g. the push steved just did to libevent) - not necessarily to
>merge it, but merging the PR should actually revert (e.g. it should
>also fix koji's state)

The "also fix koji's state" is not something a PR can do.

Look at the recent libevent so name bump. It happened and then about 3
people rebuilt their packages against it. What do you do? revert all
those? Or just fix the ones still needing rebuild? Without telling
everyone what you are doing this could result in doom.

>  - Generate a process to pick e.g. at least one person from each
>edition WG, plus perhaps more from various subsystems like Anaconda to be
>"on point" each day/week - and empower them to merge those PRs.

Well, usually if we need someone from some area we can track them
down... setting up some 'on point' person could be pretty complex...

> Another step:
> 
>  - Create a process to "close the tree", like Mozilla does for Firefox.

I don't know what this means can you expand?

Much of the problems we see would be "fixed" by gating packages into
rawhide. If they fail testing, they just don't go out until they
pass/fix all the fallout from their issue. I hope this is the year we
get this in place.

kevin




signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Fri, 2018-02-16 at 14:34 +0100, Vít Ondruch wrote:
> I wish the compose process was more transparent. May be it is just me,
> but I don't know where to start looking when I don't get the daily
> compose report.

https://kojipkgs.fedoraproject.org/compose/

All composes initially happen there. Rawhide composes happen in the
rawhide/ subdirectory. Branched happen in the branched/ subdirectory.
Each compose directory contains logs. The usual process for debugging
failed composes is to look at the pungi.global.log file, which usually
indicates what the failed fatal task was, get the task ID or full task
URL from that log, then go to Koji and look at the actual failed task
and figure out what went wrong with it.

> And the compose report email does pretty bad job explaining where the
> information comes from etc.

The compose reports are generated by https://pagure.io/compose-utils ,
which is called by the scripts that actually run the composes, which
live in https://pagure.io/pungi-fedora , along with (most of) the
Fedora-specific compose configuration bits.
https://pagure.io/pungi-fedora/blob/master/f/nightly.sh is the script
that runs the Rawhide composes. Branched composes are run by the copy
of the same script on the appropriately-numbered branch. (Personally I
hate these scripts and the branching system, FWIW, and would much
prefer to rewrite them entirely, but Mohan has said he's going to work
on that and it's more his job than it is mine, so I'm deferring to him
on that one).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Fri, 2018-02-16 at 08:31 -0500, Colin Walters wrote:
> On Thu, Feb 15, 2018, at 5:39 PM, Adam Williamson wrote:
> 
> > In practice it tends to boil down to "me, nirik, and puiterwijk".
> 
> Meanwhile, there are probably hundreds of people on this -devel
> list who are capable of debugging and fixing things - some very
> experienced engineers, yet some of them are here busily debating
> minor things about spec files.
> 
> It doesn't make any sense at all to have hundreds (thousands?)
> of people whose sole power is to increment the versions of
> packages they own, and a tiny subset of people who are capable
> of e.g. reverting changes.
> 
> First step: Close down the #fedora-releng channel and discuss
> problems in #fedora-devel.  Releasing is not a distinct problem
> from development, and people should have experience in *both*
> roles.

I don't really care either way about this. The names are just names.
What matters is that the work gets done. (In practice, most of the
folks who actually *work* on this stuff are in #fedora-releng...and
#fedora-devel...and #fedora-qa...and #fedora-admin..and a bunch of
other channels, and we frequently discuss the same issues in three of
them at once...we've been doing devops since the stone age, we're so
cool we don't even know it. :>)

> Two next steps:
> 
>  - Empower anyone to submit a pull request that can *revert* changes
>(e.g. the push steved just did to libevent) - not necessarily to
>merge it, but merging the PR should actually revert (e.g. it should
>also fix koji's state)
>  - Generate a process to pick e.g. at least one person from each
>edition WG, plus perhaps more from various subsystems like Anaconda to be
>"on point" each day/week - and empower them to merge those PRs.

I think this in practice would lead to chaos. There are often at least
two ways any given issue along these lines could be fixed (e.g.
revert/untag the 'bad' build, *or* rebuild everything against it). If
we just have a whole bunch of people deciding what to do on their own
initiative, half of them will likely try and do one, and half will do
the other. I don't think we want to run our distribution on the Twitch
Plays principle. It's better if we actually agree on a strategy before
implementing it, which is what we currently do in practice.

> Another step:
> 
>  - Create a process to "close the tree", like Mozilla does for Firefox.

I have no idea what this means. Could you explain?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Fri, 2018-02-16 at 07:21 -0500, Matthew Miller wrote:
> On Thu, Feb 15, 2018 at 03:22:29PM -0800, Adam Williamson wrote:
> > I mean...#fedora-releng ? This is kinda what release engineering *does*
> > - they engineer releases. So if a release is not being engineered
> > optimally, go ask release engineering. I guess I'm just not seeing what
> > all would actually be improved by putting another fancy named position
> > into the loop.
> 
> Well, I was looking at
> https://docs.pagure.org/releng/index.html#things-we-do, and it says
> 
> "Report progress towards release from branched creation on."
> 
> ... which implies that it's someone else's thing up until the branch. 

Well, 'report' is different from 'monitor'. In practice, releng folks
certainly do keep an eye on and work to fix Rawhide composes. We could
certainly clarify this stuff in some way, though, I guess.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Fri, 2018-02-16 at 09:54 +0100, nicolas.mail...@laposte.net wrote:
> 
> - Mail original -
> De: "Adam Williamson" 
> 
> Hi Adam
> 
> > This is...pretty much what I do for every release, and have been doing
> > since like Fedora 14 or something?
> 
> You do it end of cycle,

No I don't. I do it throughout the cycles. I tagged the first F28 Beta
blocker on 2017-11-27:

https://bugzilla.redhat.com/show_bug.cgi?id=1496562

which was actually a bit late, historically speaking; we frequently
file blockers for N+1 before N comes out. We specifically create the
blocker bugs two releases in advance so this is always possible.

>  and it is awe-inspiring to see you trying to fix all the problems
> that accumulated since the next-to-last branching at once,

This is not at all what happens. We constantly work on this throughout
the cycle. What you see at the end of the cycle (when I start sending
out mails to devel@) is just what's left at that point in time. If you
follow the process via the tracker bugs or the blocker review meetings,
you will see that we are constantly working on this process, it never
stops. If you hang out in #fedora-releng for a while you will see that
we generally work on fixing broken composes (Rawhide, Branched and
post-release stable) all the time as well.

>  but maybe some attention before the errors hit the fan and you have
> to intervene would help you?
> 
> Right now rawhide has to be *very* broken for it to get any attention.

No it does not. This is not true. Rawhide is tested every day and I
(and several others) pay attention to the results and attempt to get
bugs fixed as quickly as possible.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Fri, 2018-02-16 at 09:49 +0100, nicolas.mail...@laposte.net wrote:
> 
> It's all too easy right now for everyone to be busy with something
> else just when things break, and sometimes the authority to tell
> people "fix your builds you're blocking everyone" is needed (or even
> the authority to revert forcibly).

We do this already, using either proven packager privileges or just by
untagging builds from Rawhide.

> I suspect it would also prevent some of the release slippage Fedora
> is famed for — the slippage happens end of cycle but the delays build
> up far earlier.

I don't really see how it would. What would the release manager be
doing in this regard that is not currently being done?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[nicolas . mailhot]

- Mail original -
De: "Bruno Wolff III"

> Automation of rebuilds of dependencies might be a better approach. This 
> won't be able to address all cases, but could save people's time which we 
> seem to be short of.

I'm all for it! Some languages that are not careful about API breaks 
definitively need this!

However at the end of the day once you detect breakage you need someone to 
arbiter and act on it. Strict gating on breakage may make tricky bootstrapings 
even harder.

Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Bruno Wolff III]

On Fri, Feb 16, 2018 at 09:49:11 +0100,
 nicolas.mail...@laposte.net wrote:


It's all too easy right now for everyone to be busy with something else just when things 
break, and sometimes the authority to tell people "fix your builds you're blocking 
everyone" is needed (or even the authority to revert forcibly).


Automation of rebuilds of dependencies might be a better approach. This 
won't be able to address all cases, but could save people's time which we 
seem to be short of.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Vít Ondruch]

I wish the compose process was more transparent. May be it is just me,
but I don't know where to start looking when I don't get the daily
compose report.

And the compose report email does pretty bad job explaining where the
information comes from etc.


V.


Dne 15.2.2018 v 23:34 Matthew Miller napsal(a):
> As shown at https://www.happyassassin.net/nightlies.html, we haven't
> had a successful compose for almost two weeks. AIUI this is mostly
> worked out now, but it raises the question: who should be keeping track
> of this and coordinating fixes? For the releases, the blocker bug
> process basically handles this, so functionally the ownership ends up
> with QA (and the various heroes of QA who then track down all the
> problems). For rawhide, we don't have any such thing. Is it rel-eng? Is
> it FESCo? Is it the FPgM? Is it QA after all?
>
> I suspect that right now, the answer is kind of "It's all of us
> together", which unfortunately practically speaking often comes down to
> 0.02% per person and rounds down to 0.
>
> Or if the answer is: "Matthew, you are the FPL. It's you!" … then fine,
> but I'm going to then have to find some way to turn around and
> delegate. :)
>
> I was chatting with Kevin Fenzi and he suggested naming a release
> manager for each release — someone who would keep track of stuff
> starting at branch, and help coordinate things like this.
>
> This would help with more than just Rawhide — I'm sure everyone
> involved in the blocker bug process would be grateful for more help. At
> least, if it was someone who isn't already deeply involved in that. I'm
> thinking probably someone selected from FESCo — but it also could be a
> way for people with the particular set of skills required here to get
> involved in a way that's different from FESCo membership.
>
> What do you think?
>
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Colin Walters]

On Thu, Feb 15, 2018, at 5:39 PM, Adam Williamson wrote:

> In practice it tends to boil down to "me, nirik, and puiterwijk".

Meanwhile, there are probably hundreds of people on this -devel
list who are capable of debugging and fixing things - some very
experienced engineers, yet some of them are here busily debating
minor things about spec files.

It doesn't make any sense at all to have hundreds (thousands?)
of people whose sole power is to increment the versions of
packages they own, and a tiny subset of people who are capable
of e.g. reverting changes.

First step: Close down the #fedora-releng channel and discuss
problems in #fedora-devel.  Releasing is not a distinct problem
from development, and people should have experience in *both*
roles.

Two next steps:

 - Empower anyone to submit a pull request that can *revert* changes
   (e.g. the push steved just did to libevent) - not necessarily to
   merge it, but merging the PR should actually revert (e.g. it should
   also fix koji's state)
 - Generate a process to pick e.g. at least one person from each
   edition WG, plus perhaps more from various subsystems like Anaconda to be
   "on point" each day/week - and empower them to merge those PRs.

Another step:

 - Create a process to "close the tree", like Mozilla does for Firefox.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Matthew Miller]

On Thu, Feb 15, 2018 at 03:22:29PM -0800, Adam Williamson wrote:
> I mean...#fedora-releng ? This is kinda what release engineering *does*
> - they engineer releases. So if a release is not being engineered
> optimally, go ask release engineering. I guess I'm just not seeing what
> all would actually be improved by putting another fancy named position
> into the loop.

Well, I was looking at
https://docs.pagure.org/releng/index.html#things-we-do, and it says

"Report progress towards release from branched creation on."

... which implies that it's someone else's thing up until the branch. 

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Jonathan Wakely]

On 15/02/18 22:10 -0800, Kevin Fenzi wrote:

On 02/15/2018 03:21 PM, Adam Williamson wrote:

On Thu, 2018-02-15 at 18:05 -0500, Stephen John Smoogen wrote:


You have been doing it but have you been wanting to do it or just
doing it because no one else seems to do it when it is needed. Was it
an 'official' part of your job that no one documented or the
documentation was lost? And is it a job you have with a lot of
responsibility but no 'power' to do anything about?

Up until this conversation I thought the answer was "no one else was
doing it and someone had to, but I would much rather do the job people
pay me to do versus this".


I don't even know what the job people pay me to do *is* any more. :P

More people helping with things is always good, but I guess my concern
is that just throwing in a new person whose job is to 'co-ordinate'
isn't necessarily going to make anything any better, especially if that
person is just being randomly delegated by FESCo, isn't actually being
paid to do it full-time, and maybe doesn't have the experience of
having actually been doing it for years that some of us do have.


Yeah, I don't want that either... but I think it might help to spread
things around and increase communication if there's a known point of
contact for each release, so if you have some issue you can ask them and
they can at least find out whats going on, and that person knows they
should jump in on issues about the release they are looking after
without waiting/hoping someone else will. ;)


Yes, I think that makes sense. And it doesn't have to be some random,
inexperienced person appointed by FESCO. It could be one of you, adamw
and puiterwijk, rotating for each release. If in a couple of years
there's somebody else who's thrown themselves under that bus and shown
they have the skills and commitment to help, add them to the rota
(whether they like it or not ;-)

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Martin Kolman]

On Fri, 2018-02-16 at 09:54 +0100, nicolas.mail...@laposte.net wrote:
> 
> - Mail original -
> De: "Adam Williamson" 
> 
> Hi Adam
> 
> > This is...pretty much what I do for every release, and have been doing
> > since like Fedora 14 or something?
> 
> You do it end of cycle, and it is awe-inspiring to see you trying to fix all 
> the problems that accumulated since the
> next-to-last branching at once, but maybe some attention before the errors 
> hit the fan and you have to intervene would
> help you?
Actually, I think Adam does this about always - not just for branched releases, 
but also for Rawhide more or less
continually.


> 
> Right now rawhide has to be *very* broken for it to get any attention.
> 
> -- 
> Nicolas Mailhot
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Pierre-Yves Chibon]

On Thu, Feb 15, 2018 at 11:19:10PM +, Sérgio Basto wrote:
> On Thu, 2018-02-15 at 16:39 -0600, Rex Dieter wrote:
> > Matthew Miller wrote:
> > 
> > > As shown at https://www.happyassassin.net/nightlies.html, we
> > > haven't
> > > had a successful compose for almost two weeks. AIUI this is mostly
> > > worked out now, but it raises the question: who should be keeping
> > > track
> > > of this and coordinating fixes? 
> > 
> > ...
> > > What do you think?
> > 
> > Having a rawhide coordinator (aka herder-of-cats) is an excellent
> > idea.
> 
> 
> Branch more earlier , 2 months without branch ok , but 5 no, is
> stretching too much . Some software have 3 months cycle ,  so after one
> soname bump on beginning of rawhide , now we a second soname bump
> before branch. 
> 
> Another idea , is classify packages by importance not just in critical
> path or not, at least 3 or 4 levels, where level 1 just can be
> "upgraded" in an early rawhide and level 4, 5, or 6 can be upgraded
> anytime and in any branch .

Those are interesting ideas, but I don't see how that would make composing
rawhide more successful.
Mind expanding what you have in mind? I may just be missing it :)


Thanks,
Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[nicolas . mailhot]

- Mail original -
De: "Adam Williamson" 

Hi Adam

> This is...pretty much what I do for every release, and have been doing
> since like Fedora 14 or something?

You do it end of cycle, and it is awe-inspiring to see you trying to fix all 
the problems that accumulated since the next-to-last branching at once, but 
maybe some attention before the errors hit the fan and you have to intervene 
would help you?

Right now rawhide has to be *very* broken for it to get any attention.

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[nicolas . mailhot]

Hi,

A manager per release (including rawhide) would be awesome (as long as some 
provision is made for holiday replacements).

It's all too easy right now for everyone to be busy with something else just 
when things break, and sometimes the authority to tell people "fix your builds 
you're blocking everyone" is needed (or even the authority to revert forcibly).

I suspect it would also prevent some of the release slippage Fedora is famed 
for — the slippage happens end of cycle but the delays build up far earlier.

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Kevin Fenzi]

On 02/15/2018 03:21 PM, Adam Williamson wrote:
> On Thu, 2018-02-15 at 18:05 -0500, Stephen John Smoogen wrote:
>>
>> You have been doing it but have you been wanting to do it or just
>> doing it because no one else seems to do it when it is needed. Was it
>> an 'official' part of your job that no one documented or the
>> documentation was lost? And is it a job you have with a lot of
>> responsibility but no 'power' to do anything about?
>>
>> Up until this conversation I thought the answer was "no one else was
>> doing it and someone had to, but I would much rather do the job people
>> pay me to do versus this".
> 
> I don't even know what the job people pay me to do *is* any more. :P
> 
> More people helping with things is always good, but I guess my concern
> is that just throwing in a new person whose job is to 'co-ordinate'
> isn't necessarily going to make anything any better, especially if that
> person is just being randomly delegated by FESCo, isn't actually being
> paid to do it full-time, and maybe doesn't have the experience of
> having actually been doing it for years that some of us do have.

Yeah, I don't want that either... but I think it might help to spread
things around and increase communication if there's a known point of
contact for each release, so if you have some issue you can ask them and
they can at least find out whats going on, and that person knows they
should jump in on issues about the release they are looking after
without waiting/hoping someone else will. ;)

It's possible that just doing better tracking of issues would do this I
guess... give people a good idea where to look when things are broken
and what is going on (if known). (Which we are just trying to do now).

kevin





signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Chicago]

I would also like to thank the release engineering folks at fedora. Thank you! 

signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Thu, 2018-02-15 at 18:15 -0500, Matthew Miller wrote:
> 
> My intention here wasn't, actually, to complain, but to figure out if
> we could do something to spread the work around a little bit -- or
> maybe to attach some recognition to what's being done, and make a clear
> process for where, say, FPgM or someone should go for status on Rawhide
> in specific.

I mean...#fedora-releng ? This is kinda what release engineering *does*
- they engineer releases. So if a release is not being engineered
optimally, go ask release engineering. I guess I'm just not seeing what
all would actually be improved by putting another fancy named position
into the loop.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Thu, 2018-02-15 at 18:05 -0500, Stephen John Smoogen wrote:
> 
> You have been doing it but have you been wanting to do it or just
> doing it because no one else seems to do it when it is needed. Was it
> an 'official' part of your job that no one documented or the
> documentation was lost? And is it a job you have with a lot of
> responsibility but no 'power' to do anything about?
> 
> Up until this conversation I thought the answer was "no one else was
> doing it and someone had to, but I would much rather do the job people
> pay me to do versus this".

I don't even know what the job people pay me to do *is* any more. :P

More people helping with things is always good, but I guess my concern
is that just throwing in a new person whose job is to 'co-ordinate'
isn't necessarily going to make anything any better, especially if that
person is just being randomly delegated by FESCo, isn't actually being
paid to do it full-time, and maybe doesn't have the experience of
having actually been doing it for years that some of us do have.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Sérgio Basto]

On Thu, 2018-02-15 at 16:39 -0600, Rex Dieter wrote:
> Matthew Miller wrote:
> 
> > As shown at https://www.happyassassin.net/nightlies.html, we
> > haven't
> > had a successful compose for almost two weeks. AIUI this is mostly
> > worked out now, but it raises the question: who should be keeping
> > track
> > of this and coordinating fixes? 
> 
> ...
> > What do you think?
> 
> Having a rawhide coordinator (aka herder-of-cats) is an excellent
> idea.


Branch more earlier , 2 months without branch ok , but 5 no, is
stretching too much . Some software have 3 months cycle ,  so after one
soname bump on beginning of rawhide , now we a second soname bump
before branch. 

Another idea , is classify packages by importance not just in critical
path or not, at least 3 or 4 levels, where level 1 just can be
"upgraded" in an early rawhide and level 4, 5, or 6 can be upgraded
anytime and in any branch .




-- 
Sérgio M. B.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Matthew Miller]

On Thu, Feb 15, 2018 at 02:39:53PM -0800, Adam Williamson wrote:
> > I suspect that right now, the answer is kind of "It's all of us
> > together", which unfortunately practically speaking often comes down to
> > 0.02% per person and rounds down to 0.
> I...kinda disagree.

Sorry, that came out way more mean then I meant. In fact you and other
people who care do an amazing job of making it work over and over again
despite all sorts of problems. 

> > least, if it was someone who isn't already deeply involved in that. I'm
> > thinking probably someone selected from FESCo — but it also could be a
> > way for people with the particular set of skills required here to get
> > involved in a way that's different from FESCo membership.
> This is...pretty much what I do for every release, and have been doing
> since like Fedora 14 or something?

I know you do for the releases -- and heroically so, including on
things like "holidays", and "weekends" and "when most human beings are
sleeping". That's generally pretty thankless, so... thank you.

My intention here wasn't, actually, to complain, but to figure out if
we could do something to spread the work around a little bit -- or
maybe to attach some recognition to what's being done, and make a clear
process for where, say, FPgM or someone should go for status on Rawhide
in specific.


-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Stephen John Smoogen]

On 15 February 2018 at 17:39, Adam Williamson
 wrote:
> On Thu, 2018-02-15 at 17:34 -0500, Matthew Miller wrote:
>> As shown at https://www.happyassassin.net/nightlies.html, we haven't
>> had a successful compose for almost two weeks. AIUI this is mostly
>> worked out now, but it raises the question: who should be keeping track
>> of this and coordinating fixes? For the releases, the blocker bug
>> process basically handles this, so functionally the ownership ends up
>> with QA (and the various heroes of QA who then track down all the
>> problems). For rawhide, we don't have any such thing. Is it rel-eng? Is
>> it FESCo? Is it the FPgM? Is it QA after all?
>>
>> I suspect that right now, the answer is kind of "It's all of us
>> together", which unfortunately practically speaking often comes down to
>> 0.02% per person and rounds down to 0.
>
> I...kinda disagree.
>
> In practice it tends to boil down to "me, nirik, and puiterwijk". So,
> QA plus releng. And we don't ignore it. The current case is just
> turning out to be rather hard because it's not one case, it's about 15.
> So far we've found *four separate issues* in just the latest compose,
> never mind all the ones we fixed already:
>
> https://pagure.io/dusty/failed-composes/issue/1
>
> Note that any failed compose is implicitly a release blocking bug for
> whatever the next milestone release happens to be, and whenever we have
> something particularly intransigent blocking the compose, I usually
> file an actual blocker bug for it, so it bubbles up to the release
> blocking process that way.
>
>> Or if the answer is: "Matthew, you are the FPL. It's you!" … then fine,
>> but I'm going to then have to find some way to turn around and
>> delegate. :)
>>
>> I was chatting with Kevin Fenzi and he suggested naming a release
>> manager for each release — someone who would keep track of stuff
>> starting at branch, and help coordinate things like this.
>>
>> This would help with more than just Rawhide — I'm sure everyone
>> involved in the blocker bug process would be grateful for more help. At
>> least, if it was someone who isn't already deeply involved in that. I'm
>> thinking probably someone selected from FESCo — but it also could be a
>> way for people with the particular set of skills required here to get
>> involved in a way that's different from FESCo membership.
>
> This is...pretty much what I do for every release, and have been doing
> since like Fedora 14 or something?

You have been doing it but have you been wanting to do it or just
doing it because no one else seems to do it when it is needed. Was it
an 'official' part of your job that no one documented or the
documentation was lost? And is it a job you have with a lot of
responsibility but no 'power' to do anything about?

Up until this conversation I thought the answer was "no one else was
doing it and someone had to, but I would much rather do the job people
pay me to do versus this".


> --
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
> http://www.happyassassin.net
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Rex Dieter]

Matthew Miller wrote:

> As shown at https://www.happyassassin.net/nightlies.html, we haven't
> had a successful compose for almost two weeks. AIUI this is mostly
> worked out now, but it raises the question: who should be keeping track
> of this and coordinating fixes? 
...
> What do you think?

Having a rawhide coordinator (aka herder-of-cats) is an excellent idea.

-- rex
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Should we have a release manager for each release? (or, "who owns rawhide"?)

[Adam Williamson]

On Thu, 2018-02-15 at 17:34 -0500, Matthew Miller wrote:
> As shown at https://www.happyassassin.net/nightlies.html, we haven't
> had a successful compose for almost two weeks. AIUI this is mostly
> worked out now, but it raises the question: who should be keeping track
> of this and coordinating fixes? For the releases, the blocker bug
> process basically handles this, so functionally the ownership ends up
> with QA (and the various heroes of QA who then track down all the
> problems). For rawhide, we don't have any such thing. Is it rel-eng? Is
> it FESCo? Is it the FPgM? Is it QA after all?
> 
> I suspect that right now, the answer is kind of "It's all of us
> together", which unfortunately practically speaking often comes down to
> 0.02% per person and rounds down to 0.

I...kinda disagree.

In practice it tends to boil down to "me, nirik, and puiterwijk". So,
QA plus releng. And we don't ignore it. The current case is just
turning out to be rather hard because it's not one case, it's about 15.
So far we've found *four separate issues* in just the latest compose,
never mind all the ones we fixed already:

https://pagure.io/dusty/failed-composes/issue/1

Note that any failed compose is implicitly a release blocking bug for
whatever the next milestone release happens to be, and whenever we have
something particularly intransigent blocking the compose, I usually
file an actual blocker bug for it, so it bubbles up to the release
blocking process that way.

> Or if the answer is: "Matthew, you are the FPL. It's you!" … then fine,
> but I'm going to then have to find some way to turn around and
> delegate. :)
> 
> I was chatting with Kevin Fenzi and he suggested naming a release
> manager for each release — someone who would keep track of stuff
> starting at branch, and help coordinate things like this.
> 
> This would help with more than just Rawhide — I'm sure everyone
> involved in the blocker bug process would be grateful for more help. At
> least, if it was someone who isn't already deeply involved in that. I'm
> thinking probably someone selected from FESCo — but it also could be a
> way for people with the particular set of skills required here to get
> involved in a way that's different from FESCo membership.

This is...pretty much what I do for every release, and have been doing
since like Fedora 14 or something?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Should we have a release manager for each release? (or, "who owns rawhide"?)

[Matthew Miller]

As shown at https://www.happyassassin.net/nightlies.html, we haven't
had a successful compose for almost two weeks. AIUI this is mostly
worked out now, but it raises the question: who should be keeping track
of this and coordinating fixes? For the releases, the blocker bug
process basically handles this, so functionally the ownership ends up
with QA (and the various heroes of QA who then track down all the
problems). For rawhide, we don't have any such thing. Is it rel-eng? Is
it FESCo? Is it the FPgM? Is it QA after all?

I suspect that right now, the answer is kind of "It's all of us
together", which unfortunately practically speaking often comes down to
0.02% per person and rounds down to 0.

Or if the answer is: "Matthew, you are the FPL. It's you!" … then fine,
but I'm going to then have to find some way to turn around and
delegate. :)

I was chatting with Kevin Fenzi and he suggested naming a release
manager for each release — someone who would keep track of stuff
starting at branch, and help coordinate things like this.

This would help with more than just Rawhide — I'm sure everyone
involved in the blocker bug process would be grateful for more help. At
least, if it was someone who isn't already deeply involved in that. I'm
thinking probably someone selected from FESCo — but it also could be a
way for people with the particular set of skills required here to get
involved in a way that's different from FESCo membership.

What do you think?


-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org