> Hello fine fellows,
I hoped somebody else would reply, seems not :( To clarify, I and Dodji already
talked about this on IRC, and I asked him to send an email here to gather more
feedback. I'll summarize my thoughts here now, hopefully somebody else will
comment on that as well.
> The less good news is nobody got aware of the result of the task because
> the maintainer was most likely not notified. Because by default,
> notifications about the result of this task are turned off. And another
> message to the thread I mentionned above shows that it's not that easy
> to turn those notifications on without making any mistake:
> https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/message/J4IBLB3PMBCN4J4EQQCBEQZ4HSADMRIY.
Yes, FMN has a lot of rough edges and you definitely need to know what you're
doing in order to tweak something.
> I guess my point is that experience is showing us that if this kind of
> notification is not enabled by default, people are just not aware of it.
>
> I am thus proposing that notifications about the results of
> task-abicheck that are either NEEDS_INSPECTION or FAILED be sent to
> maintainers of the relevant packages *by default*.
We can extend the default FMN filter called "Critical taskotron tasks on my
packages" to also include dist.abicheck. I can't guarantee it will affect all
Fedora packagers, though. It will definitely not affect anyone who has done
modifications to the default taskotron filter (and they will have a hard time
to merge those changes). Also, we've seen many reports in the past in which
certain users didn't have this default filter present at all, or emails were
not delivered. All of that can be surely fixed by Fedora Infra, I'm just trying
to point out that changing a default in FMN doesn't currently guarantee all
people will really receive the messages.
My second concern is about increasing the "spam rate" from our systems. This is
probably more of a general concern, not directly related to abicheck. We
currently notify individually per check and per arch, so if there are 5
"important" checks and 3 arches, you'll receive 15 individual notifications
(provided they failed). It's not that bad, but it does not scale into the
future. I'd like to avoid a situation where people rather turn off the
notifications completely rather than receive so many individual emails. We will
definitely need something better in the mid-term future. I imagine a system
which knows that checks X, Y a Z are important for koji builds, waits until
they are all completed (but at most 60 minutes) and then sends a summary to the
user.
The best thing we have right now is Bodhi. I know it's late in the process (for
some checks), but it can tell you something's wrong in a single message
("autopush disabled because task XYZ failed"). Also allows you to look through
all the results before clicking "push to stable". Yes, it's not ideal, but
better than nothing.
So the question probably is whether to prefer Bodhi, late in the process but
less intrusive, or FMN, early in the process and possibly annoying. Truth be
told, a) I don't like FMN too much at its current state, it's more like a
proof-of-concept than something to be relied on (filter maintenance is a
tremendous pain from user POV), and b) even if FMN was perfectly polished, I
don't think it'd be the right solution to the problem anyway. It's too
low-level and can't aggregate messages into a bigger whole (by design!) as we
need it. We used it as a temporary solution to have at least /something/, at as
it usually happens, the temporary solutions grow and grow into monstrosities. I
wonder whether we're heading that way, and whether abicheck should be
considered an acceptable exception.
The good thing about abicheck is that it is well maintained with fast response
rates and its output is reasonably readable. Also, it doesn't seem to produce
too many failures[1][2], which is good because it hints there might be not too
many false negatives (remember that it does run only in critpath packages atm,
however). Those could be some of criteria we should have when deciding whether
inform people about some particular check results by default, and abicheck
passes those. A counterexample would be rpmgrill - while useful, its json
output is hardly readable (Matthew recently complained on test list about
that). So I don't have concrete objections to abicheck itself, but whether it
is a good idea to push more things (anything) to FMN and risk the backlash of a
subset of maintainers.
Regardless of the solution (even if we just decide to disable autopush in
Bodhi), we should probably make it much clearer where to report issues with
abicheck results, because it will be the first check in the "release critical"
set not directly maintained by us. That would probably be a topic for a
separate thread, but for each task I'd like to present clear visible
instructions with some FA
