Re: excess automated emails from Fedora

[Daniel Pocock]

On 05/12/17 23:53, Kevin Fenzi wrote:
> On 12/05/2017 01:56 AM, Daniel Pocock wrote:
>> When somebody writes a comment in Bugzilla, I am receiving an email from
>> Bugzilla and another email from FMN: two emails for each comment.  I
>> haven't seen this type of behavior in any other project.
> This is a bit of an oddity as we don't control bugzilla, so I guess in
> this case we should not be sending anything via FMN by default since we
> cannot supress the bugzilla email.

Yes, every Bugzilla I've ever used enables emails by default so unless
somebody is going to disable that in Bugzilla, it should probably be
disabled in FMN

>> The emails from build...@fedoraproject.org don't appear to come from
>> FMN, here is a sample from the headers:
> yep, this is known. There are plans to re-write that script, but no one
> has stepped up to do it yet.
> ...snip...


It is always useful to have small tasks like tweaking that script for
people to demonstrate their skills when applying for GSoC, is this
tracked in Bugzilla somewhere already?


>> That is a technical policy, I was thinking more about a high-level
>> community-oriented policy on notifications / use of communications channels.
> I don't know of any.
> ...snip...


Where would be the next step to start or propose a policy like that?

>>
>> iCalendar doesn't involve sending notifications: it is about
>> representing the state of things.  Does FMN keep state or is it only a
>> mechanism for relaying notifications?
> it's only sending notifications based on fedmsg's it's gotten.
>  ...snip...


From an architecture perspective that is quite OK and a lot easier to
maintain.  Assuming it will continue to be like that, it means an
iCalendar solution would have to be something parallel to FMN.

>> From the perspective of FMN architecture, I suspect that some decision
>> would need to be made about which component(s) keep the state and how it
>> is transformed.  Examples:
>>
>> - if a build breaks, should the build system talk to Bugzilla's API
>> directly, opening a bug and closing it again next time the build succeeds?
>>
>> - or should the state of the last build be directly available as an
>> iCalendar feed and interested parties have to monitor both the Bugzilla
>> iCalendar the iCalendar URL from the build system?  Most tools like
>> Mozilla Lightning can monitor multiple task-list URLs like that but it
>> is tedious for every developer to set that up in their client.
>>
>> - or can FMN be extended to take state from multiple sources (both the
>> build system and Bugzilla) and merge those things into a single
>> iCalendar file, the same way that Planet runs from time to time and
>> merges multiple RSS feeds into one list?
>>
>> - or should some other system be built for that purpose and FMN just
>> relays notifications?
> It would need to be something else... FMN just sees a specific discrete
> fedmsg and notifies the user about it.

OK, so that eliminates the third option in that list but the other
options are all possible.

>
>> My observation is that the people who "like" email are simply not aware
>> of the other options.  There is a perception that it is easy and/or the
>> lowest common denominator.  It is very easy for script developers to
>> write a couple of lines of code to generate an email.  Over time email's
>> inefficiency is a huge cost to the Fedora project and any other
>> organization doing things this way.
>>
>> Think about it like this: somebody goes away for a 6 week extended
>> vacation and they come back to find 1000 automated email messages
>> waiting, I worked in one company where that was the way things were
>> done.  The consequences:
>>
>> - the emails are sorted chronologically, not in priority order, so the
>> user doesn't know which emails to look at first
>>
>> - they don't know which emails have been fixed by somebody else
>>
>> - they don't know which emails somebody else is already investigating
>>
>> - if 50 of those emails relate to a single problem or incident, it is
>> not obvious which email represents the root cause of the problem
>>
>> The person coming back to that inbox might spend 1 day just checking the
>> emails.  Or maybe 2-3 hours each day during the week after their
>> vacation, without really working on them, just trying to work out which
>> issues were still open, which emails are duplicates, prioritizing them, etc.
>>
>> Even when people don't go on vacation, they are losing a bit of time
>> every day on such filtering chores.  Add it all up and it is a huge cost
>> to organizations.  As a large organization, Fedora "loses" more time
>> this way than a small organization.
>>
>> In contrast, another company I worked in had a strict policy against
>> emails, things couldn't be released into production unless they worked
>> with the dashboard.  Any developer, sysadmin or manager could look at
>> the dashboard in the morning and quickly see the top 3 issues nobody was
>> working on.
> I find the idea 

Re: excess automated emails from Fedora

[Daniel Pocock]

On 06/12/17 00:21, Kevin Fenzi wrote:
> Sorry to reply to myself here, but I meant to reply to this part and
> forgot:
>
> On 12/05/2017 02:53 PM, Kevin Fenzi wrote:
>> On 12/05/2017 01:56 AM, Daniel Pocock wrote:
>>
>>> My observation is that the people who "like" email are simply not aware
>>> of the other options.  There is a perception that it is easy and/or the
>>> lowest common denominator.  It is very easy for script developers to
>>> write a couple of lines of code to generate an email.  Over time email's
>>> inefficiency is a huge cost to the Fedora project and any other
>>> organization doing things this way.
> I'm not sure I agree with this. Dashboards and point in time reporting
> provides great information for some cases, but discrete history at each
> change is also sometimes the information you want.

Looking at the options in the other email, one of them was:

"if a build breaks, should the build system talk to Bugzilla's API
directly, opening a bug and closing it again next time the build succeeds?"

That approach would satisfy the requirement about keeping history and
the current state would always be visible through Bugzilla's iCalendar
API (without any modification to Bugzilla)

For the script that sends the dependency emails from buildsys, is there
a place to report this as a feature request?  Is it in Bugzilla or
Github issues?

>>> Think about it like this: somebody goes away for a 6 week extended
>>> vacation and they come back to find 1000 automated email messages
>>> waiting, I worked in one company where that was the way things were
>>> done.  The consequences:
>>>
>>> - the emails are sorted chronologically, not in priority order, so the
>>> user doesn't know which emails to look at first
>>>
>>> - they don't know which emails have been fixed by somebody else
>>>
>>> - they don't know which emails somebody else is already investigating
>>>
>>> - if 50 of those emails relate to a single problem or incident, it is
>>> not obvious which email represents the root cause of the problem
> Sure. I personally filter emails into a number of buckets. So, when I
> came back and had 1000 emails, most of them would already be filtered.
> So, I would know I don't need to immediately look at mailing lists or
> other things that are lower priority.


I also sort emails but it takes a certain amount of effort for every
developer to set up their rules and in the end it doesn't answer all of
those questions, e.g. it doesn't tell you which emails have already been
handled by somebody else while you were on vacation.

>>> The person coming back to that inbox might spend 1 day just checking the
>>> emails.  Or maybe 2-3 hours each day during the week after their
>>> vacation, without really working on them, just trying to work out which
>>> issues were still open, which emails are duplicates, prioritizing them, etc.
> I get a lot of email, but I don't think I have ever had to spend weeks
> going through backlog.
>

I once worked in a company where I got over 100 emails on my first day
and averaged 2,000 per month.  My predecessor and his manager had been
putting "| mail" into scripts and cron jobs for over 10 years.  While it
sounds absurd, this is how things end up if developers don't consciously
and deliberately do things thoughtfully.

This is why I feel it is so important for leading free software projects
to be good role models in how we use email.


>>> Even when people don't go on vacation, they are losing a bit of time
>>> every day on such filtering chores.  Add it all up and it is a huge cost
>>> to organizations.  As a large organization, Fedora "loses" more time
>>> this way than a small organization.
>>>
>>> In contrast, another company I worked in had a strict policy against
>>> emails, things couldn't be released into production unless they worked
>>> with the dashboard.  Any developer, sysadmin or manager could look at
>>> the dashboard in the morning and quickly see the top 3 issues nobody was
>>> working on.
> Yeah, but what about:
>
> * You have added another maintainer to a package you maintain. You want
> to watch their commits and tell them when you see something they should
> change or do better on.

- you could set a reminder for yourself to check their commits each morning

- you could use fedmsg to create a rule for that, maybe with an expiry
date, it would send you chat notifications when you are logged in but
wouldn't leave a trail of emails

- the chat message thing might know when you log in again and send a
single notification about how many commits you missed overnight or whatever

- or FMN could even create a task for you when there are new commits and
if you don't close the task and more commits are made, it appends them
to the existing task rather than sending new alerts

> Or
>
> * You want to see the exact series of events that led up to a package
> being fixed.

This would be possible if everything was forced to go through Bugzilla
or if FMN dumps everything into 

Re: excess automated emails from Fedora

[Kevin Fenzi]

Sorry to reply to myself here, but I meant to reply to this part and
forgot:

On 12/05/2017 02:53 PM, Kevin Fenzi wrote:
> On 12/05/2017 01:56 AM, Daniel Pocock wrote:
> 
>> My observation is that the people who "like" email are simply not aware
>> of the other options.  There is a perception that it is easy and/or the
>> lowest common denominator.  It is very easy for script developers to
>> write a couple of lines of code to generate an email.  Over time email's
>> inefficiency is a huge cost to the Fedora project and any other
>> organization doing things this way.

I'm not sure I agree with this. Dashboards and point in time reporting
provides great information for some cases, but discrete history at each
change is also sometimes the information you want.

>> Think about it like this: somebody goes away for a 6 week extended
>> vacation and they come back to find 1000 automated email messages
>> waiting, I worked in one company where that was the way things were
>> done.  The consequences:
>>
>> - the emails are sorted chronologically, not in priority order, so the
>> user doesn't know which emails to look at first
>>
>> - they don't know which emails have been fixed by somebody else
>>
>> - they don't know which emails somebody else is already investigating
>>
>> - if 50 of those emails relate to a single problem or incident, it is
>> not obvious which email represents the root cause of the problem

Sure. I personally filter emails into a number of buckets. So, when I
came back and had 1000 emails, most of them would already be filtered.
So, I would know I don't need to immediately look at mailing lists or
other things that are lower priority.

>> The person coming back to that inbox might spend 1 day just checking the
>> emails.  Or maybe 2-3 hours each day during the week after their
>> vacation, without really working on them, just trying to work out which
>> issues were still open, which emails are duplicates, prioritizing them, etc.

I get a lot of email, but I don't think I have ever had to spend weeks
going through backlog.

>> Even when people don't go on vacation, they are losing a bit of time
>> every day on such filtering chores.  Add it all up and it is a huge cost
>> to organizations.  As a large organization, Fedora "loses" more time
>> this way than a small organization.
>>
>> In contrast, another company I worked in had a strict policy against
>> emails, things couldn't be released into production unless they worked
>> with the dashboard.  Any developer, sysadmin or manager could look at
>> the dashboard in the morning and quickly see the top 3 issues nobody was
>> working on.

Yeah, but what about:

* You have added another maintainer to a package you maintain. You want
to watch their commits and tell them when you see something they should
change or do better on.

Or

* You want to see the exact series of events that led up to a package
being fixed.

Or

* You want to be notified immediately if some action takes place because
you want to talk to whoever is doing it to change a workflow.

I think there is a place for dashboards and email and irc and mobile
phone alerts and ... it depends some on the thing and some on the person
who is getting it.

kevin





signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: excess automated emails from Fedora

[Kevin Fenzi]

On 12/05/2017 01:56 AM, Daniel Pocock wrote:
> 
> When somebody writes a comment in Bugzilla, I am receiving an email from
> Bugzilla and another email from FMN: two emails for each comment.  I
> haven't seen this type of behavior in any other project.

This is a bit of an oddity as we don't control bugzilla, so I guess in
this case we should not be sending anything via FMN by default since we
cannot supress the bugzilla email.

> 
> The emails from build...@fedoraproject.org don't appear to come from
> FMN, here is a sample from the headers:

yep, this is known. There are plans to re-write that script, but no one
has stepped up to do it yet.
...snip...
> 
> That is a technical policy, I was thinking more about a high-level
> community-oriented policy on notifications / use of communications channels.

I don't know of any.
...snip...

> 
> 
> iCalendar doesn't involve sending notifications: it is about
> representing the state of things.  Does FMN keep state or is it only a
> mechanism for relaying notifications?

it's only sending notifications based on fedmsg's it's gotten.
 ...snip...

> From the perspective of FMN architecture, I suspect that some decision
> would need to be made about which component(s) keep the state and how it
> is transformed.  Examples:
> 
> - if a build breaks, should the build system talk to Bugzilla's API
> directly, opening a bug and closing it again next time the build succeeds?
> 
> - or should the state of the last build be directly available as an
> iCalendar feed and interested parties have to monitor both the Bugzilla
> iCalendar the iCalendar URL from the build system?  Most tools like
> Mozilla Lightning can monitor multiple task-list URLs like that but it
> is tedious for every developer to set that up in their client.
> 
> - or can FMN be extended to take state from multiple sources (both the
> build system and Bugzilla) and merge those things into a single
> iCalendar file, the same way that Planet runs from time to time and
> merges multiple RSS feeds into one list?
> 
> - or should some other system be built for that purpose and FMN just
> relays notifications?

It would need to be something else... FMN just sees a specific discrete
fedmsg and notifies the user about it.


> My observation is that the people who "like" email are simply not aware
> of the other options.  There is a perception that it is easy and/or the
> lowest common denominator.  It is very easy for script developers to
> write a couple of lines of code to generate an email.  Over time email's
> inefficiency is a huge cost to the Fedora project and any other
> organization doing things this way.
> 
> Think about it like this: somebody goes away for a 6 week extended
> vacation and they come back to find 1000 automated email messages
> waiting, I worked in one company where that was the way things were
> done.  The consequences:
> 
> - the emails are sorted chronologically, not in priority order, so the
> user doesn't know which emails to look at first
> 
> - they don't know which emails have been fixed by somebody else
> 
> - they don't know which emails somebody else is already investigating
> 
> - if 50 of those emails relate to a single problem or incident, it is
> not obvious which email represents the root cause of the problem
> 
> The person coming back to that inbox might spend 1 day just checking the
> emails.  Or maybe 2-3 hours each day during the week after their
> vacation, without really working on them, just trying to work out which
> issues were still open, which emails are duplicates, prioritizing them, etc.
> 
> Even when people don't go on vacation, they are losing a bit of time
> every day on such filtering chores.  Add it all up and it is a huge cost
> to organizations.  As a large organization, Fedora "loses" more time
> this way than a small organization.
> 
> In contrast, another company I worked in had a strict policy against
> emails, things couldn't be released into production unless they worked
> with the dashboard.  Any developer, sysadmin or manager could look at
> the dashboard in the morning and quickly see the top 3 issues nobody was
> working on.

I find the idea interesting and may look more at it over the holidays.

I think it would be possible to build some kind of 'dashboard' setup,
and indeed the Fedora hubs effort seems like it kind of ties into this,
so perhaps thats where it would be useful to try and get more traction?

Perhaps those more involved in that project could chime in on the
current state?

kevin



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: excess automated emails from Fedora

[Daniel Pocock]

On 05/12/17 01:48, Kevin Fenzi wrote:
> On 12/04/2017 01:07 AM, Daniel Pocock wrote:
>>
>> Hi,
>>
>> I've opened a bug report[1] about automatically generated emails from
>> Fedora Fedmsg and other parts of Fedora infrastructure.
>>
>> Maybe there are other components of Fedora where bug reports like this
>> should be opened but that isn't completely clear to me.
>>
>> I'm sending this email to the community to ask a few things:
>>
>> - to see if other people have useful information to add to the bug report
> 
> I do, and have done so. :)
> 
>> - to see if anybody can suggest other components to file bug reports
>> against, or help do so
> 
> Well, the idea of FMN was to try and put all the notifications in one
> place, so likely if there are any other places they should be moved to FMN.

That would mean filing bug reports against those other places

When somebody writes a comment in Bugzilla, I am receiving an email from
Bugzilla and another email from FMN: two emails for each comment.  I
haven't seen this type of behavior in any other project.

The emails from build...@fedoraproject.org don't appear to come from
FMN, here is a sample from the headers:


Received: from rawhide-composer.phx2.fedoraproject.org (localhost
[127.0.0.1])
by rawhide-composer.phx2.fedoraproject.org (Postfix) with ESMTP id
892DC48065D
for ; Sun,  3 Dec 2017 14:24:49
+ (UTC)
From: build...@fedoraproject.org
To: resiprocate-ow...@fedoraproject.org
Cc:
Subject: Broken dependencies: resiprocate
Message-Id:
<20171203142449.892dc480...@rawhide-composer.phx2.fedoraproject.org>
Date: Sun,  3 Dec 2017 14:24:49 + (UTC)

There are also various other bug reports indicating that FMN's default
behavior seems to be spammy.  Even if these are not intentional, it
suggests that opting people in to things by default is high risk:

https://github.com/fedora-infra/fmn/issues/64

https://github.com/fedora-infra/fmn/issues/162


>>
>> - to find out if there is existing policy on this issue, a search of the
>> wiki didn't reveal anything obvious
> 
> Well, the policy is that when you are sponsored into the packager group
> you get the default set of packager rules. You can remove them, disable
> all notifications, add your own or whatever as you like. I don't think
> opting in is a good move here as new packagers likely are the people who
> need those notices the most.
> 

That is a technical policy, I was thinking more about a high-level
community-oriented policy on notifications / use of communications channels.

> I could definitely see the welcome to packager email note this and point
> people to where they can adjust things.
> 
>> - to get any general feedback about how the Fedora community feels about
>> use of email addresses.
>>
>> Note that I'm not just writing this to complain about the issue, I've
>> previously done work on tools to help people build dashboards of their
>> issues and I've blogged about some of those, including how you can get
>> Github issues in iCalendar[2], Nagios issues in iCalendar[3], Debian
>> issues in iCalendar[4].  Bugzilla (used by Fedora, Mozilla and GNOME,
>> for example) already has native iCalendar support.  Harsh Daftary did
>> work on aggregating[5] things from these sources in a dashboard as a
>> GSoC project.  Putting this all together, even using the Mozilla
>> Calendar/Lightning plugin, you can easily merge issues from many places
>> and see them in priority[2] order which is far more comfortable than
>> being bombarded by emails and trying to work out which is most
>> important, which is already fixed by somebody else, etc.
> 
> Perhaps you would like to write a FMN backend for this model? :)
> (we have email and irc now, but adding a ical one should be definitely
> possible I think).
> 


iCalendar doesn't involve sending notifications: it is about
representing the state of things.  Does FMN keep state or is it only a
mechanism for relaying notifications?

I can see the state of all Bugzilla issues assigned to me by going to
the advanced search, entering my email address as the bug assignee,
running the search and in the results list, clicking the "iCalendar"
link.  The link can be refreshed, bookmarked or used in other programs
to get a current snapshot of open issues at any time, it looks like this:


https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW_status=ASSIGNED=daniel%40pocock.com.au_to1=1=substring_id=8185597_format=advanced=ics


and here is yours, you can quickly try putting it in Mozilla Lightning
or GNOME Evolution (tasks) and see how it looks, when you open and close
things in Bugzilla they will automatically appear/disappear from this feed:


https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW_status=ASSIGNED=kevin%40scrye.com_to1=1=substring_id=8185597_format=advanced=ics


From the perspective of FMN architecture, I suspect that some decision
would need to be made about which component(s) keep the state and how it
is 

Re: excess automated emails from Fedora

[Kevin Fenzi]

On 12/04/2017 01:07 AM, Daniel Pocock wrote:
> 
> Hi,
> 
> I've opened a bug report[1] about automatically generated emails from
> Fedora Fedmsg and other parts of Fedora infrastructure.
> 
> Maybe there are other components of Fedora where bug reports like this
> should be opened but that isn't completely clear to me.
> 
> I'm sending this email to the community to ask a few things:
> 
> - to see if other people have useful information to add to the bug report

I do, and have done so. :)

> - to see if anybody can suggest other components to file bug reports
> against, or help do so

Well, the idea of FMN was to try and put all the notifications in one
place, so likely if there are any other places they should be moved to FMN.
> 
> - to find out if there is existing policy on this issue, a search of the
> wiki didn't reveal anything obvious

Well, the policy is that when you are sponsored into the packager group
you get the default set of packager rules. You can remove them, disable
all notifications, add your own or whatever as you like. I don't think
opting in is a good move here as new packagers likely are the people who
need those notices the most.

I could definitely see the welcome to packager email note this and point
people to where they can adjust things.

> - to get any general feedback about how the Fedora community feels about
> use of email addresses.
> 
> Note that I'm not just writing this to complain about the issue, I've
> previously done work on tools to help people build dashboards of their
> issues and I've blogged about some of those, including how you can get
> Github issues in iCalendar[2], Nagios issues in iCalendar[3], Debian
> issues in iCalendar[4].  Bugzilla (used by Fedora, Mozilla and GNOME,
> for example) already has native iCalendar support.  Harsh Daftary did
> work on aggregating[5] things from these sources in a dashboard as a
> GSoC project.  Putting this all together, even using the Mozilla
> Calendar/Lightning plugin, you can easily merge issues from many places
> and see them in priority[2] order which is far more comfortable than
> being bombarded by emails and trying to work out which is most
> important, which is already fixed by somebody else, etc.

Perhaps you would like to write a FMN backend for this model? :)
(we have email and irc now, but adding a ical one should be definitely
possible I think).

Everyone is different... some people like email, some people like irc,
some people like syncing taskwarrior to their tasks, some people like
nothing and only want to run queries when they have time.

> 
> - can anybody comment on other tools like this that put the developers
> in control rather than bombarding our inboxes with things?

You can control things at https://apps.fedoraproject.org/notifications/

> - would anybody like to help contribute to things like this in future,
> for example, mentoring another GSoC project on the dashboard concept?

I'm no developer, but perhaps someone else would like to. You could ask
on the infrastructure list.

kevin



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

excess automated emails from Fedora

[Daniel Pocock]

Hi,

I've opened a bug report[1] about automatically generated emails from
Fedora Fedmsg and other parts of Fedora infrastructure.

Maybe there are other components of Fedora where bug reports like this
should be opened but that isn't completely clear to me.

I'm sending this email to the community to ask a few things:

- to see if other people have useful information to add to the bug report

- to see if anybody can suggest other components to file bug reports
against, or help do so

- to find out if there is existing policy on this issue, a search of the
wiki didn't reveal anything obvious

- to get any general feedback about how the Fedora community feels about
use of email addresses.

Note that I'm not just writing this to complain about the issue, I've
previously done work on tools to help people build dashboards of their
issues and I've blogged about some of those, including how you can get
Github issues in iCalendar[2], Nagios issues in iCalendar[3], Debian
issues in iCalendar[4].  Bugzilla (used by Fedora, Mozilla and GNOME,
for example) already has native iCalendar support.  Harsh Daftary did
work on aggregating[5] things from these sources in a dashboard as a
GSoC project.  Putting this all together, even using the Mozilla
Calendar/Lightning plugin, you can easily merge issues from many places
and see them in priority[2] order which is far more comfortable than
being bombarded by emails and trying to work out which is most
important, which is already fixed by somebody else, etc.

- can anybody comment on other tools like this that put the developers
in control rather than bombarding our inboxes with things?

- would anybody like to help contribute to things like this in future,
for example, mentoring another GSoC project on the dashboard concept?

Regards,

Daniel



1. https://github.com/fedora-infra/fmn/issues/266
2. https://danielpocock.com/github-issues-as-an-icalendar-feed
3. https://danielpocock.com/get-your-nagios-issues-as-an-icalendar-feed
4.
https://danielpocock.com/debian-maintainer-dashboard-now-provides-icalendar-feeds
5.
https://danielpocock.com/aggregating-tasks-multiple-issue-trackers-gsoc-2015-summary
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org