Re: Detecting errors in Buildbot

2023-01-05 Thread fluxo 

> On Jan 5, 2023, at 12:03 AM, sebb  wrote:
> 
> On Thu, 5 Jan 2023 at 01:10, fluxo mailto:c...@apache.org>> 
> wrote:
>> 
>>> On Jan 4, 2023, at 3:15 PM, sebb  wrote:
>>> 
>>> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus >> > wrote:
 
 
 
> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> 
> The www-site builds failed recently, however this was not obvious
> because the failures don't appear to have been reported anywhere
> outside the dashboard.
> 
> As I recall, the previous version of BuildBot would send an email for
> broken (and mended) builds. This was very handy for detecting and
> fixing errors.
> 
> Are there any plans to provide something similar for the current Buildbot?
> 
 
 The functionality is available. Here is an example configuration for 
 setting up an email notifier.
>>> 
>>> Thanks, AFAICT something similar is already set up here:
>>> 
>>> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
>>> 
>>> However it does not seem to be working.
>> 
>> 
>> I've opened INFRA-24056 to track this. The error is attached below. As a 
>> stopgap, we can add an extraRecipients field, because it's not correctly 
>> detecting the contributors, and is thus unable to generate a list of 
>> "interested users."
> 
> I see.
> 
>> What list makes most sense to see failures related to the pelican websites 
>> builds?
> 
> I'm not sure if there is a single list that would be correct for all
> those builds.

We're creating a site-builds@infra list for this purpose, as currently the 
workaround requires all pelican build messages to go to a single address. 
People will be able to opt-in (subscribe) for build output, but it will be 
noisy, yes.




> However it would probably be OK to use the public list site-dev@a.o 
> 
> which was used in the past for bb builds. See for example:
> 
> https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p
> 
> If this proves too noisy it might be necessary to redirect some build
> mails elsewhere.
> 
> There also needs to be a way to ensure that script failures such as
> the one below are brought to the attention of builds and/or infra.
> 
>> 2022-12-31 19:22:37+ [-] INVALID EMAIL: asfinfra
>> 2022-12-31 19:22:37+ [-] sending mail (653 bytes) to []
>> 2022-12-31 19:22:37+ [-] Starting factory 
>> 
>> 2022-12-31 19:22:37+ [ESMTPSender,client] Got exception when handling 
>> reporter events
>>Traceback (most recent call last):
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", 
>> line 1658, in _inlineCallbacks
>>cast(Failure, result).throwExceptionIntoGenerator, gen
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", 
>> line 63, in run
>>return f(*args, **kwargs)
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", 
>> line 500, in throwExceptionIntoGenerato
>> r
>>return g.throw(self.type, self.value, self.tb)
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
>> line 95, in _got_event
>>log.err(e, 'Got exception when handling reporter events')
>>---  ---
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
>> line 93, in _got_event
>>yield self.sendMessage(reports)
>>twisted.mail._except.SMTPDeliveryError: No recipients accepted
> MAIL FROM:
>><<< 250 2.1.0 Ok
>> 
>> 
>>> 
>>> Sebb
 template=u'''\
 Build status: {{ summary }}
 Worker used: {{ workername }}
 URL: {{ build_url }}
 Blamelist: {{ blamelist | join(', ') }}
 Build Text: {{ build['state_string'] }}
 Status Detected: {{ status_detected }}
 {{ sourcestamps }}
 
 Steps:
 {% for step in build['steps'] %}
 {{ step['name'] }}: {{ step['results'] }}
 {% endfor %}
 
 -- ASF Buildbot
 '''
 simple_generator = reporters.BuildStatusGenerator(
 mode=('all'),
 builders=["infrastructure-test",],
 message_formatter=reporters.MessageFormatter(
 template=template,
 template_type='plain',
 want_properties=True,
 want_steps=True,
 ),
 )
 itn = reporters.MailNotifier(
 fromaddr="build...@apache.org",
 sendToInterestedUsers=True,
 generators=[simple_generator],
 extraRecipients=["gno...@infra.apache.org"],
 )
 bb.add_service(itn)
 
 
 
 
> Sebb

Re: Detecting errors in Buildbot

2023-01-05 Thread sebb 
On Thu, 5 Jan 2023 at 01:10, fluxo  wrote:
>
> > On Jan 4, 2023, at 3:15 PM, sebb  wrote:
> >
> > On Wed, 4 Jan 2023 at 18:18, Chris Lambertus  > > wrote:
> >>
> >>
> >>
> >>> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> >>>
> >>> The www-site builds failed recently, however this was not obvious
> >>> because the failures don't appear to have been reported anywhere
> >>> outside the dashboard.
> >>>
> >>> As I recall, the previous version of BuildBot would send an email for
> >>> broken (and mended) builds. This was very handy for detecting and
> >>> fixing errors.
> >>>
> >>> Are there any plans to provide something similar for the current Buildbot?
> >>>
> >>
> >> The functionality is available. Here is an example configuration for 
> >> setting up an email notifier.
> >
> > Thanks, AFAICT something similar is already set up here:
> >
> > https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
> >
> > However it does not seem to be working.
>
>
> I've opened INFRA-24056 to track this. The error is attached below. As a 
> stopgap, we can add an extraRecipients field, because it's not correctly 
> detecting the contributors, and is thus unable to generate a list of 
> "interested users."

I see.

> What list makes most sense to see failures related to the pelican websites 
> builds?

I'm not sure if there is a single list that would be correct for all
those builds.
However it would probably be OK to use the public list site-dev@a.o
which was used in the past for bb builds. See for example:

https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p

If this proves too noisy it might be necessary to redirect some build
mails elsewhere.

There also needs to be a way to ensure that script failures such as
the one below are brought to the attention of builds and/or infra.

> 2022-12-31 19:22:37+ [-] INVALID EMAIL: asfinfra
> 2022-12-31 19:22:37+ [-] sending mail (653 bytes) to []
> 2022-12-31 19:22:37+ [-] Starting factory 
> 
> 2022-12-31 19:22:37+ [ESMTPSender,client] Got exception when handling 
> reporter events
> Traceback (most recent call last):
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
> 1658, in _inlineCallbacks
> cast(Failure, result).throwExceptionIntoGenerator, gen
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
> 63, in run
> return f(*args, **kwargs)
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 
> 500, in throwExceptionIntoGenerato
> r
> return g.throw(self.type, self.value, self.tb)
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
> line 95, in _got_event
> log.err(e, 'Got exception when handling reporter events')
> ---  ---
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
> line 93, in _got_event
> yield self.sendMessage(reports)
> twisted.mail._except.SMTPDeliveryError: No recipients accepted
> >>> MAIL FROM:
> <<< 250 2.1.0 Ok
>
>
> >
> > Sebb
> >> template=u'''\
> >> Build status: {{ summary }}
> >> Worker used: {{ workername }}
> >> URL: {{ build_url }}
> >> Blamelist: {{ blamelist | join(', ') }}
> >> Build Text: {{ build['state_string'] }}
> >> Status Detected: {{ status_detected }}
> >> {{ sourcestamps }}
> >>
> >> Steps:
> >> {% for step in build['steps'] %}
> >> {{ step['name'] }}: {{ step['results'] }}
> >> {% endfor %}
> >>
> >> -- ASF Buildbot
> >> '''
> >> simple_generator = reporters.BuildStatusGenerator(
> >> mode=('all'),
> >> builders=["infrastructure-test",],
> >> message_formatter=reporters.MessageFormatter(
> >> template=template,
> >> template_type='plain',
> >> want_properties=True,
> >> want_steps=True,
> >> ),
> >> )
> >> itn = reporters.MailNotifier(
> >> fromaddr="build...@apache.org",
> >> sendToInterestedUsers=True,
> >> generators=[simple_generator],
> >> extraRecipients=["gno...@infra.apache.org"],
> >> )
> >> bb.add_service(itn)
> >>
> >>
> >>
> >>
> >>> Sebb
>

Re: Detecting errors in Buildbot

2023-01-04 Thread fluxo 
> On Jan 4, 2023, at 3:15 PM, sebb  wrote:
> 
> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus  > wrote:
>> 
>> 
>> 
>>> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
>>> 
>>> The www-site builds failed recently, however this was not obvious
>>> because the failures don't appear to have been reported anywhere
>>> outside the dashboard.
>>> 
>>> As I recall, the previous version of BuildBot would send an email for
>>> broken (and mended) builds. This was very handy for detecting and
>>> fixing errors.
>>> 
>>> Are there any plans to provide something similar for the current Buildbot?
>>> 
>> 
>> The functionality is available. Here is an example configuration for setting 
>> up an email notifier.
> 
> Thanks, AFAICT something similar is already set up here:
> 
> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
> 
> However it does not seem to be working.


I've opened INFRA-24056 to track this. The error is attached below. As a 
stopgap, we can add an extraRecipients field, because it's not correctly 
detecting the contributors, and is thus unable to generate a list of 
"interested users." What list makes most sense to see failures related to the 
pelican websites builds?

2022-12-31 19:22:37+ [-] INVALID EMAIL: asfinfra
2022-12-31 19:22:37+ [-] sending mail (653 bytes) to []
2022-12-31 19:22:37+ [-] Starting factory 

2022-12-31 19:22:37+ [ESMTPSender,client] Got exception when handling 
reporter events
Traceback (most recent call last):
  File 
"/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
1658, in _inlineCallbacks
cast(Failure, result).throwExceptionIntoGenerator, gen
  File 
"/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
63, in run
return f(*args, **kwargs)
  File 
"/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 
500, in throwExceptionIntoGenerato
r
return g.throw(self.type, self.value, self.tb)
  File 
"/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 
95, in _got_event
log.err(e, 'Got exception when handling reporter events')
---  ---
  File 
"/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 
93, in _got_event
yield self.sendMessage(reports)
twisted.mail._except.SMTPDeliveryError: No recipients accepted
>>> MAIL FROM:
<<< 250 2.1.0 Ok


> 
> Sebb
>> template=u'''\
>> Build status: {{ summary }}
>> Worker used: {{ workername }}
>> URL: {{ build_url }}
>> Blamelist: {{ blamelist | join(', ') }}
>> Build Text: {{ build['state_string'] }}
>> Status Detected: {{ status_detected }}
>> {{ sourcestamps }}
>> 
>> Steps:
>> {% for step in build['steps'] %}
>> {{ step['name'] }}: {{ step['results'] }}
>> {% endfor %}
>> 
>> -- ASF Buildbot
>> '''
>> simple_generator = reporters.BuildStatusGenerator(
>> mode=('all'),
>> builders=["infrastructure-test",],
>> message_formatter=reporters.MessageFormatter(
>> template=template,
>> template_type='plain',
>> want_properties=True,
>> want_steps=True,
>> ),
>> )
>> itn = reporters.MailNotifier(
>> fromaddr="build...@apache.org",
>> sendToInterestedUsers=True,
>> generators=[simple_generator],
>> extraRecipients=["gno...@infra.apache.org"],
>> )
>> bb.add_service(itn)
>> 
>> 
>> 
>> 
>>> Sebb

Re: Detecting errors in Buildbot

2023-01-04 Thread sebb 
On Wed, 4 Jan 2023 at 18:18, Chris Lambertus  wrote:
>
>
>
> > On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> >
> > The www-site builds failed recently, however this was not obvious
> > because the failures don't appear to have been reported anywhere
> > outside the dashboard.
> >
> > As I recall, the previous version of BuildBot would send an email for
> > broken (and mended) builds. This was very handy for detecting and
> > fixing errors.
> >
> > Are there any plans to provide something similar for the current Buildbot?
> >
>
> The functionality is available. Here is an example configuration for setting 
> up an email notifier.

Thanks, AFAICT something similar is already set up here:

https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56

However it does not seem to be working.

Sebb
> template=u'''\
> Build status: {{ summary }}
> Worker used: {{ workername }}
> URL: {{ build_url }}
> Blamelist: {{ blamelist | join(', ') }}
> Build Text: {{ build['state_string'] }}
> Status Detected: {{ status_detected }}
> {{ sourcestamps }}
>
> Steps:
> {% for step in build['steps'] %}
> {{ step['name'] }}: {{ step['results'] }}
> {% endfor %}
>
> -- ASF Buildbot
> '''
> simple_generator = reporters.BuildStatusGenerator(
> mode=('all'),
> builders=["infrastructure-test",],
> message_formatter=reporters.MessageFormatter(
> template=template,
> template_type='plain',
> want_properties=True,
> want_steps=True,
> ),
> )
> itn = reporters.MailNotifier(
> fromaddr="build...@apache.org",
> sendToInterestedUsers=True,
> generators=[simple_generator],
> extraRecipients=["gno...@infra.apache.org"],
> )
> bb.add_service(itn)
>
>
>
>
> > Sebb
>

Re: Detecting errors in Buildbot

2023-01-04 Thread Chris Lambertus 



> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> 
> The www-site builds failed recently, however this was not obvious
> because the failures don't appear to have been reported anywhere
> outside the dashboard.
> 
> As I recall, the previous version of BuildBot would send an email for
> broken (and mended) builds. This was very handy for detecting and
> fixing errors.
> 
> Are there any plans to provide something similar for the current Buildbot?
> 

The functionality is available. Here is an example configuration for setting up 
an email notifier.

template=u'''\
Build status: {{ summary }}
Worker used: {{ workername }}
URL: {{ build_url }}
Blamelist: {{ blamelist | join(', ') }}
Build Text: {{ build['state_string'] }}
Status Detected: {{ status_detected }}
{{ sourcestamps }}

Steps:
{% for step in build['steps'] %}
{{ step['name'] }}: {{ step['results'] }}
{% endfor %}

-- ASF Buildbot
'''
simple_generator = reporters.BuildStatusGenerator(
mode=('all'),
builders=["infrastructure-test",],
message_formatter=reporters.MessageFormatter(
template=template,
template_type='plain',
want_properties=True,
want_steps=True,
),
)
itn = reporters.MailNotifier(
fromaddr="build...@apache.org", 
sendToInterestedUsers=True, 
generators=[simple_generator],
extraRecipients=["gno...@infra.apache.org"],
)
bb.add_service(itn)




> Sebb

Detecting errors in Buildbot

2023-01-04 Thread sebb 
The www-site builds failed recently, however this was not obvious
because the failures don't appear to have been reported anywhere
outside the dashboard.

As I recall, the previous version of BuildBot would send an email for
broken (and mended) builds. This was very handy for detecting and
fixing errors.

Are there any plans to provide something similar for the current Buildbot?

Sebb