Re: Detecting errors in Buildbot

2023-01-05 Thread fluxo 

> On Jan 5, 2023, at 12:03 AM, sebb  wrote:
> 
> On Thu, 5 Jan 2023 at 01:10, fluxo mailto:c...@apache.org>> 
> wrote:
>> 
>>> On Jan 4, 2023, at 3:15 PM, sebb  wrote:
>>> 
>>> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus >> > wrote:
 
 
 
> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> 
> The www-site builds failed recently, however this was not obvious
> because the failures don't appear to have been reported anywhere
> outside the dashboard.
> 
> As I recall, the previous version of BuildBot would send an email for
> broken (and mended) builds. This was very handy for detecting and
> fixing errors.
> 
> Are there any plans to provide something similar for the current Buildbot?
> 
 
 The functionality is available. Here is an example configuration for 
 setting up an email notifier.
>>> 
>>> Thanks, AFAICT something similar is already set up here:
>>> 
>>> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
>>> 
>>> However it does not seem to be working.
>> 
>> 
>> I've opened INFRA-24056 to track this. The error is attached below. As a 
>> stopgap, we can add an extraRecipients field, because it's not correctly 
>> detecting the contributors, and is thus unable to generate a list of 
>> "interested users."
> 
> I see.
> 
>> What list makes most sense to see failures related to the pelican websites 
>> builds?
> 
> I'm not sure if there is a single list that would be correct for all
> those builds.

We're creating a site-builds@infra list for this purpose, as currently the 
workaround requires all pelican build messages to go to a single address. 
People will be able to opt-in (subscribe) for build output, but it will be 
noisy, yes.




> However it would probably be OK to use the public list site-dev@a.o 
> 
> which was used in the past for bb builds. See for example:
> 
> https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p
> 
> If this proves too noisy it might be necessary to redirect some build
> mails elsewhere.
> 
> There also needs to be a way to ensure that script failures such as
> the one below are brought to the attention of builds and/or infra.
> 
>> 2022-12-31 19:22:37+ [-] INVALID EMAIL: asfinfra
>> 2022-12-31 19:22:37+ [-] sending mail (653 bytes) to []
>> 2022-12-31 19:22:37+ [-] Starting factory 
>> 
>> 2022-12-31 19:22:37+ [ESMTPSender,client] Got exception when handling 
>> reporter events
>>Traceback (most recent call last):
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", 
>> line 1658, in _inlineCallbacks
>>cast(Failure, result).throwExceptionIntoGenerator, gen
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", 
>> line 63, in run
>>return f(*args, **kwargs)
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", 
>> line 500, in throwExceptionIntoGenerato
>> r
>>return g.throw(self.type, self.value, self.tb)
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
>> line 95, in _got_event
>>log.err(e, 'Got exception when handling reporter events')
>>---  ---
>>  File 
>> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
>> line 93, in _got_event
>>yield self.sendMessage(reports)
>>twisted.mail._except.SMTPDeliveryError: No recipients accepted
> MAIL FROM:
>><<< 250 2.1.0 Ok
>> 
>> 
>>> 
>>> Sebb
 template=u'''\
 Build status: {{ summary }}
 Worker used: {{ workername }}
 URL: {{ build_url }}
 Blamelist: {{ blamelist | join(', ') }}
 Build Text: {{ build['state_string'] }}
 Status Detected: {{ status_detected }}
 {{ sourcestamps }}
 
 Steps:
 {% for step in build['steps'] %}
 {{ step['name'] }}: {{ step['results'] }}
 {% endfor %}
 
 -- ASF Buildbot
 '''
 simple_generator = reporters.BuildStatusGenerator(
 mode=('all'),
 builders=["infrastructure-test",],
 message_formatter=reporters.MessageFormatter(
 template=template,
 template_type='plain',
 want_properties=True,
 want_steps=True,
 ),
 )
 itn = reporters.MailNotifier(
 fromaddr="build...@apache.org",
 sendToInterestedUsers=True,
 generators=[simple_generator],
 extraRecipients=["gno...@infra.apache.org"],
 )
 bb.add_service(itn)
 
 
 
 
> Sebb

Re: Detecting errors in Buildbot

2023-01-05 Thread sebb 
On Thu, 5 Jan 2023 at 01:10, fluxo  wrote:
>
> > On Jan 4, 2023, at 3:15 PM, sebb  wrote:
> >
> > On Wed, 4 Jan 2023 at 18:18, Chris Lambertus  > > wrote:
> >>
> >>
> >>
> >>> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> >>>
> >>> The www-site builds failed recently, however this was not obvious
> >>> because the failures don't appear to have been reported anywhere
> >>> outside the dashboard.
> >>>
> >>> As I recall, the previous version of BuildBot would send an email for
> >>> broken (and mended) builds. This was very handy for detecting and
> >>> fixing errors.
> >>>
> >>> Are there any plans to provide something similar for the current Buildbot?
> >>>
> >>
> >> The functionality is available. Here is an example configuration for 
> >> setting up an email notifier.
> >
> > Thanks, AFAICT something similar is already set up here:
> >
> > https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
> >
> > However it does not seem to be working.
>
>
> I've opened INFRA-24056 to track this. The error is attached below. As a 
> stopgap, we can add an extraRecipients field, because it's not correctly 
> detecting the contributors, and is thus unable to generate a list of 
> "interested users."

I see.

> What list makes most sense to see failures related to the pelican websites 
> builds?

I'm not sure if there is a single list that would be correct for all
those builds.
However it would probably be OK to use the public list site-dev@a.o
which was used in the past for bb builds. See for example:

https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p

If this proves too noisy it might be necessary to redirect some build
mails elsewhere.

There also needs to be a way to ensure that script failures such as
the one below are brought to the attention of builds and/or infra.

> 2022-12-31 19:22:37+ [-] INVALID EMAIL: asfinfra
> 2022-12-31 19:22:37+ [-] sending mail (653 bytes) to []
> 2022-12-31 19:22:37+ [-] Starting factory 
> 
> 2022-12-31 19:22:37+ [ESMTPSender,client] Got exception when handling 
> reporter events
> Traceback (most recent call last):
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
> 1658, in _inlineCallbacks
> cast(Failure, result).throwExceptionIntoGenerator, gen
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
> 63, in run
> return f(*args, **kwargs)
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 
> 500, in throwExceptionIntoGenerato
> r
> return g.throw(self.type, self.value, self.tb)
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
> line 95, in _got_event
> log.err(e, 'Got exception when handling reporter events')
> ---  ---
>   File 
> "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", 
> line 93, in _got_event
> yield self.sendMessage(reports)
> twisted.mail._except.SMTPDeliveryError: No recipients accepted
> >>> MAIL FROM:
> <<< 250 2.1.0 Ok
>
>
> >
> > Sebb
> >> template=u'''\
> >> Build status: {{ summary }}
> >> Worker used: {{ workername }}
> >> URL: {{ build_url }}
> >> Blamelist: {{ blamelist | join(', ') }}
> >> Build Text: {{ build['state_string'] }}
> >> Status Detected: {{ status_detected }}
> >> {{ sourcestamps }}
> >>
> >> Steps:
> >> {% for step in build['steps'] %}
> >> {{ step['name'] }}: {{ step['results'] }}
> >> {% endfor %}
> >>
> >> -- ASF Buildbot
> >> '''
> >> simple_generator = reporters.BuildStatusGenerator(
> >> mode=('all'),
> >> builders=["infrastructure-test",],
> >> message_formatter=reporters.MessageFormatter(
> >> template=template,
> >> template_type='plain',
> >> want_properties=True,
> >> want_steps=True,
> >> ),
> >> )
> >> itn = reporters.MailNotifier(
> >> fromaddr="build...@apache.org",
> >> sendToInterestedUsers=True,
> >> generators=[simple_generator],
> >> extraRecipients=["gno...@infra.apache.org"],
> >> )
> >> bb.add_service(itn)
> >>
> >>
> >>
> >>
> >>> Sebb
>

Re: Detecting errors in Buildbot

2023-01-04 Thread fluxo 
> On Jan 4, 2023, at 3:15 PM, sebb  wrote:
> 
> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus  > wrote:
>> 
>> 
>> 
>>> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
>>> 
>>> The www-site builds failed recently, however this was not obvious
>>> because the failures don't appear to have been reported anywhere
>>> outside the dashboard.
>>> 
>>> As I recall, the previous version of BuildBot would send an email for
>>> broken (and mended) builds. This was very handy for detecting and
>>> fixing errors.
>>> 
>>> Are there any plans to provide something similar for the current Buildbot?
>>> 
>> 
>> The functionality is available. Here is an example configuration for setting 
>> up an email notifier.
> 
> Thanks, AFAICT something similar is already set up here:
> 
> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
> 
> However it does not seem to be working.


I've opened INFRA-24056 to track this. The error is attached below. As a 
stopgap, we can add an extraRecipients field, because it's not correctly 
detecting the contributors, and is thus unable to generate a list of 
"interested users." What list makes most sense to see failures related to the 
pelican websites builds?

2022-12-31 19:22:37+ [-] INVALID EMAIL: asfinfra
2022-12-31 19:22:37+ [-] sending mail (653 bytes) to []
2022-12-31 19:22:37+ [-] Starting factory 

2022-12-31 19:22:37+ [ESMTPSender,client] Got exception when handling 
reporter events
Traceback (most recent call last):
  File 
"/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
1658, in _inlineCallbacks
cast(Failure, result).throwExceptionIntoGenerator, gen
  File 
"/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 
63, in run
return f(*args, **kwargs)
  File 
"/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 
500, in throwExceptionIntoGenerato
r
return g.throw(self.type, self.value, self.tb)
  File 
"/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 
95, in _got_event
log.err(e, 'Got exception when handling reporter events')
---  ---
  File 
"/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 
93, in _got_event
yield self.sendMessage(reports)
twisted.mail._except.SMTPDeliveryError: No recipients accepted
>>> MAIL FROM:
<<< 250 2.1.0 Ok


> 
> Sebb
>> template=u'''\
>> Build status: {{ summary }}
>> Worker used: {{ workername }}
>> URL: {{ build_url }}
>> Blamelist: {{ blamelist | join(', ') }}
>> Build Text: {{ build['state_string'] }}
>> Status Detected: {{ status_detected }}
>> {{ sourcestamps }}
>> 
>> Steps:
>> {% for step in build['steps'] %}
>> {{ step['name'] }}: {{ step['results'] }}
>> {% endfor %}
>> 
>> -- ASF Buildbot
>> '''
>> simple_generator = reporters.BuildStatusGenerator(
>> mode=('all'),
>> builders=["infrastructure-test",],
>> message_formatter=reporters.MessageFormatter(
>> template=template,
>> template_type='plain',
>> want_properties=True,
>> want_steps=True,
>> ),
>> )
>> itn = reporters.MailNotifier(
>> fromaddr="build...@apache.org",
>> sendToInterestedUsers=True,
>> generators=[simple_generator],
>> extraRecipients=["gno...@infra.apache.org"],
>> )
>> bb.add_service(itn)
>> 
>> 
>> 
>> 
>>> Sebb

Re: Detecting errors in Buildbot

2023-01-04 Thread sebb 
On Wed, 4 Jan 2023 at 18:18, Chris Lambertus  wrote:
>
>
>
> > On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> >
> > The www-site builds failed recently, however this was not obvious
> > because the failures don't appear to have been reported anywhere
> > outside the dashboard.
> >
> > As I recall, the previous version of BuildBot would send an email for
> > broken (and mended) builds. This was very handy for detecting and
> > fixing errors.
> >
> > Are there any plans to provide something similar for the current Buildbot?
> >
>
> The functionality is available. Here is an example configuration for setting 
> up an email notifier.

Thanks, AFAICT something similar is already set up here:

https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56

However it does not seem to be working.

Sebb
> template=u'''\
> Build status: {{ summary }}
> Worker used: {{ workername }}
> URL: {{ build_url }}
> Blamelist: {{ blamelist | join(', ') }}
> Build Text: {{ build['state_string'] }}
> Status Detected: {{ status_detected }}
> {{ sourcestamps }}
>
> Steps:
> {% for step in build['steps'] %}
> {{ step['name'] }}: {{ step['results'] }}
> {% endfor %}
>
> -- ASF Buildbot
> '''
> simple_generator = reporters.BuildStatusGenerator(
> mode=('all'),
> builders=["infrastructure-test",],
> message_formatter=reporters.MessageFormatter(
> template=template,
> template_type='plain',
> want_properties=True,
> want_steps=True,
> ),
> )
> itn = reporters.MailNotifier(
> fromaddr="build...@apache.org",
> sendToInterestedUsers=True,
> generators=[simple_generator],
> extraRecipients=["gno...@infra.apache.org"],
> )
> bb.add_service(itn)
>
>
>
>
> > Sebb
>

Re: Detecting errors in Buildbot

2023-01-04 Thread Chris Lambertus 



> On Jan 4, 2023, at 4:03 AM, sebb  wrote:
> 
> The www-site builds failed recently, however this was not obvious
> because the failures don't appear to have been reported anywhere
> outside the dashboard.
> 
> As I recall, the previous version of BuildBot would send an email for
> broken (and mended) builds. This was very handy for detecting and
> fixing errors.
> 
> Are there any plans to provide something similar for the current Buildbot?
> 

The functionality is available. Here is an example configuration for setting up 
an email notifier.

template=u'''\
Build status: {{ summary }}
Worker used: {{ workername }}
URL: {{ build_url }}
Blamelist: {{ blamelist | join(', ') }}
Build Text: {{ build['state_string'] }}
Status Detected: {{ status_detected }}
{{ sourcestamps }}

Steps:
{% for step in build['steps'] %}
{{ step['name'] }}: {{ step['results'] }}
{% endfor %}

-- ASF Buildbot
'''
simple_generator = reporters.BuildStatusGenerator(
mode=('all'),
builders=["infrastructure-test",],
message_formatter=reporters.MessageFormatter(
template=template,
template_type='plain',
want_properties=True,
want_steps=True,
),
)
itn = reporters.MailNotifier(
fromaddr="build...@apache.org", 
sendToInterestedUsers=True, 
generators=[simple_generator],
extraRecipients=["gno...@infra.apache.org"],
)
bb.add_service(itn)




> Sebb