Re: Failing beam_SeedJob

[Alexey Romanenko]

Thanks Valentyn, it may help! 

I’m wondering if the seed job was broken because of Pablo’s change or it just 
happened the same issue like in this thread and it was fixed just by Jenkins 
restart.

> On 3 Jun 2022, at 15:22, Valentyn Tymofieiev  wrote:
> 
> I am not aware of a straightforward way. I have been using this script: 
> https://gist.github.com/tvalentyn/08cdb2836d311c55ae6ec09a276aea2a 
>  
> +Pablo Estrada  investigated this in 
> https://issues.apache.org/jira/browse/BEAM-14377 
>  but Pablo's changes were 
> rolled back because somehow seed job became broken.
> 
> On Fri, Jun 3, 2022 at 2:56 PM Alexey Romanenko  > wrote:
> Yes, it’s ok now. 
> Thanks all!
> 
> Side question - is there a way to run a Jenkins job against specific PR? I 
> tried to specify different git refs as an input parameter but it didn’t work 
> for me…
> 
> —
> Alexey 
> 
>> On 3 Jun 2022, at 05:02, Ahmet Altay > > wrote:
>> 
>> Seed job looks healthy again: 
>> https://ci-beam.apache.org/job/beam_SeedJob/9762/ 
>>  .
>> 
>> On Thu, Jun 2, 2022 at 10:24 AM Ryan Thompson > > wrote:
>> I've filed a jira to infra and asked on the slack channel. Hopefully someone 
>> there has the power to restart the server.
>> 
>> https://issues.apache.org/jira/browse/INFRA-23335 
>> 
>> 
>> On Thu, Jun 2, 2022 at 12:41 PM Kenneth Knowles > > wrote:
>> Commented on the Jira. Seems like it happens somewhat rarely, but is 
>> sometimes resolved by a restart, and sometimes has to do with some version 
>> mismatch issues. I did not find anything like a real root cause, just 
>> trial-and-error fixes. I'm not certain what may have occurred around May 19 
>> to this infrastructure. Hoping INFRA can help us sort it out.
>> 
>> Kenn
>> 
>> On Thu, Jun 2, 2022 at 9:15 AM Alexey Romanenko > > wrote:
>> Thanks Ahmet and Ryan for taking a look!
>> 
>> I agree that referenced commits seems are not related, that us why I was 
>> puzzled with this.
>> 
>>> On 2 Jun 2022, at 17:56, Ahmet Altay >> > wrote:
>>> 
>>> 
>>> 
>>> On Thu, Jun 2, 2022 at 8:53 AM Ryan Thompson >> > wrote:
>>> I asked in the slack channel to restart jenkins. I'm looking through the 
>>> past messages to see if there's someone there I can tag.
>>> 
>>> Thank you. Unfortunately, I do not believe we have a single expert we can 
>>> tag.
>>>  
>>> Am I right in understanding this service is managed by someone in apache 
>>> and not us?
>>> 
>>> It is mixed. ASF Infra works with a vendor and they run a hosted jenkins 
>>> instance. We control the configuration of the instance, and the worker 
>>> nodes.
>>>  
>>> 
>>> On Thu, Jun 2, 2022 at 11:47 AM Ahmet Altay >> > wrote:
>>> /cc @Kenneth Knowles 
>>> On Thu, Jun 2, 2022 at 8:44 AM Ahmet Altay >> > wrote:
>>> I do not have a great idea but googling about the error, similar errors 
>>> were resolved by restarting jenkins. We could try that. We may need to ask 
>>> infra. ( @Ryan Thompson  - could you please 
>>> infra to restart jenkins?)
>>> 
>>> I do not think this issue is related to any change in the source control. 
>>> The commit referenced in the first failed job is an unrelated doc change. 
>>> 
>>> 
>>> On Thu, Jun 2, 2022 at 7:57 AM Alexey Romanenko >> > wrote:
>>> I created a jira for this (not sure if it’s P0, but P1 for sure):
>>> https://issues.apache.org/jira/browse/BEAM-14548 
>>> 
>>> 
>>> Could someone, who has more knowledge than me in Beam 
>>> infrastructure/Jenkins, take a look, please?
>>> 
>>> —
>>> Alexey
>>> 
 On 31 May 2022, at 17:08, Alexey Romanenko >>> > wrote:
 
 The first failed job is https://ci-beam.apache.org/job/beam_SeedJob/9696/ 
 
 It fails with this error (that says not so much):
 
 Processing DSL script .test-infra/jenkins/job_00_seed.groovy
 Processing DSL script 
 .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
 Processing DSL script 
 .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
 Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
 ERROR: java.io.IOException: Failed to persist config.xml
 
  and I don’t see any recent changes for these files.
 
> On 31 May 2022, at 16:24, Alexey Romanenko  > wrote:
> 
> Hi everyone,
> 
> Jenkins 

Re: Failing beam_SeedJob

[Alexey Romanenko]

Yes, it’s ok now. 
Thanks all!

Side question - is there a way to run a Jenkins job against specific PR? I 
tried to specify different git refs as an input parameter but it didn’t work 
for me…

—
Alexey 

> On 3 Jun 2022, at 05:02, Ahmet Altay  wrote:
> 
> Seed job looks healthy again: 
> https://ci-beam.apache.org/job/beam_SeedJob/9762/ 
>  .
> 
> On Thu, Jun 2, 2022 at 10:24 AM Ryan Thompson  > wrote:
> I've filed a jira to infra and asked on the slack channel. Hopefully someone 
> there has the power to restart the server.
> 
> https://issues.apache.org/jira/browse/INFRA-23335 
> 
> 
> On Thu, Jun 2, 2022 at 12:41 PM Kenneth Knowles  > wrote:
> Commented on the Jira. Seems like it happens somewhat rarely, but is 
> sometimes resolved by a restart, and sometimes has to do with some version 
> mismatch issues. I did not find anything like a real root cause, just 
> trial-and-error fixes. I'm not certain what may have occurred around May 19 
> to this infrastructure. Hoping INFRA can help us sort it out.
> 
> Kenn
> 
> On Thu, Jun 2, 2022 at 9:15 AM Alexey Romanenko  > wrote:
> Thanks Ahmet and Ryan for taking a look!
> 
> I agree that referenced commits seems are not related, that us why I was 
> puzzled with this.
> 
>> On 2 Jun 2022, at 17:56, Ahmet Altay > > wrote:
>> 
>> 
>> 
>> On Thu, Jun 2, 2022 at 8:53 AM Ryan Thompson > > wrote:
>> I asked in the slack channel to restart jenkins. I'm looking through the 
>> past messages to see if there's someone there I can tag.
>> 
>> Thank you. Unfortunately, I do not believe we have a single expert we can 
>> tag.
>>  
>> Am I right in understanding this service is managed by someone in apache and 
>> not us?
>> 
>> It is mixed. ASF Infra works with a vendor and they run a hosted jenkins 
>> instance. We control the configuration of the instance, and the worker nodes.
>>  
>> 
>> On Thu, Jun 2, 2022 at 11:47 AM Ahmet Altay > > wrote:
>> /cc @Kenneth Knowles 
>> On Thu, Jun 2, 2022 at 8:44 AM Ahmet Altay > > wrote:
>> I do not have a great idea but googling about the error, similar errors were 
>> resolved by restarting jenkins. We could try that. We may need to ask infra. 
>> ( @Ryan Thompson  - could you please infra 
>> to restart jenkins?)
>> 
>> I do not think this issue is related to any change in the source control. 
>> The commit referenced in the first failed job is an unrelated doc change. 
>> 
>> 
>> On Thu, Jun 2, 2022 at 7:57 AM Alexey Romanenko > > wrote:
>> I created a jira for this (not sure if it’s P0, but P1 for sure):
>> https://issues.apache.org/jira/browse/BEAM-14548 
>> 
>> 
>> Could someone, who has more knowledge than me in Beam 
>> infrastructure/Jenkins, take a look, please?
>> 
>> —
>> Alexey
>> 
>>> On 31 May 2022, at 17:08, Alexey Romanenko >> > wrote:
>>> 
>>> The first failed job is https://ci-beam.apache.org/job/beam_SeedJob/9696/ 
>>> 
>>> It fails with this error (that says not so much):
>>> 
>>> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
>>> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
>>> Processing DSL script 
>>> .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
>>> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
>>> ERROR: java.io.IOException: Failed to persist config.xml
>>> 
>>>  and I don’t see any recent changes for these files.
>>> 
 On 31 May 2022, at 16:24, Alexey Romanenko >>> > wrote:
 
 Hi everyone,
 
 Jenkins job `beam_SeedJob` keeps failing starting from May 19th. The last 
 successful build [2] was 12 days ago.
 Does anyone know the reasons of this?
 
 —
 Alexey
 
 
 [1] https://ci-beam.apache.org/job/beam_SeedJob/ 
 
 [2] https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/ 
 

 
>>> 
>> 
> 

Re: Failing beam_SeedJob

[Kenneth Knowles]

Commented on the Jira. Seems like it happens somewhat rarely, but is
sometimes resolved by a restart, and sometimes has to do with some version
mismatch issues. I did not find anything like a real root cause, just
trial-and-error fixes. I'm not certain what may have occurred around May 19
to this infrastructure. Hoping INFRA can help us sort it out.

Kenn

On Thu, Jun 2, 2022 at 9:15 AM Alexey Romanenko 
wrote:

> Thanks Ahmet and Ryan for taking a look!
>
> I agree that referenced commits seems are not related, that us why I was
> puzzled with this.
>
> On 2 Jun 2022, at 17:56, Ahmet Altay  wrote:
>
>
>
> On Thu, Jun 2, 2022 at 8:53 AM Ryan Thompson 
> wrote:
>
>> I asked in the slack channel to restart jenkins. I'm looking through the
>> past messages to see if there's someone there I can tag.
>>
>
> Thank you. Unfortunately, I do not believe we have a single expert we can
> tag.
>
>
>> Am I right in understanding this service is managed by someone in apache
>> and not us?
>>
>
> It is mixed. ASF Infra works with a vendor and they run a hosted jenkins
> instance. We control the configuration of the instance, and the worker
> nodes.
>
>
>>
>> On Thu, Jun 2, 2022 at 11:47 AM Ahmet Altay  wrote:
>>
>>> /cc @Kenneth Knowles 
>>>
>>> On Thu, Jun 2, 2022 at 8:44 AM Ahmet Altay  wrote:
>>>
 I do not have a great idea but googling about the error, similar errors
 were resolved by restarting jenkins. We could try that. We may need to ask
 infra. ( @Ryan Thompson  - could you please
 infra to restart jenkins?)

 I do not think this issue is related to any change in the source
 control. The commit referenced in the first failed job is an unrelated doc
 change.


 On Thu, Jun 2, 2022 at 7:57 AM Alexey Romanenko <
 aromanenko@gmail.com> wrote:

> I created a jira for this (not sure if it’s P0, but P1 for sure):
> https://issues.apache.org/jira/browse/BEAM-14548
>
> Could someone, who has more knowledge than me in Beam
> infrastructure/Jenkins, take a look, please?
>
> —
> Alexey
>
> On 31 May 2022, at 17:08, Alexey Romanenko 
> wrote:
>
> The first failed job is
> https://ci-beam.apache.org/job/beam_SeedJob/9696/
> It fails with this error (that says not so much):
>
> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
> Processing DSL script
> .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
> Processing DSL script
> .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
> ERROR: java.io.IOException: Failed to persist config.xml
>
>  and I don’t see any recent changes for these files.
>
> On 31 May 2022, at 16:24, Alexey Romanenko 
> wrote:
>
> Hi everyone,
>
> Jenkins job `beam_SeedJob` keeps failing starting from May 19th. The
> last successful build [2] was 12 days ago.
> Does anyone know the reasons of this?
>
> —
> Alexey
>
>
> [1] https://ci-beam.apache.org/job/beam_SeedJob/
> [2] https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/
>
>
>
>
>

Re: Failing beam_SeedJob

[Alexey Romanenko]

Thanks Ahmet and Ryan for taking a look!

I agree that referenced commits seems are not related, that us why I was 
puzzled with this.

> On 2 Jun 2022, at 17:56, Ahmet Altay  wrote:
> 
> 
> 
> On Thu, Jun 2, 2022 at 8:53 AM Ryan Thompson  > wrote:
> I asked in the slack channel to restart jenkins. I'm looking through the past 
> messages to see if there's someone there I can tag.
> 
> Thank you. Unfortunately, I do not believe we have a single expert we can tag.
>  
> Am I right in understanding this service is managed by someone in apache and 
> not us?
> 
> It is mixed. ASF Infra works with a vendor and they run a hosted jenkins 
> instance. We control the configuration of the instance, and the worker nodes.
>  
> 
> On Thu, Jun 2, 2022 at 11:47 AM Ahmet Altay  > wrote:
> /cc @Kenneth Knowles 
> On Thu, Jun 2, 2022 at 8:44 AM Ahmet Altay  > wrote:
> I do not have a great idea but googling about the error, similar errors were 
> resolved by restarting jenkins. We could try that. We may need to ask infra. 
> ( @Ryan Thompson  - could you please infra to 
> restart jenkins?)
> 
> I do not think this issue is related to any change in the source control. The 
> commit referenced in the first failed job is an unrelated doc change. 
> 
> 
> On Thu, Jun 2, 2022 at 7:57 AM Alexey Romanenko  > wrote:
> I created a jira for this (not sure if it’s P0, but P1 for sure):
> https://issues.apache.org/jira/browse/BEAM-14548 
> 
> 
> Could someone, who has more knowledge than me in Beam infrastructure/Jenkins, 
> take a look, please?
> 
> —
> Alexey
> 
>> On 31 May 2022, at 17:08, Alexey Romanenko > > wrote:
>> 
>> The first failed job is https://ci-beam.apache.org/job/beam_SeedJob/9696/ 
>> 
>> It fails with this error (that says not so much):
>> 
>> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
>> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
>> Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
>> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
>> ERROR: java.io.IOException: Failed to persist config.xml
>> 
>>  and I don’t see any recent changes for these files.
>> 
>>> On 31 May 2022, at 16:24, Alexey Romanenko >> > wrote:
>>> 
>>> Hi everyone,
>>> 
>>> Jenkins job `beam_SeedJob` keeps failing starting from May 19th. The last 
>>> successful build [2] was 12 days ago.
>>> Does anyone know the reasons of this?
>>> 
>>> —
>>> Alexey
>>> 
>>> 
>>> [1] https://ci-beam.apache.org/job/beam_SeedJob/ 
>>> 
>>> [2] https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/ 
>>> 
>>> 
>>> 
>> 
> 

Re: Failing beam_SeedJob

[Alexey Romanenko]

I created a jira for this (not sure if it’s P0, but P1 for sure):
https://issues.apache.org/jira/browse/BEAM-14548 


Could someone, who has more knowledge than me in Beam infrastructure/Jenkins, 
take a look, please?

—
Alexey

> On 31 May 2022, at 17:08, Alexey Romanenko  wrote:
> 
> The first failed job is https://ci-beam.apache.org/job/beam_SeedJob/9696/ 
> 
> It fails with this error (that says not so much):
> 
> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
> Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
> ERROR: java.io.IOException: Failed to persist config.xml
> 
>  and I don’t see any recent changes for these files.
> 
>> On 31 May 2022, at 16:24, Alexey Romanenko > > wrote:
>> 
>> Hi everyone,
>> 
>> Jenkins job `beam_SeedJob` keeps failing starting from May 19th. The last 
>> successful build [2] was 12 days ago.
>> Does anyone know the reasons of this?
>> 
>> —
>> Alexey
>> 
>> 
>> [1] https://ci-beam.apache.org/job/beam_SeedJob/ 
>> 
>> [2] https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/ 
>> 
>>  
>> 
> 

Re: Failing beam_SeedJob

[Alexey Romanenko]

The first failed job is https://ci-beam.apache.org/job/beam_SeedJob/9696/ 

It fails with this error (that says not so much):

Processing DSL script .test-infra/jenkins/job_00_seed.groovy
Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
ERROR: java.io.IOException: Failed to persist config.xml

 and I don’t see any recent changes for these files.

> On 31 May 2022, at 16:24, Alexey Romanenko  wrote:
> 
> Hi everyone,
> 
> Jenkins job `beam_SeedJob` keeps failing starting from May 19th. The last 
> successful build [2] was 12 days ago.
> Does anyone know the reasons of this?
> 
> —
> Alexey
> 
> 
> [1] https://ci-beam.apache.org/job/beam_SeedJob/ 
> 
> [2] https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/ 
> 
>   
> 

Failing beam_SeedJob

[Alexey Romanenko]

Hi everyone,

Jenkins job `beam_SeedJob` keeps failing starting from May 19th. The last 
successful build [2] was 12 days ago.
Does anyone know the reasons of this?

—
Alexey


[1] https://ci-beam.apache.org/job/beam_SeedJob/ 

[2] https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/