Re: Jenkins CI currently unavailable

[Alexey Romanenko]

Additionally to what Kenn said, we have some documentation here:
https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips 


Though, not sure how up-to-date it is.

—
Alexey

> On 14 Jun 2022, at 16:42, Kenneth Knowles  wrote:
> 
> The UI is https://ci-beam.apache.org/  and it is 
> integrated with ASF's LDAP. I don't know if this URL is documented anywhere.
> 
> Usage of the UI is standard Jenkins. You can select any job and click "build 
> with parameters" and put in a git ref to build from.
> 
> Kenn
> 
> On Mon, Jun 13, 2022 at 5:54 PM Reuven Lax  > wrote:
> I am a committer, but I'm not sure how to even get to the Jenkins UI! Is this 
> documented anywhere?
> 
> This PR affects how the Dataflow runner works, so we should run Dataflow 
> postcommits on it.
> 
> On Mon, Jun 13, 2022 at 4:22 PM Kiley Sok  > wrote:
> Reuven, it looks like yours may have been stuck in a weird state when we 
> re-enabled the precommits. I kicked off the tests on your PR with 'retest 
> this please'
> 
> To clarify, precommits are working as before (pr comment and update 
> triggered) and so you should be able to check in code. 
> 
> If you want further testing with post commits, you'll need a committer to 
> manually trigger them on the Jenkins UI. I believe you can do this by 'Build 
> with Parameter' and putting in the PR number (correct me if I'm wrong @Robert 
> Burke ). If there are no objections, I can 
> re-enable triggers for the common postcommits. 
> 
> 
> 
> On Mon, Jun 13, 2022 at 4:06 PM Reuven Lax  > wrote:
> Are there any pointers on how to manually trigger the runs?
> 
> On Mon, Jun 13, 2022 at 4:04 PM Robert Burke  > wrote:
> You know, I do forget that committers can manually trigger Jenkins runs.
> 
> And after fiddling with the Jenkins options and filling in the expected, but 
> missing PR number parameter i think I've managed it.
> 
> Thanks for the reminder!
> 
> On Mon, Jun 13, 2022, 3:38 PM Kiley Sok  > wrote:
> Can you run the post commits from the Jenkins UI to unblock? We've turned off 
> the triggers for all post commits, but could turn it on for a select few as 
> well.
> 
> On Mon, Jun 13, 2022 at 3:31 PM Robert Burke  > wrote:
> There are a couple of Go SDK PRs that are basically blocked on final manual 
> runs of the post commits, that we'd like to get in for the 2.40 cut.
> 
> Are we intending on delaying the 2.40 cut a little bit so PRs like those can 
> make it in?
> 
> 
> On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay  > wrote:
> Thank you all for working on this.
> 
> On Mon, Jun 13, 2022 at 10:09 AM Kenneth Knowles  > wrote:
> Yes, the ghprb plugin was disabled. That was the entire action. I believe my 
> PR will reduce the load caused by the ghprb plugin; we are currently 
> restarting Jenkins to re-enable it. So we can unfreeze master as soon as 
> Jenkins reboots. Basically, if your PR has a precommit status great, 
> otherwise please wait.
> 
> What we lose from my change is postcommit comment triggers. I see how this is 
> unfortunate for our established release process that runs them all on the 
> release branch.
> 
> Going forward, we are using old/unmaintained plugins and need to stop relying 
> on them. There are two obvious choices:
> 
> (1) use some "latest and greatest" Jenkins plugin - most likely the 
> multibranch pipeline plugin (aka Jenkinsfile plugin)
> (2) use GitHub Actions
> 
> I believe both of these will be comparable in migration effort. I'm in favor 
> of expanding our GitHub Actions usage to take over what we do with Jenkins. 
> We have figured out how to have self-hosted workers, just like we do for 
> Jenkins. I don't know what other pitfalls may exist.
> 
> A good first step would be to bring the GitHub Actions precommits to parity 
> with the Jenkins precommits.
> 
> +1. After spending some time, these two plugins are not very compatible and 
> migration to the new plugin would itself be a sufficiently large migration. 
> We are using GH actions to an extent today and in general they were working 
> fine.
>  
> 
> Kenn
> 
> On Mon, Jun 13, 2022 at 9:44 AM Brian Hulette  > wrote:
> Can someone familiar with this clarify the current status? It looks like 
> PostCommits and PreCommit_Cron jobs are still running on a schedule, e.g. 
> [1,2]. Was the ghprb plugin (responsible from triggering jobs based on new 
> PRs and comments) just disabled?
> 
> If that's the case then we have a full suite of PostCommits, but the only 
> precommit checks we have are GitHub Actions checks. These are pretty 
> thorough, but off the top of my head a decent amount is missing:
> - No PyLint, PyDoc precommits
> - We can't trigger 

Re: Jenkins CI currently unavailable

[Kenneth Knowles]

The UI is https://ci-beam.apache.org/ and it is integrated with ASF's LDAP.
I don't know if this URL is documented anywhere.

Usage of the UI is standard Jenkins. You can select any job and click
"build with parameters" and put in a git ref to build from.

Kenn

On Mon, Jun 13, 2022 at 5:54 PM Reuven Lax  wrote:

> I am a committer, but I'm not sure how to even get to the Jenkins UI! Is
> this documented anywhere?
>
> This PR affects how the Dataflow runner works, so we should run Dataflow
> postcommits on it.
>
> On Mon, Jun 13, 2022 at 4:22 PM Kiley Sok  wrote:
>
>> Reuven, it looks like yours may have been stuck in a weird state when we
>> re-enabled the precommits. I kicked off the tests on your PR with 'retest
>> this please'
>>
>> To clarify, precommits are working as before (pr comment and update
>> triggered) and so you should be able to check in code.
>>
>> If you want further testing with post commits, you'll need a committer to
>> manually trigger them on the Jenkins UI. I believe you can do this by
>> 'Build with Parameter' and putting in the PR number (correct me if I'm
>> wrong @Robert Burke ). If there are no objections, I
>> can re-enable triggers for the common postcommits.
>>
>>
>>
>> On Mon, Jun 13, 2022 at 4:06 PM Reuven Lax  wrote:
>>
>>> Are there any pointers on how to manually trigger the runs?
>>>
>>> On Mon, Jun 13, 2022 at 4:04 PM Robert Burke  wrote:
>>>
 You know, I do forget that committers can manually trigger Jenkins runs.

 And after fiddling with the Jenkins options and filling in the
 expected, but missing PR number parameter i think I've managed it.

 Thanks for the reminder!

 On Mon, Jun 13, 2022, 3:38 PM Kiley Sok  wrote:

> Can you run the post commits from the Jenkins UI to unblock? We've
> turned off the triggers for all post commits, but could turn it on for a
> select few as well.
>
> On Mon, Jun 13, 2022 at 3:31 PM Robert Burke 
> wrote:
>
>> There are a couple of Go SDK PRs that are basically blocked on final
>> manual runs of the post commits, that we'd like to get in for the 2.40 
>> cut.
>>
>> Are we intending on delaying the 2.40 cut a little bit so PRs like
>> those can make it in?
>>
>>
>> On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay  wrote:
>>
>>> Thank you all for working on this.
>>>
>>> On Mon, Jun 13, 2022 at 10:09 AM Kenneth Knowles 
>>> wrote:
>>>
 Yes, the ghprb plugin was disabled. That was the entire action. I
 believe my PR will reduce the load caused by the ghprb plugin; we are
 currently restarting Jenkins to re-enable it. So we can unfreeze 
 master as
 soon as Jenkins reboots. Basically, if your PR has a precommit status
 great, otherwise please wait.

 What we lose from my change is postcommit comment triggers. I see
 how this is unfortunate for our established release process that runs 
 them
 all on the release branch.

 Going forward, we are using old/unmaintained plugins and need to
 stop relying on them. There are two obvious choices:

 (1) use some "latest and greatest" Jenkins plugin - most likely the
 multibranch pipeline plugin (aka Jenkinsfile plugin)
 (2) use GitHub Actions

 I believe both of these will be comparable in migration effort. I'm
 in favor of expanding our GitHub Actions usage to take over what we do 
 with
 Jenkins. We have figured out how to have self-hosted workers, just 
 like we
 do for Jenkins. I don't know what other pitfalls may exist.

 A good first step would be to bring the GitHub Actions precommits
 to parity with the Jenkins precommits.

>>>
>>> +1. After spending some time, these two plugins are not very
>>> compatible and migration to the new plugin would itself be a 
>>> sufficiently
>>> large migration. We are using GH actions to an extent today and in 
>>> general
>>> they were working fine.
>>>
>>>

 Kenn

 On Mon, Jun 13, 2022 at 9:44 AM Brian Hulette 
 wrote:

> Can someone familiar with this clarify the current status? It
> looks like PostCommits and PreCommit_Cron jobs are still running on a
> schedule, e.g. [1,2]. Was the ghprb plugin (responsible from 
> triggering
> jobs based on new PRs and comments) just disabled?
>
> If that's the case then we have a full suite of PostCommits, but
> the only precommit checks we have are GitHub Actions checks. These are
> pretty thorough, but off the top of my head a decent amount is 
> missing:
> - No PyLint, PyDoc precommits
> - We can't trigger PostCommits before merge
> - Java build doesn't have null checker? (I know 

Re: Jenkins CI currently unavailable

[Kenneth Knowles]

I just also want to say that you can run the tests locally and have people
believe you when you say they passed. It isn't as good as CI but it is
better than nothing. I am referring to the case where you want to manually
add a postcommit status to your PR. If you orchestrate with gradle then you
can add `--scan` to produce an artifact you can share and link on the PR.
Just listing a number of workarounds that can increase confidence while
this shakes out.

On Mon, Jun 13, 2022 at 5:03 PM Ahmet Altay  wrote:

>
>
> On Mon, Jun 13, 2022 at 4:54 PM Robert Burke  wrote:
>
>> Logged in as a Committer, You need to open the configuration menu for the
>> task, add in a string parameter for 'ghprPullId'. Save.
>>
>> Then you can go to Build With Parameters, populate the new field with the
>> PR number
>> And the sha1 with origin/pr//merge
>>
>> The you can kick off the job. It doesn't automatically link to the PR, so
>> you'll need to copy paste the run info into the PR.
>>
>> This matches what the normal runs seem to do. I have confirmed that it
>> executes everything with the correct commits and so on.
>>
>> It's unfortunate that the PR variants weren't already configured to
>> simply accept a PR number.
>>
>
> For future reference, there is also
> https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips#JenkinsTips-Triggeringjobs
> with similar information
>
>
>>
>> On Mon, Jun 13, 2022, 4:22 PM Kiley Sok  wrote:
>>
>>> Reuven, it looks like yours may have been stuck in a weird state when we
>>> re-enabled the precommits. I kicked off the tests on your PR with 'retest
>>> this please'
>>>
>>> To clarify, precommits are working as before (pr comment and update
>>> triggered) and so you should be able to check in code.
>>>
>>
> To be explicit: Is the master unfrozen at this point?
>
>
>>
>>> If you want further testing with post commits, you'll need a committer
>>> to manually trigger them on the Jenkins UI. I believe you can do this by
>>> 'Build with Parameter' and putting in the PR number (correct me if I'm
>>> wrong @Robert Burke ). If there are no objections,
>>> I can re-enable triggers for the common postcommits.
>>>
>>>
>>>
>>> On Mon, Jun 13, 2022 at 4:06 PM Reuven Lax  wrote:
>>>
 Are there any pointers on how to manually trigger the runs?

 On Mon, Jun 13, 2022 at 4:04 PM Robert Burke 
 wrote:

> You know, I do forget that committers can manually trigger Jenkins
> runs.
>
> And after fiddling with the Jenkins options and filling in the
> expected, but missing PR number parameter i think I've managed it.
>
> Thanks for the reminder!
>
> On Mon, Jun 13, 2022, 3:38 PM Kiley Sok  wrote:
>
>> Can you run the post commits from the Jenkins UI to unblock? We've
>> turned off the triggers for all post commits, but could turn it on for a
>> select few as well.
>>
>> On Mon, Jun 13, 2022 at 3:31 PM Robert Burke 
>> wrote:
>>
>>> There are a couple of Go SDK PRs that are basically blocked on final
>>> manual runs of the post commits, that we'd like to get in for the 2.40 
>>> cut.
>>>
>>> Are we intending on delaying the 2.40 cut a little bit so PRs like
>>> those can make it in?
>>>
>>
> I assume this is no longer needed since you were able to trigger the post
> commits manually?
>
>
>>
>>>
>>> On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay  wrote:
>>>
 Thank you all for working on this.

 On Mon, Jun 13, 2022 at 10:09 AM Kenneth Knowles 
 wrote:

> Yes, the ghprb plugin was disabled. That was the entire action. I
> believe my PR will reduce the load caused by the ghprb plugin; we are
> currently restarting Jenkins to re-enable it. So we can unfreeze 
> master as
> soon as Jenkins reboots. Basically, if your PR has a precommit status
> great, otherwise please wait.
>
> What we lose from my change is postcommit comment triggers. I see
> how this is unfortunate for our established release process that runs 
> them
> all on the release branch.
>
> Going forward, we are using old/unmaintained plugins and need to
> stop relying on them. There are two obvious choices:
>
> (1) use some "latest and greatest" Jenkins plugin - most likely
> the multibranch pipeline plugin (aka Jenkinsfile plugin)
> (2) use GitHub Actions
>
> I believe both of these will be comparable in migration effort.
> I'm in favor of expanding our GitHub Actions usage to take over what 
> we do
> with Jenkins. We have figured out how to have self-hosted workers, 
> just
> like we do for Jenkins. I don't know what other pitfalls may exist.
>
> A good first step would be to bring the GitHub Actions precommits
> to parity with the Jenkins precommits.
>

Re: Jenkins CI currently unavailable

[Robert Burke]

Logged in as a Committer, You need to open the configuration menu for the
task, add in a string parameter for 'ghprPullId'. Save.

Then you can go to Build With Parameters, populate the new field with the
PR number
And the sha1 with origin/pr//merge

The you can kick off the job. It doesn't automatically link to the PR, so
you'll need to copy paste the run info into the PR.

This matches what the normal runs seem to do. I have confirmed that it
executes everything with the correct commits and so on.

It's unfortunate that the PR variants weren't already configured to simply
accept a PR number.

On Mon, Jun 13, 2022, 4:22 PM Kiley Sok  wrote:

> Reuven, it looks like yours may have been stuck in a weird state when we
> re-enabled the precommits. I kicked off the tests on your PR with 'retest
> this please'
>
> To clarify, precommits are working as before (pr comment and update
> triggered) and so you should be able to check in code.
>
> If you want further testing with post commits, you'll need a committer to
> manually trigger them on the Jenkins UI. I believe you can do this by
> 'Build with Parameter' and putting in the PR number (correct me if I'm
> wrong @Robert Burke ). If there are no objections, I
> can re-enable triggers for the common postcommits.
>
>
>
> On Mon, Jun 13, 2022 at 4:06 PM Reuven Lax  wrote:
>
>> Are there any pointers on how to manually trigger the runs?
>>
>> On Mon, Jun 13, 2022 at 4:04 PM Robert Burke  wrote:
>>
>>> You know, I do forget that committers can manually trigger Jenkins runs.
>>>
>>> And after fiddling with the Jenkins options and filling in the expected,
>>> but missing PR number parameter i think I've managed it.
>>>
>>> Thanks for the reminder!
>>>
>>> On Mon, Jun 13, 2022, 3:38 PM Kiley Sok  wrote:
>>>
 Can you run the post commits from the Jenkins UI to unblock? We've
 turned off the triggers for all post commits, but could turn it on for a
 select few as well.

 On Mon, Jun 13, 2022 at 3:31 PM Robert Burke 
 wrote:

> There are a couple of Go SDK PRs that are basically blocked on final
> manual runs of the post commits, that we'd like to get in for the 2.40 
> cut.
>
> Are we intending on delaying the 2.40 cut a little bit so PRs like
> those can make it in?
>
>
> On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay  wrote:
>
>> Thank you all for working on this.
>>
>> On Mon, Jun 13, 2022 at 10:09 AM Kenneth Knowles 
>> wrote:
>>
>>> Yes, the ghprb plugin was disabled. That was the entire action. I
>>> believe my PR will reduce the load caused by the ghprb plugin; we are
>>> currently restarting Jenkins to re-enable it. So we can unfreeze master 
>>> as
>>> soon as Jenkins reboots. Basically, if your PR has a precommit status
>>> great, otherwise please wait.
>>>
>>> What we lose from my change is postcommit comment triggers. I see
>>> how this is unfortunate for our established release process that runs 
>>> them
>>> all on the release branch.
>>>
>>> Going forward, we are using old/unmaintained plugins and need to
>>> stop relying on them. There are two obvious choices:
>>>
>>> (1) use some "latest and greatest" Jenkins plugin - most likely the
>>> multibranch pipeline plugin (aka Jenkinsfile plugin)
>>> (2) use GitHub Actions
>>>
>>> I believe both of these will be comparable in migration effort. I'm
>>> in favor of expanding our GitHub Actions usage to take over what we do 
>>> with
>>> Jenkins. We have figured out how to have self-hosted workers, just like 
>>> we
>>> do for Jenkins. I don't know what other pitfalls may exist.
>>>
>>> A good first step would be to bring the GitHub Actions precommits to
>>> parity with the Jenkins precommits.
>>>
>>
>> +1. After spending some time, these two plugins are not very
>> compatible and migration to the new plugin would itself be a sufficiently
>> large migration. We are using GH actions to an extent today and in 
>> general
>> they were working fine.
>>
>>
>>>
>>> Kenn
>>>
>>> On Mon, Jun 13, 2022 at 9:44 AM Brian Hulette 
>>> wrote:
>>>
 Can someone familiar with this clarify the current status? It looks
 like PostCommits and PreCommit_Cron jobs are still running on a 
 schedule,
 e.g. [1,2]. Was the ghprb plugin (responsible from triggering jobs 
 based on
 new PRs and comments) just disabled?

 If that's the case then we have a full suite of PostCommits, but
 the only precommit checks we have are GitHub Actions checks. These are
 pretty thorough, but off the top of my head a decent amount is missing:
 - No PyLint, PyDoc precommits
 - We can't trigger PostCommits before merge
 - Java build doesn't have null checker? (I know Jenkins java
 PreCommit turns on 

Re: Jenkins CI currently unavailable

[Robert Burke]

You know, I do forget that committers can manually trigger Jenkins runs.

And after fiddling with the Jenkins options and filling in the expected,
but missing PR number parameter i think I've managed it.

Thanks for the reminder!

On Mon, Jun 13, 2022, 3:38 PM Kiley Sok  wrote:

> Can you run the post commits from the Jenkins UI to unblock? We've turned
> off the triggers for all post commits, but could turn it on for a select
> few as well.
>
> On Mon, Jun 13, 2022 at 3:31 PM Robert Burke  wrote:
>
>> There are a couple of Go SDK PRs that are basically blocked on final
>> manual runs of the post commits, that we'd like to get in for the 2.40 cut.
>>
>> Are we intending on delaying the 2.40 cut a little bit so PRs like those
>> can make it in?
>>
>>
>> On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay  wrote:
>>
>>> Thank you all for working on this.
>>>
>>> On Mon, Jun 13, 2022 at 10:09 AM Kenneth Knowles 
>>> wrote:
>>>
 Yes, the ghprb plugin was disabled. That was the entire action. I
 believe my PR will reduce the load caused by the ghprb plugin; we are
 currently restarting Jenkins to re-enable it. So we can unfreeze master as
 soon as Jenkins reboots. Basically, if your PR has a precommit status
 great, otherwise please wait.

 What we lose from my change is postcommit comment triggers. I see how
 this is unfortunate for our established release process that runs them all
 on the release branch.

 Going forward, we are using old/unmaintained plugins and need to stop
 relying on them. There are two obvious choices:

 (1) use some "latest and greatest" Jenkins plugin - most likely the
 multibranch pipeline plugin (aka Jenkinsfile plugin)
 (2) use GitHub Actions

 I believe both of these will be comparable in migration effort. I'm in
 favor of expanding our GitHub Actions usage to take over what we do with
 Jenkins. We have figured out how to have self-hosted workers, just like we
 do for Jenkins. I don't know what other pitfalls may exist.

 A good first step would be to bring the GitHub Actions precommits to
 parity with the Jenkins precommits.

>>>
>>> +1. After spending some time, these two plugins are not very compatible
>>> and migration to the new plugin would itself be a sufficiently large
>>> migration. We are using GH actions to an extent today and in general they
>>> were working fine.
>>>
>>>

 Kenn

 On Mon, Jun 13, 2022 at 9:44 AM Brian Hulette 
 wrote:

> Can someone familiar with this clarify the current status? It looks
> like PostCommits and PreCommit_Cron jobs are still running on a schedule,
> e.g. [1,2]. Was the ghprb plugin (responsible from triggering jobs based 
> on
> new PRs and comments) just disabled?
>
> If that's the case then we have a full suite of PostCommits, but the
> only precommit checks we have are GitHub Actions checks. These are pretty
> thorough, but off the top of my head a decent amount is missing:
> - No PyLint, PyDoc precommits
> - We can't trigger PostCommits before merge
> - Java build doesn't have null checker? (I know Jenkins java PreCommit
> turns on null checker, I'm not sure about GitHub actions)
>
> It would be painful to untangle the inevitable PostCommit breakages if
> we keep submitting code, so I do think a code freeze is in order, but it's
> a very inopportune time given the release cut is this week...
>
> Brian
>
> [1] https://ci-beam.apache.org/job/beam_PostCommit_Python38/
> [2] https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>
>
> On Sun, Jun 12, 2022 at 9:04 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> Should we install a code commit freeze till this is addressed to
>> maintain code health ? I noticed that it's easy to miss certain 
>> untriggered
>> test suites for new PRs.
>>
>> Thanks,
>> Cham
>>
>> On Sun, Jun 12, 2022 at 5:25 PM Danny McCormick <
>> dannymccorm...@google.com> wrote:
>>
>>> In case anyone else is planning on taking a look at this, I did a
>>> deep dive on it and wrote up my findings here -
>>> https://docs.google.com/document/d/10qyUsvB_uVy5jftfTiwohlvN8Qwix5AuadssyoC4JsE/edit?usp=sharing
>>>
>>> My high level takeaways were:
>>> 1. I don't think this is actually caused by Issues, I think it's a
>>> structural issue with the plugin. At worst, Issues very minorly 
>>> contributed
>>> to it (with a smaller impact than adding an extra build trigger 
>>> would/did),
>>> but I'm not even sure that is true. As a result, our fix probably needs 
>>> to
>>> focus on patching or switching to a new plugin. 2. Patching the
>>> plugin would be hard.
>>> 3. I agree that switching to a different plugin is the best path
>>> forward.
>>>
>>> If anyone else is looking into this 

Re: Jenkins CI currently unavailable

[Robert Burke]

There are a couple of Go SDK PRs that are basically blocked on final manual
runs of the post commits, that we'd like to get in for the 2.40 cut.

Are we intending on delaying the 2.40 cut a little bit so PRs like those
can make it in?


On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay  wrote:

> Thank you all for working on this.
>
> On Mon, Jun 13, 2022 at 10:09 AM Kenneth Knowles  wrote:
>
>> Yes, the ghprb plugin was disabled. That was the entire action. I believe
>> my PR will reduce the load caused by the ghprb plugin; we are currently
>> restarting Jenkins to re-enable it. So we can unfreeze master as soon as
>> Jenkins reboots. Basically, if your PR has a precommit status great,
>> otherwise please wait.
>>
>> What we lose from my change is postcommit comment triggers. I see how
>> this is unfortunate for our established release process that runs them all
>> on the release branch.
>>
>> Going forward, we are using old/unmaintained plugins and need to stop
>> relying on them. There are two obvious choices:
>>
>> (1) use some "latest and greatest" Jenkins plugin - most likely the
>> multibranch pipeline plugin (aka Jenkinsfile plugin)
>> (2) use GitHub Actions
>>
>> I believe both of these will be comparable in migration effort. I'm in
>> favor of expanding our GitHub Actions usage to take over what we do with
>> Jenkins. We have figured out how to have self-hosted workers, just like we
>> do for Jenkins. I don't know what other pitfalls may exist.
>>
>> A good first step would be to bring the GitHub Actions precommits to
>> parity with the Jenkins precommits.
>>
>
> +1. After spending some time, these two plugins are not very compatible
> and migration to the new plugin would itself be a sufficiently large
> migration. We are using GH actions to an extent today and in general they
> were working fine.
>
>
>>
>> Kenn
>>
>> On Mon, Jun 13, 2022 at 9:44 AM Brian Hulette 
>> wrote:
>>
>>> Can someone familiar with this clarify the current status? It looks like
>>> PostCommits and PreCommit_Cron jobs are still running on a schedule, e.g.
>>> [1,2]. Was the ghprb plugin (responsible from triggering jobs based on new
>>> PRs and comments) just disabled?
>>>
>>> If that's the case then we have a full suite of PostCommits, but the
>>> only precommit checks we have are GitHub Actions checks. These are pretty
>>> thorough, but off the top of my head a decent amount is missing:
>>> - No PyLint, PyDoc precommits
>>> - We can't trigger PostCommits before merge
>>> - Java build doesn't have null checker? (I know Jenkins java PreCommit
>>> turns on null checker, I'm not sure about GitHub actions)
>>>
>>> It would be painful to untangle the inevitable PostCommit breakages if
>>> we keep submitting code, so I do think a code freeze is in order, but it's
>>> a very inopportune time given the release cut is this week...
>>>
>>> Brian
>>>
>>> [1] https://ci-beam.apache.org/job/beam_PostCommit_Python38/
>>> [2] https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>>>
>>>
>>> On Sun, Jun 12, 2022 at 9:04 PM Chamikara Jayalath 
>>> wrote:
>>>
 Should we install a code commit freeze till this is addressed to
 maintain code health ? I noticed that it's easy to miss certain untriggered
 test suites for new PRs.

 Thanks,
 Cham

 On Sun, Jun 12, 2022 at 5:25 PM Danny McCormick <
 dannymccorm...@google.com> wrote:

> In case anyone else is planning on taking a look at this, I did a deep
> dive on it and wrote up my findings here -
> https://docs.google.com/document/d/10qyUsvB_uVy5jftfTiwohlvN8Qwix5AuadssyoC4JsE/edit?usp=sharing
>
> My high level takeaways were:
> 1. I don't think this is actually caused by Issues, I think it's a
> structural issue with the plugin. At worst, Issues very minorly 
> contributed
> to it (with a smaller impact than adding an extra build trigger 
> would/did),
> but I'm not even sure that is true. As a result, our fix probably needs to
> focus on patching or switching to a new plugin. 2. Patching the
> plugin would be hard.
> 3. I agree that switching to a different plugin is the best path
> forward.
>
> If anyone else is looking into this and wants edit permissions on the
> doc, let me know!
>
> Thanks,
> Danny
>
> On Fri, Jun 10, 2022 at 5:21 PM Kiley Sok  wrote:
>
>> There's currently an issue with too many connections from Jenkins to
>> Github bringing down Jenkins. The hypothesis is that we were already 
>> close
>> to the limit, but the extra triggers from GH issues pushed it over the 
>> edge.
>>
>> We're looking to switch to a different plugin that'll hopefully
>> resolve the issue.
>>
>> More info: https://issues.apache.org/jira/browse/INFRA-23358 and in
>> INFRA slack channel.
>>
>> Thanks for your patience!
>>
>

Re: Jenkins CI currently unavailable

[Kenneth Knowles]

Yes, the ghprb plugin was disabled. That was the entire action. I believe
my PR will reduce the load caused by the ghprb plugin; we are currently
restarting Jenkins to re-enable it. So we can unfreeze master as soon as
Jenkins reboots. Basically, if your PR has a precommit status great,
otherwise please wait.

What we lose from my change is postcommit comment triggers. I see how this
is unfortunate for our established release process that runs them all on
the release branch.

Going forward, we are using old/unmaintained plugins and need to stop
relying on them. There are two obvious choices:

(1) use some "latest and greatest" Jenkins plugin - most likely the
multibranch pipeline plugin (aka Jenkinsfile plugin)
(2) use GitHub Actions

I believe both of these will be comparable in migration effort. I'm in
favor of expanding our GitHub Actions usage to take over what we do with
Jenkins. We have figured out how to have self-hosted workers, just like we
do for Jenkins. I don't know what other pitfalls may exist.

A good first step would be to bring the GitHub Actions precommits to parity
with the Jenkins precommits.

Kenn

On Mon, Jun 13, 2022 at 9:44 AM Brian Hulette  wrote:

> Can someone familiar with this clarify the current status? It looks like
> PostCommits and PreCommit_Cron jobs are still running on a schedule, e.g.
> [1,2]. Was the ghprb plugin (responsible from triggering jobs based on new
> PRs and comments) just disabled?
>
> If that's the case then we have a full suite of PostCommits, but the only
> precommit checks we have are GitHub Actions checks. These are pretty
> thorough, but off the top of my head a decent amount is missing:
> - No PyLint, PyDoc precommits
> - We can't trigger PostCommits before merge
> - Java build doesn't have null checker? (I know Jenkins java PreCommit
> turns on null checker, I'm not sure about GitHub actions)
>
> It would be painful to untangle the inevitable PostCommit breakages if we
> keep submitting code, so I do think a code freeze is in order, but it's a
> very inopportune time given the release cut is this week...
>
> Brian
>
> [1] https://ci-beam.apache.org/job/beam_PostCommit_Python38/
> [2] https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>
>
> On Sun, Jun 12, 2022 at 9:04 PM Chamikara Jayalath 
> wrote:
>
>> Should we install a code commit freeze till this is addressed to maintain
>> code health ? I noticed that it's easy to miss certain untriggered test
>> suites for new PRs.
>>
>> Thanks,
>> Cham
>>
>> On Sun, Jun 12, 2022 at 5:25 PM Danny McCormick <
>> dannymccorm...@google.com> wrote:
>>
>>> In case anyone else is planning on taking a look at this, I did a deep
>>> dive on it and wrote up my findings here -
>>> https://docs.google.com/document/d/10qyUsvB_uVy5jftfTiwohlvN8Qwix5AuadssyoC4JsE/edit?usp=sharing
>>>
>>> My high level takeaways were:
>>> 1. I don't think this is actually caused by Issues, I think it's a
>>> structural issue with the plugin. At worst, Issues very minorly contributed
>>> to it (with a smaller impact than adding an extra build trigger would/did),
>>> but I'm not even sure that is true. As a result, our fix probably needs to
>>> focus on patching or switching to a new plugin. 2. Patching the plugin
>>> would be hard.
>>> 3. I agree that switching to a different plugin is the best path forward.
>>>
>>> If anyone else is looking into this and wants edit permissions on the
>>> doc, let me know!
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Fri, Jun 10, 2022 at 5:21 PM Kiley Sok  wrote:
>>>
 There's currently an issue with too many connections from Jenkins to
 Github bringing down Jenkins. The hypothesis is that we were already close
 to the limit, but the extra triggers from GH issues pushed it over the 
 edge.

 We're looking to switch to a different plugin that'll hopefully resolve
 the issue.

 More info: https://issues.apache.org/jira/browse/INFRA-23358 and in
 INFRA slack channel.

 Thanks for your patience!

>>>

Re: Jenkins CI currently unavailable

[Kenneth Knowles]

I definitely think we should freeze until we have precommits re-enabled. We
have a hypothesis that https://github.com/apache/beam/pull/21821 may reduce
the impact on the Jenkins master. We would lose phrase triggering on
postcommits, but we would still have it on precommits for retriggering
after flakes.

Kenn

On Sun, Jun 12, 2022 at 9:04 PM Chamikara Jayalath 
wrote:

> Should we install a code commit freeze till this is addressed to maintain
> code health ? I noticed that it's easy to miss certain untriggered test
> suites for new PRs.
>
> Thanks,
> Cham
>
> On Sun, Jun 12, 2022 at 5:25 PM Danny McCormick 
> wrote:
>
>> In case anyone else is planning on taking a look at this, I did a deep
>> dive on it and wrote up my findings here -
>> https://docs.google.com/document/d/10qyUsvB_uVy5jftfTiwohlvN8Qwix5AuadssyoC4JsE/edit?usp=sharing
>>
>> My high level takeaways were:
>> 1. I don't think this is actually caused by Issues, I think it's a
>> structural issue with the plugin. At worst, Issues very minorly contributed
>> to it (with a smaller impact than adding an extra build trigger would/did),
>> but I'm not even sure that is true. As a result, our fix probably needs to
>> focus on patching or switching to a new plugin. 2. Patching the plugin
>> would be hard.
>> 3. I agree that switching to a different plugin is the best path forward.
>>
>> If anyone else is looking into this and wants edit permissions on the
>> doc, let me know!
>>
>> Thanks,
>> Danny
>>
>> On Fri, Jun 10, 2022 at 5:21 PM Kiley Sok  wrote:
>>
>>> There's currently an issue with too many connections from Jenkins to
>>> Github bringing down Jenkins. The hypothesis is that we were already close
>>> to the limit, but the extra triggers from GH issues pushed it over the edge.
>>>
>>> We're looking to switch to a different plugin that'll hopefully resolve
>>> the issue.
>>>
>>> More info: https://issues.apache.org/jira/browse/INFRA-23358 and in
>>> INFRA slack channel.
>>>
>>> Thanks for your patience!
>>>
>>