Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Damon Douglas]

Adding more helpful information about Cloud Build.  With it's additional
features, we can:

1. Define GitHub triggers on certain branches and optionally multiple
wildcarded includes and exclude paths.
2. Flag whether to trigger on pull requests or specific branch commits
3. Version control the terraform code that provisions the triggers;
typically applied once per GCP project setup
4. See on GitHub progress as the trigger executes and success/failure

On Fri, Nov 18, 2022, 4:20 PM Damon Douglas  wrote:

> I wonder if some integration tests could be offloaded to their respective
> cloud provider. For example, the Google cloud related integration tests
> could be executed on Cloud build.  Cloud Build's service account, or
> custom, could have the minimally necessary IAM roles to access Google Cloud
> resources as part of its execution.  The 'build' in its name shouldn't
> mislead one to think it's only for building.  It's essentially just a
> container that when connected to the GitHub repository its triggers can
> report back the success or failure of a run.  As a bonus: no need for
> static service account keys 朗朗朗
>
> For those reading this and do not know what Cloud Build is, please see
> https://youtu.be/Bvo6jzC3J_A
>
> For information about service accounts, please see
> https://youtu.be/xXk1YlkKW_k
>
> For information about service account keys, please see
> https://youtu.be/SDhMwyyd9_0
>
> And finally, IAM permissions, please see https://youtu.be/Sdt-i-Q7tyA
>
> On Wed, Oct 19, 2022, 8:32 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> As you probably noticed, there's a lot of work going on around adding
>> more GitHub Actions workflows.
>>
>> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
>> migration (but less user-facing) it would bring us on to "default"
>> infrastructure that more people understand and is maintained by GitHub.
>>
>> So far we have hit some serious roadblocks. It isn't just a simple
>> migration. We have to weigh doing the work to get there.
>>
>> I started a document with a table of the things we get from Jenkins that
>> we need to be sure to have for GitHub Actions before we could think about
>> migrating:
>>
>> https://s.apache.org/beam-jenkins-to-gha
>>
>> Can you please help me by adding things that we get from Jenkins, and if
>> you know how to get them from GitHub Actions add that too.
>>
>> Thanks!
>>
>> Kenn
>>
>

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Damon Douglas]

I wonder if some integration tests could be offloaded to their respective
cloud provider. For example, the Google cloud related integration tests
could be executed on Cloud build.  Cloud Build's service account, or
custom, could have the minimally necessary IAM roles to access Google Cloud
resources as part of its execution.  The 'build' in its name shouldn't
mislead one to think it's only for building.  It's essentially just a
container that when connected to the GitHub repository its triggers can
report back the success or failure of a run.  As a bonus: no need for
static service account keys 朗朗朗

For those reading this and do not know what Cloud Build is, please see
https://youtu.be/Bvo6jzC3J_A

For information about service accounts, please see
https://youtu.be/xXk1YlkKW_k

For information about service account keys, please see
https://youtu.be/SDhMwyyd9_0

And finally, IAM permissions, please see https://youtu.be/Sdt-i-Q7tyA

On Wed, Oct 19, 2022, 8:32 AM Kenneth Knowles  wrote:

> Hi all,
>
> As you probably noticed, there's a lot of work going on around adding more
> GitHub Actions workflows.
>
> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
> migration (but less user-facing) it would bring us on to "default"
> infrastructure that more people understand and is maintained by GitHub.
>
> So far we have hit some serious roadblocks. It isn't just a simple
> migration. We have to weigh doing the work to get there.
>
> I started a document with a table of the things we get from Jenkins that
> we need to be sure to have for GitHub Actions before we could think about
> migrating:
>
> https://s.apache.org/beam-jenkins-to-gha
>
> Can you please help me by adding things that we get from Jenkins, and if
> you know how to get them from GitHub Actions add that too.
>
> Thanks!
>
> Kenn
>

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Tomo Suzuki via dev]

Kenn, thank you for the summary. SGTM. Looking forward to GitHub Actions.

On Mon, Nov 7, 2022 at 12:58 PM Kenneth Knowles  wrote:

> OK, it seems like there is general consensus. Not too much action on the
> document. I will summarize the gaps that don't have an answer in the doc,
> and my new opinion of how important they are:
>
>  - [required] Run specific non-default workflow on PR
>  - [required] View history of a workflow
>  - [required] Publish nightly snapshots
>  - [required] Run workflow on dedicated worker pool for performance testing
>  - [important but not required] Summarize flakiness statistics of one or
> all workflows
>  - [important but not required] History of all/many workflows in a single
> view
>  - [nice to have] History of specific test case (not just the workflow
> level)
>
> Do any of these seem like I got the importance wrong?
>
> Kenn
>
> On Mon, Nov 7, 2022 at 9:09 AM Austin Bennett  wrote:
>
>> +1
>>
>> Also would help address a good amount of what concerns me that was
>> [sorta] raised by
>> https://lists.apache.org/thread/7jr99nc5xsb3ft1d75kb0ml32bzw89rv
>>
>>
>> Once we think this is something we want to do, but might be
>> blocked/concerned because of lack of definitively comparable features, I'd
>> be happy to take a look at what exists in the wider ecosystem or could be
>> built.
>>
>> Cheers -
>>
>>
>>
>> On Fri, Oct 21, 2022 at 11:10 AM Ismaël Mejía  wrote:
>>
>>> +1 Github Actions are more intuitive and easy to modify and test for
>>> everyone.
>>> Also Beam wins because that makes one less system to maintain.
>>>
>>> Regards,
>>> Ismaël
>>>
>>> On Wed, Oct 19, 2022 at 5:50 PM Danny McCormick via dev
>>>  wrote:
>>> >
>>> > Thanks for kicking this conversation off. I'm +1 on migrating, but
>>> only once we've found a specific replacement for easy observability (which
>>> workflows have been failing lately, and how often) and trigger phrases (for
>>> retries and workflows that aren't automatically kicked off but should be
>>> run for extra validation, e.g. postcommits). Until we have viable
>>> replacements, I don't think we should make the move. Publishing nightly
>>> snapshots is eventually also a must to fully migrate, but probably doesn't
>>> need to block us from making progress here.
>>> >
>>> > With those caveats, the reason that I'm +1 on moving is that our
>>> Jenkins reliability has been rough. Since I joined the project in January,
>>> I can think of 3 different incidents that significantly harmed our ability
>>> to do work.
>>> >
>>> > 1. Jenkins triggers cause multi-day outage - this led to a multi-day
>>> code freeze, and we lost our trigger functionality for days afterwards.
>>> Investigating/restoring our state ate up a pretty full week for me.
>>> > 2. Jenkins plugin cause multi-day outage - this led to multiple days
>>> of Jenkins downtime before eventually being resolved by Infra.
>>> > 3. Cert issues cause many workers to go down - I don't have a thread
>>> for this because I handled most of the investigation the day of, but many
>>> of our workers went down for around a day and nobody noticed until queue
>>> time reached 6+ hours for each workflow.
>>> >
>>> > There may be others that I'm overlooking.
>>> >
>>> > GitHub Actions isn't a magic bullet to fix these problems, but it
>>> minimizes the amount of infra that we're maintaining ourselves, increases
>>> the isolation between workflows (catastrophic failure is less likely), has
>>> uptime guarantees, and is more likely to receive investment going forward
>>> (we're likely to get increasing benefits over time for free). We've also
>>> done a lot of exploration in this area already, so we're not starting from
>>> scratch.
>>> >
>>> > Thanks,
>>> > Danny
>>> >
>>> > On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles 
>>> wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> As you probably noticed, there's a lot of work going on around adding
>>> more GitHub Actions workflows.
>>> >>
>>> >> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
>>> migration (but less user-facing) it would bring us on to "default"
>>> infrastructure that more people understand and is maintained by GitHub.
>>> >>
>>> >> So far we have hit some serious roadblocks. It isn't just a simple
>>> migration. We have to weigh doing the work to get there.
>>> >>
>>> >> I started a document with a table of the things we get from Jenkins
>>> that we need to be sure to have for GitHub Actions before we could think
>>> about migrating:
>>> >>
>>> >> https://s.apache.org/beam-jenkins-to-gha
>>> >>
>>> >> Can you please help me by adding things that we get from Jenkins, and
>>> if you know how to get them from GitHub Actions add that too.
>>> >>
>>> >> Thanks!
>>> >>
>>> >> Kenn
>>>
>>

-- 
Regards,
Tomo

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Kenneth Knowles]

OK, it seems like there is general consensus. Not too much action on the
document. I will summarize the gaps that don't have an answer in the doc,
and my new opinion of how important they are:

 - [required] Run specific non-default workflow on PR
 - [required] View history of a workflow
 - [required] Publish nightly snapshots
 - [required] Run workflow on dedicated worker pool for performance testing
 - [important but not required] Summarize flakiness statistics of one or
all workflows
 - [important but not required] History of all/many workflows in a single
view
 - [nice to have] History of specific test case (not just the workflow
level)

Do any of these seem like I got the importance wrong?

Kenn

On Mon, Nov 7, 2022 at 9:09 AM Austin Bennett  wrote:

> +1
>
> Also would help address a good amount of what concerns me that was [sorta]
> raised by https://lists.apache.org/thread/7jr99nc5xsb3ft1d75kb0ml32bzw89rv
>
>
>
> Once we think this is something we want to do, but might be
> blocked/concerned because of lack of definitively comparable features, I'd
> be happy to take a look at what exists in the wider ecosystem or could be
> built.
>
> Cheers -
>
>
>
> On Fri, Oct 21, 2022 at 11:10 AM Ismaël Mejía  wrote:
>
>> +1 Github Actions are more intuitive and easy to modify and test for
>> everyone.
>> Also Beam wins because that makes one less system to maintain.
>>
>> Regards,
>> Ismaël
>>
>> On Wed, Oct 19, 2022 at 5:50 PM Danny McCormick via dev
>>  wrote:
>> >
>> > Thanks for kicking this conversation off. I'm +1 on migrating, but only
>> once we've found a specific replacement for easy observability (which
>> workflows have been failing lately, and how often) and trigger phrases (for
>> retries and workflows that aren't automatically kicked off but should be
>> run for extra validation, e.g. postcommits). Until we have viable
>> replacements, I don't think we should make the move. Publishing nightly
>> snapshots is eventually also a must to fully migrate, but probably doesn't
>> need to block us from making progress here.
>> >
>> > With those caveats, the reason that I'm +1 on moving is that our
>> Jenkins reliability has been rough. Since I joined the project in January,
>> I can think of 3 different incidents that significantly harmed our ability
>> to do work.
>> >
>> > 1. Jenkins triggers cause multi-day outage - this led to a multi-day
>> code freeze, and we lost our trigger functionality for days afterwards.
>> Investigating/restoring our state ate up a pretty full week for me.
>> > 2. Jenkins plugin cause multi-day outage - this led to multiple days of
>> Jenkins downtime before eventually being resolved by Infra.
>> > 3. Cert issues cause many workers to go down - I don't have a thread
>> for this because I handled most of the investigation the day of, but many
>> of our workers went down for around a day and nobody noticed until queue
>> time reached 6+ hours for each workflow.
>> >
>> > There may be others that I'm overlooking.
>> >
>> > GitHub Actions isn't a magic bullet to fix these problems, but it
>> minimizes the amount of infra that we're maintaining ourselves, increases
>> the isolation between workflows (catastrophic failure is less likely), has
>> uptime guarantees, and is more likely to receive investment going forward
>> (we're likely to get increasing benefits over time for free). We've also
>> done a lot of exploration in this area already, so we're not starting from
>> scratch.
>> >
>> > Thanks,
>> > Danny
>> >
>> > On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles 
>> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> As you probably noticed, there's a lot of work going on around adding
>> more GitHub Actions workflows.
>> >>
>> >> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
>> migration (but less user-facing) it would bring us on to "default"
>> infrastructure that more people understand and is maintained by GitHub.
>> >>
>> >> So far we have hit some serious roadblocks. It isn't just a simple
>> migration. We have to weigh doing the work to get there.
>> >>
>> >> I started a document with a table of the things we get from Jenkins
>> that we need to be sure to have for GitHub Actions before we could think
>> about migrating:
>> >>
>> >> https://s.apache.org/beam-jenkins-to-gha
>> >>
>> >> Can you please help me by adding things that we get from Jenkins, and
>> if you know how to get them from GitHub Actions add that too.
>> >>
>> >> Thanks!
>> >>
>> >> Kenn
>>
>

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Austin Bennett]

+1

Also would help address a good amount of what concerns me that was [sorta]
raised by https://lists.apache.org/thread/7jr99nc5xsb3ft1d75kb0ml32bzw89rv


Once we think this is something we want to do, but might be
blocked/concerned because of lack of definitively comparable features, I'd
be happy to take a look at what exists in the wider ecosystem or could be
built.

Cheers -



On Fri, Oct 21, 2022 at 11:10 AM Ismaël Mejía  wrote:

> +1 Github Actions are more intuitive and easy to modify and test for
> everyone.
> Also Beam wins because that makes one less system to maintain.
>
> Regards,
> Ismaël
>
> On Wed, Oct 19, 2022 at 5:50 PM Danny McCormick via dev
>  wrote:
> >
> > Thanks for kicking this conversation off. I'm +1 on migrating, but only
> once we've found a specific replacement for easy observability (which
> workflows have been failing lately, and how often) and trigger phrases (for
> retries and workflows that aren't automatically kicked off but should be
> run for extra validation, e.g. postcommits). Until we have viable
> replacements, I don't think we should make the move. Publishing nightly
> snapshots is eventually also a must to fully migrate, but probably doesn't
> need to block us from making progress here.
> >
> > With those caveats, the reason that I'm +1 on moving is that our Jenkins
> reliability has been rough. Since I joined the project in January, I can
> think of 3 different incidents that significantly harmed our ability to do
> work.
> >
> > 1. Jenkins triggers cause multi-day outage - this led to a multi-day
> code freeze, and we lost our trigger functionality for days afterwards.
> Investigating/restoring our state ate up a pretty full week for me.
> > 2. Jenkins plugin cause multi-day outage - this led to multiple days of
> Jenkins downtime before eventually being resolved by Infra.
> > 3. Cert issues cause many workers to go down - I don't have a thread for
> this because I handled most of the investigation the day of, but many of
> our workers went down for around a day and nobody noticed until queue time
> reached 6+ hours for each workflow.
> >
> > There may be others that I'm overlooking.
> >
> > GitHub Actions isn't a magic bullet to fix these problems, but it
> minimizes the amount of infra that we're maintaining ourselves, increases
> the isolation between workflows (catastrophic failure is less likely), has
> uptime guarantees, and is more likely to receive investment going forward
> (we're likely to get increasing benefits over time for free). We've also
> done a lot of exploration in this area already, so we're not starting from
> scratch.
> >
> > Thanks,
> > Danny
> >
> > On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles 
> wrote:
> >>
> >> Hi all,
> >>
> >> As you probably noticed, there's a lot of work going on around adding
> more GitHub Actions workflows.
> >>
> >> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
> migration (but less user-facing) it would bring us on to "default"
> infrastructure that more people understand and is maintained by GitHub.
> >>
> >> So far we have hit some serious roadblocks. It isn't just a simple
> migration. We have to weigh doing the work to get there.
> >>
> >> I started a document with a table of the things we get from Jenkins
> that we need to be sure to have for GitHub Actions before we could think
> about migrating:
> >>
> >> https://s.apache.org/beam-jenkins-to-gha
> >>
> >> Can you please help me by adding things that we get from Jenkins, and
> if you know how to get them from GitHub Actions add that too.
> >>
> >> Thanks!
> >>
> >> Kenn
>

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Ismaël Mejía]

+1 Github Actions are more intuitive and easy to modify and test for everyone.
Also Beam wins because that makes one less system to maintain.

Regards,
Ismaël

On Wed, Oct 19, 2022 at 5:50 PM Danny McCormick via dev
 wrote:
>
> Thanks for kicking this conversation off. I'm +1 on migrating, but only once 
> we've found a specific replacement for easy observability (which workflows 
> have been failing lately, and how often) and trigger phrases (for retries and 
> workflows that aren't automatically kicked off but should be run for extra 
> validation, e.g. postcommits). Until we have viable replacements, I don't 
> think we should make the move. Publishing nightly snapshots is eventually 
> also a must to fully migrate, but probably doesn't need to block us from 
> making progress here.
>
> With those caveats, the reason that I'm +1 on moving is that our Jenkins 
> reliability has been rough. Since I joined the project in January, I can 
> think of 3 different incidents that significantly harmed our ability to do 
> work.
>
> 1. Jenkins triggers cause multi-day outage - this led to a multi-day code 
> freeze, and we lost our trigger functionality for days afterwards. 
> Investigating/restoring our state ate up a pretty full week for me.
> 2. Jenkins plugin cause multi-day outage - this led to multiple days of 
> Jenkins downtime before eventually being resolved by Infra.
> 3. Cert issues cause many workers to go down - I don't have a thread for this 
> because I handled most of the investigation the day of, but many of our 
> workers went down for around a day and nobody noticed until queue time 
> reached 6+ hours for each workflow.
>
> There may be others that I'm overlooking.
>
> GitHub Actions isn't a magic bullet to fix these problems, but it minimizes 
> the amount of infra that we're maintaining ourselves, increases the isolation 
> between workflows (catastrophic failure is less likely), has uptime 
> guarantees, and is more likely to receive investment going forward (we're 
> likely to get increasing benefits over time for free). We've also done a lot 
> of exploration in this area already, so we're not starting from scratch.
>
> Thanks,
> Danny
>
> On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles  wrote:
>>
>> Hi all,
>>
>> As you probably noticed, there's a lot of work going on around adding more 
>> GitHub Actions workflows.
>>
>> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues 
>> migration (but less user-facing) it would bring us on to "default" 
>> infrastructure that more people understand and is maintained by GitHub.
>>
>> So far we have hit some serious roadblocks. It isn't just a simple 
>> migration. We have to weigh doing the work to get there.
>>
>> I started a document with a table of the things we get from Jenkins that we 
>> need to be sure to have for GitHub Actions before we could think about 
>> migrating:
>>
>> https://s.apache.org/beam-jenkins-to-gha
>>
>> Can you please help me by adding things that we get from Jenkins, and if you 
>> know how to get them from GitHub Actions add that too.
>>
>> Thanks!
>>
>> Kenn

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Danny McCormick via dev]

Thanks for kicking this conversation off. I'm +1 on migrating, but only
once we've found a specific replacement for easy observability (which
workflows have been failing lately, and how often) and trigger phrases (for
retries and workflows that aren't automatically kicked off but should be
run for extra validation, e.g. postcommits). Until we have viable
replacements, I don't think we should make the move. Publishing nightly
snapshots is eventually also a must to fully migrate, but probably doesn't
need to block us from making progress here.

With those caveats, the reason that I'm +1 on moving is that our Jenkins
reliability has been rough. Since I joined the project in January, I can
think of 3 different incidents that significantly harmed our ability to do
work.

1. Jenkins triggers cause multi-day outage
 - this
led to a multi-day code freeze, and we lost our trigger functionality for
days afterwards. Investigating/restoring our state ate up a pretty full
week for me.
2. Jenkins plugin cause multi-day outage
 - this led to multiple
days of Jenkins downtime before eventually being resolved by Infra.
3. Cert issues cause many workers to go down - I don't have a thread for
this because I handled most of the investigation the day of, but many of
our workers went down for around a day and nobody noticed until queue time
reached 6+ hours for each workflow.

There may be others that I'm overlooking.

GitHub Actions isn't a magic bullet to fix these problems, but it minimizes
the amount of infra that we're maintaining ourselves, increases the
isolation between workflows (catastrophic failure is less likely), has
uptime guarantees, and is more likely to receive investment going forward
(we're likely to get increasing benefits over time for free). We've also
done a lot of exploration in this area already, so we're not starting from
scratch.

Thanks,
Danny

On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles  wrote:

> Hi all,
>
> As you probably noticed, there's a lot of work going on around adding more
> GitHub Actions workflows.
>
> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
> migration (but less user-facing) it would bring us on to "default"
> infrastructure that more people understand and is maintained by GitHub.
>
> So far we have hit some serious roadblocks. It isn't just a simple
> migration. We have to weigh doing the work to get there.
>
> I started a document with a table of the things we get from Jenkins that
> we need to be sure to have for GitHub Actions before we could think about
> migrating:
>
> https://s.apache.org/beam-jenkins-to-gha
>
> Can you please help me by adding things that we get from Jenkins, and if
> you know how to get them from GitHub Actions add that too.
>
> Thanks!
>
> Kenn
>

[DISCUSS] Jenkins -> GitHub Actions ?

[Kenneth Knowles]

Hi all,

As you probably noticed, there's a lot of work going on around adding more
GitHub Actions workflows.

Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
migration (but less user-facing) it would bring us on to "default"
infrastructure that more people understand and is maintained by GitHub.

So far we have hit some serious roadblocks. It isn't just a simple
migration. We have to weigh doing the work to get there.

I started a document with a table of the things we get from Jenkins that we
need to be sure to have for GitHub Actions before we could think about
migrating:

https://s.apache.org/beam-jenkins-to-gha

Can you please help me by adding things that we get from Jenkins, and if
you know how to get them from GitHub Actions add that too.

Thanks!

Kenn