Hi Jarek! Thanks a lot for the insight and resources you shared with us. I will add that to the document I'm drafting containing the changes needed for this effort to work.
Pablo, I will go through these documents today, and let's say we have an initial call on Wednesday? Sharing the call info here in case someone else would like to join. Wednesday, August 11 · 2:00 – 3:00pm EST Google Meet joining info Video call link: https://meet.google.com/xsd-azwb-rxz More phone numbers: https://tel.meet/xsd-azwb-rxz?pin=7716629663982 Thanks again, Pablo and Jarek! - Fer On Thu, Aug 5, 2021 at 11:37 AM Jarek Potiuk <[email protected]> wrote: > I'd love to help, but I am on vacation next week, just one word of > warning. If you want to run GitHub Runner on your own infrastructure, that > might introduce several security risks. > > Basically anyone who makes a PR to your repo can compromise your runners. > The dangers of compromising runners are explained here: > https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#potential-impact-of-a-compromised-runner > And Github still very very strongly discourages people to use self-hosted > runners for Public repositories: > > > https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#hardening-for-self-hosted-runners > > Self-hosted runners on GitHub do not have guarantees around running in > ephemeral clean virtual machines, and can be persistently compromised by > untrusted code in a workflow. > As a result, self-hosted runners should almost never be used for public > repositories on GitHub, because any user can open pull requests against the > repository and compromise the environment. > > We've workarounded it in Airflow and we run our self-hosted runners on > Amazon, however we have a patched version of runner: > https://github.com/ashb/runner. We needed to patch it as the original > runner cannot reject builds started from "unknown" users. > Which is automatically rebased and patch re-applied whenever GitHub > releases their new runner (the old versions stop working ~ few days after > new version is released). We keep a list of allowed people ("committers") > who can utilise such runners. > We also maintain list of maintainers in our workflow that are "routed" to > self-hosted runners: > https://github.com/apache/airflow/blob/1bd3a5c68c88cf3840073d6276460a108f864187/.github/workflows/ci.yml#L86 > > Those are a bit of hacks (until GitHub implements proper support for it), > but if you want to avoid people using your runners to mine bitcoin, or > stealing secrets or potentially being able to modify your repository > without you knowing it, similar steps should be taken. > > J. > > > > > > On Thu, Aug 5, 2021 at 7:17 PM Pablo Estrada <[email protected]> wrote: > >> That works for me! Maybe create a calendar invite and share it in this >> thread. >> Best >> -P. >> >> On Thu, Aug 5, 2021, 8:36 AM Fernando Morales Martinez < >> [email protected]> wrote: >> >>> Thanks for the help, Pablo! >>> I'm also available most of Monday, Tuesday and Wednesday; how about we >>> set the meeting for Tuesday August 10th 2pm EST? In case someone interested >>> can't make it, we can adjust accordingly. >>> Thanks again! >>> -Fer >>> >>> On Wed, Aug 4, 2021 at 5:55 PM Pablo Estrada <[email protected]> wrote: >>> >>>> Hello Fernando! >>>> The people that built the GKE cluster infrastructure have not been >>>> involved with Beam for a while. >>>> I think you can set a time that is convenient for you, and invite >>>> others to participate - and we'll all figure it out together. >>>> I'm available most of Monday, Tuesday, Wednesday of next week. I'll be >>>> happy to jump on a call and be confused with you (and maybe others will >>>> too). >>>> Best >>>> -P. >>>> >>>> On Tue, Aug 3, 2021 at 11:20 AM Fernando Morales Martinez < >>>> [email protected]> wrote: >>>> >>>>> Hi everyone, >>>>> As part of the work done to migrate the GitHub Actions runners over to >>>>> GKE, the, not exhaustive, changes below were performed: >>>>> >>>>> - added a new secret to the apache-beam-testing project (this is the >>>>> PAT needed by the docker image to connect to GitHub) >>>>> - added a new docker image (which will execute the test flows still >>>>> being ran by GitHub) >>>>> >>>>> and other changes which you can take a look at here: >>>>> https://github.com/apache/beam/pull/15039/files >>>>> Among those, the ones of notice are the following: >>>>> - .github/workflows/build_wheels.yml >>>>> - .github/workflows/cancel.yml >>>>> - .github/workflows/java_tests.yml >>>>> - .github/workflows/python_tests.yml >>>>> - .test-infra/metrics/build.gradle >>>>> - .test-infra/metrics/build_and_publish_containers.sh >>>>> - .test-infra/metrics/docker-compose.yml >>>>> - .test-infra/metrics/kubernetes/beamgrafana-deploy.yaml >>>>> - .test-infra/metrics/sync/githubactions/Dockerfile >>>>> - .test-infra/metrics/sync/githubactions/entrypoint.sh >>>>> >>>>> The docker image (.test-infra/metrics/sync/githubactions/Dockerfile) >>>>> has been built and pushed to the GCP container registry already. >>>>> >>>>> Now, I need to deploy the new docker image to GKE. I've done that >>>>> before, but in a testing cluster/namespace. I'd definitely feel more >>>>> comfortable if someone knowledgeable with the Apache Beam Testing project >>>>> GKE architecture watches over my shoulder. >>>>> I think I have everything covered, but I could be missing some >>>>> important piece. >>>>> >>>>> Thanks for the help! >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *This email and its contents (including any attachments) are being >>>>> sent toyou on the condition of confidentiality and may be protected by >>>>> legalprivilege. Access to this email by anyone other than the intended >>>>> recipientis unauthorized. If you are not the intended recipient, please >>>>> immediatelynotify the sender by replying to this message and delete the >>>>> materialimmediately from your system. Any further use, dissemination, >>>>> distributionor reproduction of this email is strictly prohibited. Further, >>>>> norepresentation is made with respect to any content contained in this >>>>> email.* >>>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> *This email and its contents (including any attachments) are being sent >>> toyou on the condition of confidentiality and may be protected by >>> legalprivilege. Access to this email by anyone other than the intended >>> recipientis unauthorized. If you are not the intended recipient, please >>> immediatelynotify the sender by replying to this message and delete the >>> materialimmediately from your system. Any further use, dissemination, >>> distributionor reproduction of this email is strictly prohibited. Further, >>> norepresentation is made with respect to any content contained in this >>> email.* >> >> > > -- > +48 660 796 129 > -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
