tvalentyn commented on code in PR #27723: URL: https://github.com/apache/beam/pull/27723#discussion_r1280933013
########## .github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml: ########## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Python ValidatesContainer Dataflow ARM + +on: + issue_comment: + types: [created] + push: + tags: ['v*'] + branches: ['master', 'release-*'] + paths: ["sdks/python/**",".github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml"] + schedule: + - cron: '0 */6 * * *' + workflow_dispatch: + +#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event +permissions: + actions: write + pull-requests: read + checks: read + contents: read + deployments: read + id-token: none + issues: read + discussions: read + packages: read + pages: read + repository-projects: read + security-events: read + statuses: read + +# This allows a subsequently queued workflow run to interrupt previous runs +concurrency: + group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}' + cancel-in-progress: true + +jobs: + beam_Python_ValidatesContainer_Dataflow_ARM: + name: beam_Python_ValidatesContainer_Dataflow_ARM + strategy: + fail-fast: false + matrix: + python_version: ['3.8','3.9','3.10','3.11'] + if: | + github.event_name == 'push' || + startsWith(github.event.comment.body, 'Run Python ValidatesContainer Dataflow ARM') || + github.event_name == 'schedule' + runs-on: [self-hosted, ubuntu-20.04, main] + steps: + - name: Check out repository code + uses: actions/checkout@v3 + with: + ref: ${{ github.event.pull_request.head.sha }} + - name: Set comment body with matrix + id: set_comment_body + run: | + echo "comment_body=Run Python ValidatesContainer Dataflow ARM (${{ matrix.python_version }})" >> $GITHUB_OUTPUT + - name: Rerun on comment + if: github.event.comment.body == steps.set_comment_body.outputs.comment_body + uses: ./.github/actions/rerun-job-action + with: + pull_request_url: ${{ github.event.issue.pull_request.url }} + github_repository: ${{ github.repository }} + github_token: ${{ secrets.GITHUB_TOKEN }} + github_job: "${{ github.job }} (${{ matrix.python_version }})" + github_current_run_id: ${{ github.run_id }} + - name: Install Python + uses: actions/setup-python@v4 + with: + python-version: ${{ matrix.python_version }} + - name: Setup Gradle + uses: gradle/gradle-build-action@v2 + with: + cache-read-only: false + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + - name: Authenticate on GCP + uses: google-github-actions/setup-gcloud@v0 + with: + service_account_email: ${{ secrets.GCP_SA_EMAIL }} + service_account_key: ${{ secrets.GCP_SA_KEY }} + project_id: ${{ secrets.GCP_PROJECT_ID }} + export_default_credentials: true + - name: GCloud Docker credential helper + run: | + gcloud auth configure-docker us-central1-docker.pkg.dev + - name: Set PY_VER_CLEAN + id: set_py_ver_clean + run: | + PY_VER=${{ matrix.python_version }} + PY_VER_CLEAN=${PY_VER//.} + echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT Review Comment: nit: debugging leftover? ########## sdks/python/test-suites/dataflow/common.gradle: ########## @@ -348,6 +348,21 @@ task validatesContainer() { } } +task validatesContainerARM() { + def pyversion = "${project.ext.pythonVersion.replace('.', '')}" + dependsOn 'initializeForDataflowJob' + dependsOn ":sdks:python:container:py${pyversion}:docker" + def runScriptsPath = "${rootDir}/sdks/python/container/run_validatescontainer.sh" Review Comment: as mentioned in the other comment, I think it would be better to configure the script to run in ARM mode around here. ########## .github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml: ########## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Python ValidatesContainer Dataflow ARM + +on: + issue_comment: + types: [created] + push: + tags: ['v*'] + branches: ['master', 'release-*'] + paths: ["sdks/python/**",".github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml"] + schedule: + - cron: '0 */6 * * *' + workflow_dispatch: + +#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event +permissions: + actions: write + pull-requests: read + checks: read + contents: read + deployments: read + id-token: none + issues: read + discussions: read + packages: read + pages: read + repository-projects: read + security-events: read + statuses: read + +# This allows a subsequently queued workflow run to interrupt previous runs +concurrency: + group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}' + cancel-in-progress: true + +jobs: + beam_Python_ValidatesContainer_Dataflow_ARM: + name: beam_Python_ValidatesContainer_Dataflow_ARM + strategy: + fail-fast: false + matrix: + python_version: ['3.8','3.9','3.10','3.11'] + if: | + github.event_name == 'push' || + startsWith(github.event.comment.body, 'Run Python ValidatesContainer Dataflow ARM') || + github.event_name == 'schedule' + runs-on: [self-hosted, ubuntu-20.04, main] + steps: + - name: Check out repository code + uses: actions/checkout@v3 + with: + ref: ${{ github.event.pull_request.head.sha }} + - name: Set comment body with matrix + id: set_comment_body + run: | + echo "comment_body=Run Python ValidatesContainer Dataflow ARM (${{ matrix.python_version }})" >> $GITHUB_OUTPUT + - name: Rerun on comment + if: github.event.comment.body == steps.set_comment_body.outputs.comment_body + uses: ./.github/actions/rerun-job-action + with: + pull_request_url: ${{ github.event.issue.pull_request.url }} + github_repository: ${{ github.repository }} + github_token: ${{ secrets.GITHUB_TOKEN }} + github_job: "${{ github.job }} (${{ matrix.python_version }})" + github_current_run_id: ${{ github.run_id }} + - name: Install Python + uses: actions/setup-python@v4 + with: + python-version: ${{ matrix.python_version }} + - name: Setup Gradle + uses: gradle/gradle-build-action@v2 + with: + cache-read-only: false + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + - name: Authenticate on GCP + uses: google-github-actions/setup-gcloud@v0 + with: + service_account_email: ${{ secrets.GCP_SA_EMAIL }} + service_account_key: ${{ secrets.GCP_SA_KEY }} + project_id: ${{ secrets.GCP_PROJECT_ID }} + export_default_credentials: true + - name: GCloud Docker credential helper + run: | + gcloud auth configure-docker us-central1-docker.pkg.dev + - name: Set PY_VER_CLEAN + id: set_py_ver_clean + run: | + PY_VER=${{ matrix.python_version }} + PY_VER_CLEAN=${PY_VER//.} + echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT + - name: Set current time as docker container image tag + id: set_tag + run: | + echo "buildtime=$(date +'%Y%m%d-%H%M%S%N')" >> $GITHUB_OUTPUT Review Comment: nit: debugging leftover? ########## sdks/python/container/run_validatescontainer.sh: ########## @@ -24,10 +24,13 @@ # REGION -> Region name to use for Dataflow # # Execute from the root of the repository: -# test Python3.8 container: +# test Python3.8 x86 container: # ./gradlew :sdks:python:test-suites:dataflow:py38:validatesContainer -# or test all supported python versions together: +# or test all supported python versions x86 containers together: # ./gradlew :sdks:python:test-suites:dataflow:validatesContainer +# +# Note: ARM test suites only run on github actions. For example, to test Python3.8 ARM containers, Review Comment: I would make platform an argument to this script, or at least set the environment variable in gradle, instead of in GH actions. That would make it easier to trigger these tests locally, and would make it easier for us to pivot to test on Jenkins if we don't sort out pushing to GCR from GH actions in time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
