[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r443506463 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 +steps: + - name: Checkout code +uses: actions/checkout@v2 + - name: Install python +uses: actions/setup-python@v2 +with: + python-version: 3.7 + - name: Get build dependencies +working-directory: ./sdks/python +run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt + - name: Install wheels +run: python3 -m pip install wheel + - name: Buld source +working-directory: ./sdks/python +run: python3 setup.py sdist --formats=gztar,zip + - name: Unzip source +working-directory: ./sdks/python +run: unzip dist/$(ls dist | grep .zip | head -n 1) + - name: Rename source directory +working-directory: ./sdks/python +run: mv $(ls | grep apache-beam) apache-beam-source + - name: Upload source +uses: actions/upload-artifact@v2 +with: + name: source + path: sdks/python/apache-beam-source + - name: Upload compressed sources +uses: actions/upload-artifact@v2 +with: + name: source_gztar_zip Review comment: >I think it would be much cleaner and explicit. Did I understand correctly your idea? Yes. Your understanding is correct. This sounds good to me. I renamed steps. Please take a look is there no mistakes. > Clean up: Both options are good. Can we do both? Sure. We can do both. Related to versioning and lifecycle management: I added information in the PR description. For periodic cleaning tasks I created separate dependent draft PR: https://github.com/apache/beam/pull/12049 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436068238
##
File path: .github/workflows/build_wheels.yml
##
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+branches:
+ - master
+ - release-*
+tags:
+ - v*
+
+jobs:
+
+ build_source:
+runs-on: ubuntu-18.04
+steps:
+ - name: Checkout code
+uses: actions/checkout@v2
+ - name: Install python
+uses: actions/setup-python@v2
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+run: python3 -m pip install wheel
+ - name: Buld source
+working-directory: ./sdks/python
+run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+working-directory: ./sdks/python
+run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+working-directory: ./sdks/python
+run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+uses: actions/upload-artifact@v2
+with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+uses: actions/upload-artifact@v2
+with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+name: Prepare GCS
+needs: build_source
+runs-on: ubuntu-18.04
+steps:
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/"
|| true
+
+ upload_source_to_gcs:
+name: Upload source to GCS bucket
+needs: prepare_gcs
+runs-on: ubuntu-18.04
+steps:
+ - name: Download wheels
+uses: actions/download-artifact@v2
+with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET
}}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+name: Build wheels on ${{ matrix.os }}
+needs: prepare_gcs
+runs-on: ${{ matrix.os }}
+strategy:
+ matrix:
+os : [ubuntu-18.04, macos-10.15]
Review comment:
Hopefully yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436068604
##
File path: .github/workflows/build_wheels.yml
##
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+branches:
+ - master
+ - release-*
+tags:
+ - v*
+
+jobs:
+
+ build_source:
+runs-on: ubuntu-18.04
+steps:
+ - name: Checkout code
+uses: actions/checkout@v2
+ - name: Install python
+uses: actions/setup-python@v2
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+run: python3 -m pip install wheel
+ - name: Buld source
+working-directory: ./sdks/python
+run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+working-directory: ./sdks/python
+run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+working-directory: ./sdks/python
+run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+uses: actions/upload-artifact@v2
+with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+uses: actions/upload-artifact@v2
+with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+name: Prepare GCS
+needs: build_source
+runs-on: ubuntu-18.04
+steps:
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/"
|| true
+
+ upload_source_to_gcs:
+name: Upload source to GCS bucket
+needs: prepare_gcs
+runs-on: ubuntu-18.04
+steps:
+ - name: Download wheels
+uses: actions/download-artifact@v2
+with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET
}}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+name: Build wheels on ${{ matrix.os }}
+needs: prepare_gcs
+runs-on: ${{ matrix.os }}
+strategy:
+ matrix:
+os : [ubuntu-18.04, macos-10.15]
+steps:
+- name: Download source
+ uses: actions/download-artifact@v2
+ with:
+name: source
+path: apache-beam-source
+- name: Install Python
+ uses: actions/setup-python@v2
+ with:
+python-version: 3.7
+- name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+brew update
+brew install pkg-config
+- name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+- name: Build wheel
+ working-directory: apache-beam-source
+ env:
+CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
+CIBW_BUILD_VERBOSITY: 3
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r436068410 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 +steps: + - name: Checkout code +uses: actions/checkout@v2 + - name: Install python +uses: actions/setup-python@v2 +with: + python-version: 3.7 + - name: Get build dependencies +working-directory: ./sdks/python +run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt + - name: Install wheels +run: python3 -m pip install wheel + - name: Buld source Review comment: Thanks!. Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r436067638 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 +steps: + - name: Checkout code +uses: actions/checkout@v2 + - name: Install python +uses: actions/setup-python@v2 +with: + python-version: 3.7 + - name: Get build dependencies +working-directory: ./sdks/python +run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt + - name: Install wheels +run: python3 -m pip install wheel + - name: Buld source +working-directory: ./sdks/python +run: python3 setup.py sdist --formats=gztar,zip + - name: Unzip source +working-directory: ./sdks/python +run: unzip dist/$(ls dist | grep .zip | head -n 1) + - name: Rename source directory +working-directory: ./sdks/python +run: mv $(ls | grep apache-beam) apache-beam-source + - name: Upload source +uses: actions/upload-artifact@v2 +with: + name: source + path: sdks/python/apache-beam-source + - name: Upload compressed sources +uses: actions/upload-artifact@v2 +with: + name: source_gztar_zip Review comment: 1. Currently two steps: `List sources on GCS bucket` and `Copy wheels to GCS bucket` are listing files of specific types. Instead of this two separate steps I could create job which will list all files in specific gcs folder. I think it would be much cleaner and explicit. Did I understand correctly your idea? About cleaning up these GCS locations I consider two options: - setting lifecycle management on the bucket which will delete files older than some arbitrary age, e.g. 365 days. I think advantage of this is that will be maintenance free. - creating another scheduled workflow on github actions which will delete gcs folders if corresponding branch does not exist anymore. Could be scheduled to run e.g. once pre week. Which option has more sense for you? 2. "Upload" steps perform file upload as artifacts so they could be passed between jobs and being available for download for 90 days (if not deleted earlier). These artifacts are picked up later by "Upload to GCS" jobs. What do you think about renaming these steps e.g.: "Upload wheels" -> "Upload wheels as artifacts" ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r435407993 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 +steps: + - name: Checkout code +uses: actions/checkout@v2 + - name: Install python +uses: actions/setup-python@v2 +with: + python-version: 3.7 + - name: Get build dependencies +working-directory: ./sdks/python +run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt Review comment: Good point I've tested and for `build_source` job only `python3 -m pip install -r build-requirements.txt` is required, then `python -m pip install cython` is executed in `build_wheels`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435404202
##
File path: .github/workflows/build_wheels.yml
##
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+branches:
+ - master
+ - release-*
+tags:
+ - v*
+
+jobs:
+
+ build_source:
+runs-on: ubuntu-18.04
+steps:
+ - name: Checkout code
+uses: actions/checkout@v2
+ - name: Install python
+uses: actions/setup-python@v2
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+run: python3 -m pip install wheel
+ - name: Buld source
+working-directory: ./sdks/python
+run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+working-directory: ./sdks/python
+run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+working-directory: ./sdks/python
+run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+uses: actions/upload-artifact@v2
+with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+uses: actions/upload-artifact@v2
+with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+name: Prepare GCS
+needs: build_source
+runs-on: ubuntu-18.04
+steps:
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/"
|| true
+
+ upload_source_to_gcs:
+name: Upload source to GCS bucket
+needs: prepare_gcs
+runs-on: ubuntu-18.04
+steps:
+ - name: Download wheels
+uses: actions/download-artifact@v2
+with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET
}}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+name: Build wheels on ${{ matrix.os }}
+needs: prepare_gcs
+runs-on: ${{ matrix.os }}
+strategy:
+ matrix:
+os : [ubuntu-18.04, macos-10.15]
+steps:
+- name: Download source
+ uses: actions/download-artifact@v2
+ with:
+name: source
+path: apache-beam-source
+- name: Install Python
+ uses: actions/setup-python@v2
+ with:
+python-version: 3.7
+- name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+brew update
+brew install pkg-config
+- name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+- name: Build wheel
+ working-directory: apache-beam-source
+ env:
+CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
Review comment:
Sure, I will add it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435403856
##
File path: .github/workflows/build_wheels.yml
##
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+branches:
+ - master
+ - release-*
+tags:
+ - v*
+
+jobs:
+
+ build_source:
+runs-on: ubuntu-18.04
+steps:
+ - name: Checkout code
+uses: actions/checkout@v2
+ - name: Install python
+uses: actions/setup-python@v2
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+run: python3 -m pip install wheel
+ - name: Buld source
+working-directory: ./sdks/python
+run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+working-directory: ./sdks/python
+run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+working-directory: ./sdks/python
+run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+uses: actions/upload-artifact@v2
+with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+uses: actions/upload-artifact@v2
+with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+name: Prepare GCS
+needs: build_source
+runs-on: ubuntu-18.04
+steps:
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/"
|| true
+
+ upload_source_to_gcs:
+name: Upload source to GCS bucket
+needs: prepare_gcs
+runs-on: ubuntu-18.04
+steps:
+ - name: Download wheels
+uses: actions/download-artifact@v2
+with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET
}}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+name: Build wheels on ${{ matrix.os }}
+needs: prepare_gcs
+runs-on: ${{ matrix.os }}
+strategy:
+ matrix:
+os : [ubuntu-18.04, macos-10.15]
+steps:
+- name: Download source
+ uses: actions/download-artifact@v2
+ with:
+name: source
+path: apache-beam-source
+- name: Install Python
+ uses: actions/setup-python@v2
+ with:
+python-version: 3.7
+- name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+brew update
+brew install pkg-config
+- name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+- name: Build wheel
+ working-directory: apache-beam-source
+ env:
+CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
+CIBW_BUILD_VERBOSITY: 3
Review comment:
Good point, we don't need it I think. I will remove this line so the
default verbosity will be used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435212390
##
File path: .github/workflows/build_wheels.yml
##
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+branches:
+ - master
+ - release-*
+tags:
+ - v*
+
+jobs:
+
+ build_source:
+runs-on: ubuntu-18.04
+steps:
+ - name: Checkout code
+uses: actions/checkout@v2
+ - name: Install python
+uses: actions/setup-python@v2
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+run: python3 -m pip install wheel
+ - name: Buld source
+working-directory: ./sdks/python
+run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+working-directory: ./sdks/python
+run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+working-directory: ./sdks/python
+run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+uses: actions/upload-artifact@v2
+with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+uses: actions/upload-artifact@v2
+with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+name: Prepare GCS
+needs: build_source
+runs-on: ubuntu-18.04
+steps:
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/"
|| true
+
+ upload_source_to_gcs:
+name: Upload source to GCS bucket
+needs: prepare_gcs
+runs-on: ubuntu-18.04
+steps:
+ - name: Download wheels
+uses: actions/download-artifact@v2
+with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET
}}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+name: Build wheels on ${{ matrix.os }}
+needs: prepare_gcs
+runs-on: ${{ matrix.os }}
+strategy:
+ matrix:
+os : [ubuntu-18.04, macos-10.15]
Review comment:
Since some unix specific c libraries are not present on windows
(`unistd.h`, `get_clocktime`) I will work on it in separate branch/PR. I think
there will be no need to create seperate file for windows
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r435203092 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 +steps: + - name: Checkout code +uses: actions/checkout@v2 + - name: Install python +uses: actions/setup-python@v2 +with: + python-version: 3.7 + - name: Get build dependencies +working-directory: ./sdks/python +run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt + - name: Install wheels +run: python3 -m pip install wheel + - name: Buld source +working-directory: ./sdks/python +run: python3 setup.py sdist --formats=gztar,zip + - name: Unzip source +working-directory: ./sdks/python +run: unzip dist/$(ls dist | grep .zip | head -n 1) + - name: Rename source directory +working-directory: ./sdks/python +run: mv $(ls | grep apache-beam) apache-beam-source + - name: Upload source +uses: actions/upload-artifact@v2 +with: + name: source + path: sdks/python/apache-beam-source + - name: Upload compressed sources +uses: actions/upload-artifact@v2 +with: + name: source_gztar_zip Review comment: It is uploading sdist output (`zip`,`tag.gz`) files as artifacts where they are picked up by `upload_source_to_gcs` jobs. An additional advantage of the artifacts is fact that they can be downloaded from GitHub Actions workflow web view as well. Example of GCS output for `build_source -> upload_source_to_gcs` jobs: ``` gs://**/apache-beam-2.23.0.dev0.tar.gz gs://**/apache-beam-2.23.0.dev0.zip ``` [here](https://github.com/TobKed/beam/actions/runs/123711355) you can check GitHub Action which ran on my fork. Wheel files (`*.whl`) are processed accordingly by `build_wheels -> upload_wheels_to_gcs` jobs. Example of GCS output at the end of the workflow: ``` gs://**/apache-beam-2.23.0.dev0.tar.gz gs://**/apache-beam-2.23.0.dev0.zip gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_x86_64.whl ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r435203092 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 +steps: + - name: Checkout code +uses: actions/checkout@v2 + - name: Install python +uses: actions/setup-python@v2 +with: + python-version: 3.7 + - name: Get build dependencies +working-directory: ./sdks/python +run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt + - name: Install wheels +run: python3 -m pip install wheel + - name: Buld source +working-directory: ./sdks/python +run: python3 setup.py sdist --formats=gztar,zip + - name: Unzip source +working-directory: ./sdks/python +run: unzip dist/$(ls dist | grep .zip | head -n 1) + - name: Rename source directory +working-directory: ./sdks/python +run: mv $(ls | grep apache-beam) apache-beam-source + - name: Upload source +uses: actions/upload-artifact@v2 +with: + name: source + path: sdks/python/apache-beam-source + - name: Upload compressed sources +uses: actions/upload-artifact@v2 +with: + name: source_gztar_zip Review comment: It is uploading sdist output (`zip`,`tag.gz`) files as artifacts where they are picked up by `upload_source_to_gcs` jobs. An additional advantage of the artifacts is fact that they can be downloaded from GitHub Actions workflow web view as well. Example of GCS output for `build_source -> upload_source_to_gcs` jobs: ``` gs://**/apache-beam-2.23.0.dev0.tar.gz gs://**/apache-beam-2.23.0.dev0.zip ``` Wheel files (`*.whl`) are processed accordingly by `build_wheels -> upload_wheels_to_gcs` jobs. Example of GCS output at the end of the workflow: ``` gs://**/apache-beam-2.23.0.dev0.tar.gz gs://**/apache-beam-2.23.0.dev0.zip gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-macosx_10_9_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_i686.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_x86_64.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_i686.whl gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_x86_64.whl ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r435192970 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 Review comment: I found information in the documentation ([source](https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources)) which says: > Note: The Ubuntu 20.04 virtual environment is currently provided as a preview only. The ubuntu-latest YAML workflow label still uses the Ubuntu 18.04 virtual environment. In this case using `ubuntu-latest` label may be a good idea. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
TobKed commented on a change in pull request #11877: URL: https://github.com/apache/beam/pull/11877#discussion_r435190449 ## File path: .github/workflows/build_wheels.yml ## @@ -0,0 +1,141 @@ +name: Build python wheels + +on: + push: +branches: + - master + - release-* +tags: + - v* + +jobs: + + build_source: +runs-on: ubuntu-18.04 Review comment: > You can specify the runner type for each job in a workflow. Each job in a workflow executes in a fresh instance of the virtual machine. All steps in the job execute in the same instance of the virtual machine, allowing the actions in that job to share information using the filesystem. source: https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
