pierrejeambrun opened a new pull request #20386: URL: https://github.com/apache/airflow/pull/20386
closes: https://github.com/apache/airflow/issues/20283 In this PR: - [x] Upgrade the minimum package requirement to 2.33.0 for apache-beam (first stable for beam go sdk) - [x] Refactor `operators/beam.py` with an abstract `BeamBasePipelineOperator` class to factorize initialization and common code, also fixed mypy hook on ``BeamDataflowMixin`` - [x] Add `BeamRunGoPipelineOperator` and `BeamHook.start_go_pipeline` (+tests) - [x] Add `utils/go_module.py` to handle initialisation and dependency installation for a module. (+ tests) - [x] Slightly modified `process_util` + tests to be able to handle an extra optional parameter `cwd`. (This way we can move to the module directory to build it) - [x] Write docs - [x] Add dags examples, with system tests - [x] Run pre-commits - [ ] We need to install Go in docker images to be able to run it. Go 1.16 minimum for beam 2.33 ``` bash wget https://dl.google.com/go/go1.16.4.linux-amd64.tar.gz sudo tar -xvf go1.16.4.linux-amd64.tar.gz sudo mv go /usr/local # in bashrc export GOROOT=/usr/local/go export PATH=$GOPATH/bin:$GOROOT/bin:$PATH ``` --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
