johannaojeling opened a new pull request, #28764: URL: https://github.com/apache/airflow/pull/28764
The `BeamRunGoPipelineOperator` currently has a `go_file` parameter, which represents the path to a Go source file with the pipeline code. The operator starts the pipeline with `go run`, i.e. compiles the code into a temporary binary and executes. This PR adds support for the operator to start the pipeline with an already compiled binary, as an alternative to the source file approach. It introduces two new parameters: 1. `go_binary` path to a binary compiled for the launching platform, i.e. the platform where Airflow is deployed 2. `worker_binary` (optional) path to a binary compiled for the worker platform if using a remote runner Some motivations to introduce this feature: - It does not require a Go installation on the system where Airflow is run (which is more similar to how the `BeamRunJavaPipelineOperator` works, running a jar) - It does not involve the extra steps of initializing a Go module, installing dependences and compiling the code every task run, which is what currently happens when the Go source file is downloaded from GCS - In the current implementation only a single Go source file can downloaded from GCS. This can be limiting if the project comprises multiple files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
