jaketf edited a comment on issue #10454: URL: https://github.com/apache/airflow/issues/10454#issuecomment-680139027
@kaxil How would you pull your terraform configuration source? in the bash operator or the setup environment or was this very small terraform configurations embeded in your DAG code? Our idea would be to provide some abstraction of that setup the environment step to make it easy to have a terraform binary running in your airflow execution environment. My first thought was to essentially create a wrapper of KubernetesPodOperator with a git-sync initialization container and a terraform container. The idea being we could provide reasonable defaults for terraform image (e.g. official image from docker hub) or the user could override this to be a container w/ additional binaries (gcloud, providers, etc). In theory the user could also specify an image for popular wrappers like [terragrunt](https://terragrunt.gruntwork.io/) This would give a lot of flexibility to the advanced user and takes care of a lot of boilerplate. ```python tf_task = TerraformOperator( command='apply -auto-approve', git_ssh_key_secret_name='my_tf_git_sync', sub_path='terraform/my_dir', terraform_image='hashicorp/terraform:latest', gcp_secret_path='/var/secrets/key.json', ) ``` The drawback naturally is for non-k8s based airflow deployments. I think to make this really useful at most enterprises we need to think about how to best handle secrets. Terraform often needs a lot of permissions so if there was an opportunity to mange the secret for terraform's cedentials outside of airflow this would be ideal (so not every DAG can bootstrap the god-like permissions for terraform). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
