VijayantSoni commented on pull request #8701: URL: https://github.com/apache/airflow/pull/8701#issuecomment-645456253
Hi @ashb , I see your point and yes we can discuss about it. Please find below details for adding this. Consider following scenario: 1. We have some tasks that need to use redis (elasticache replication group in aws world) 2. We will have some task that will create and setup redis (replication group) 3. Downstream tasks will use it and then at the end one tasks will terminate the replication group. The basic idea is to create resources when needed and terminate them once the work is done. This flow can be orchestrated within Airflow itself hence the PR. Regarding managing resources via Terraform or Cloudformation, I agree with that but is it an anti-pattern to have this kind of hook in airflow ? I do see following hooks which provision resources in AWS infra: 1. https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/emr.py 2. https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/redshift.py 3. https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/s3.py (creates bucket but maybe that's not comparable here, still adding) 4. https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/sagemaker.py 5. https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/sqs.py ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
