mik-laj edited a comment on pull request #16571: URL: https://github.com/apache/airflow/pull/16571#issuecomment-879367442
> How common is it for a DAG to run for over an hour? In the case of KubernetesPodOperator, this is not so uncommon as this operator is just used to run heavy and resource-consuming tasks. We can deal with this later as this is a separate effort, but could you describe this limitation in docs and create a ticket about it after merging? > What do we gain from rewriting that, other than another chunk of code to maintain and another point of failure to watch? There are two reasons: - Better support for credential management. Airflow has a different chain for establishing credentials which are not compatible with [the Default Credential Provider Chain for AWS CLI](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html). In particular, we support [the secret backend for retrieving credentials](http://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html), [SPEGO Authentication](https://github.com/apache/airflow/blob/a3f5c93806258b5ad396a638ba0169eca7f9d065/airflow/providers/amazon/aws/hooks/base_aws.py#L256) and other. You can keep CLI, but you will find it difficult to handle all of these changes while we continue to use AWS CLI. - Reducing system dependencies, which makes it easier to use this operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
