john-jac commented on pull request #19324: URL: https://github.com/apache/airflow/pull/19324#issuecomment-968327100
> > The boto3 secrets manager library will fail with ResourceNotFoundException if there are no restrictions in the IAM role, i.e. it has full access to secrets manager. That isn't practical for most organizations, as they will have several entities accessing secrets manager and do not want to give full access to all secrets by all of them. > > OK so what you're saying here is that it's `not practical for most organizations` to catch `ResourceNotFoundException` because in most organizations they'll get `AccessDeniedException` instead, because the cred exists but the instance does not have permission to access it. > > But why is the airflow instance trying to retrieve the cred that it does not have access to in the first place? That seems like a misconfiguration issue. If the scheduler is trying to access, for example the value for `sql_alchemy_conn` from secrets backend (that's your scenario right?), and it is unable to do so, isn't the scheduler going to fail anyway? > > You have to jump through some real hoops to make secrets backend retrieve config keys from secrets backend. And this might not be such a good thing. But this reality means that if you have set up your cluster to retrieve a config key from secrets backend, I am not seeing how you'd also expect your cluster not to have access to that secret. > > Please help me understand. I will make time to be more responsive to this one going forward so we can make sure to get to resolution soon. The fundamental way that a secrets backend works is to try all connections, variables, and configurations there first and, if it returns None, try the OS environment variables and finally the metadb. In boto3, as a security feature, whenever you have any condition or resource constraint on Secrets Manager it ALWAYS returns Access Denied on a missing value, presumably so that one cannot phish for missing vs inaccessible values. This is currently uncaught, and as no user will ever have all of their configs and connections in Secrets Manager, leads to uncaught exceptions anywhere in Airflow trying to pull a config, which is pretty much everywhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
