kiranpreet-kaur17 opened a new issue, #61217:
URL: https://github.com/apache/airflow/issues/61217
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==19.4.0
### Apache Airflow version
3.x affected
### Operating System
All
### Deployment
Other
### Deployment details
N/A
### What happened
After upgrading to `apache-airflow-providers-google==19.4.0`,
`CloudSecretManagerBackend` fails to initialize when using Application Default
Credentials (ADC) without a default project configured in gcloud, even when an
explicit `project_id` is provided in `BACKEND_KWARGS`.
This issue was encountered in a local development environment.
**Error:**
```
airflow.exceptions.AirflowException: Project ID could not be determined from
default credentials. Please provide `key_secret_project_id` parameter.
```
**Configuration:**
```bash
AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"project_id":"my-project-id"}'
# Using ADC without default project (common when working with multiple
projects)
gcloud auth application-default login
gcloud config unset project
```
This configuration worked fine in version 19.3.0 and earlier.
### What you think should happen instead
`CloudSecretManagerBackend` should use the explicit `project_id` provided in
`BACKEND_KWARGS`, regardless of whether ADC returns a default project. The
validation should only fail if no project_id is available from ANY source.
### How to reproduce
1. Install `apache-airflow-providers-google==19.4.0`
2. Configure secrets backend:
```bash
export
AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"project_id":"my-project-id"}'
```
3. Set up ADC without a default project:
```bash
gcloud auth application-default login
gcloud config unset project
```
4. Start Airflow
5. Observe the error during secrets backend initialization
### Anything else
`CloudSecretManagerBackend` should use the explicit `project_id` provided in
`BACKEND_KWARGS`, regardless of whether ADC returns a default project. The
validation should only fail if no project_id is available from ANY source.
**Root Cause:**
Commit
[`0a3ee82ba3`](https://github.com/apache/airflow/commit/0a3ee82ba3005c8e20e82be56b2320b45a519f0e)
(PR #60146) added validation in
`_CredentialProvider._get_credentials_using_adc()` that raises an exception if
`google.auth.default()` returns `None` for `project_id`.
The problem is that this validation happens **before**
`CloudSecretManagerBackend.__init__` can use the explicit `project_id`
parameter:
```python
# In CloudSecretManagerBackend.__init__ (lines 120-137)
self.credentials, self.project_id = get_credentials_and_project_id(...)
# ↑ This raises exception in 19.4.0 before reaching the code below ↓
# This code is never reached:
if project_id:
self.project_id = project_id # Would have used explicit project_id!
```
**Why the error message is misleading:**
The error suggests using `key_secret_project_id`, but this parameter is for
a different purpose (specifying which project contains the **credentials
secret**, not which project contains the **Airflow secrets**). It doesn't solve
the problem for users providing `project_id` in `BACKEND_KWARGS`.
**Comparison:**
Version 19.3.0 (working):
```python
def _get_credentials_using_adc(self) -> tuple[Credentials, str]:
credentials, project_id = google.auth.default(scopes=self.scopes)
return credentials, project_id # No validation
```
Version 19.4.0 (broken):
```python
def _get_credentials_using_adc(self) -> tuple[Credentials, str]:
scopes = list(self.scopes) if self.scopes else None
credentials, project_id = google.auth.default(scopes=scopes)
if not project_id: # Fails before caller can use explicit project_id
raise AirflowException(...)
return credentials, project_id
```
**Proposed Fix:**
Move the validation to the caller level, after checking all possible sources
of `project_id`:
```python
# In CloudSecretManagerBackend.__init__
self.credentials, adc_project_id = get_credentials_and_project_id(...)
# Use explicit project_id if provided, otherwise use ADC project_id
if project_id:
self.project_id = project_id
else:
self.project_id = adc_project_id
# NOW validate after checking all sources
if not self.project_id:
raise AirflowException(
"Project ID could not be determined. "
"Please provide 'project_id' in backend configuration or ensure "
"your credentials include a default project."
)
```
**Workaround:**
Set the `GOOGLE_CLOUD_PROJECT` environment variable:
```bash
export GOOGLE_CLOUD_PROJECT=my-project-id
```
This makes `google.auth.default()` return a project_id, satisfying the
validation.
**Impact:**
This is a regression that breaks a valid and common use case:
- Using ADC for authentication (Google's recommended approach)
- Working with multiple GCP projects without a default
- Explicitly configuring project_id in secrets backend settings
Affects both Airflow 2.x and 3.x users who upgrade to google provider
19.4.0+.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
