dv-gorasiya commented on PR #61654:
URL: https://github.com/apache/airflow/pull/61654#issuecomment-3977575699
Hi @shahar1, thanks for the feedback. Here are the results from **real DAG
runs against a live GCP instance** (`airflow-pr-test-61654`), demonstrating
both the regression and the fix.
### Test Environment
- **GCP Project**: `airflow-pr-test-61654` (fresh project with Secret
Manager, BigQuery, and GCS APIs enabled)
- **ADC Configuration**: Application Default Credentials with **no quota
project** and **no default project** (`google.auth.default()` returns
`project_id=None`)
- **gcloud config**: `core/project` explicitly unset, `GOOGLE_CLOUD_PROJECT`
env var unset
- **Airflow**: 3.0.6 with `apache-airflow-providers-google` 16.0.0, patched
to match `main` branch behavior
- **Secret**: `airflow-test-secret` created in Secret Manager with a test
value
### Test DAGs
Three DAGs were created, each exercising a different GCP operator pattern
with an **explicit `project_id`**:
1. **`test_secret_manager_backend`** — Initializes
`CloudSecretManagerBackend(project_id="airflow-pr-test-61654")` and reads a
secret from Secret Manager
2. **`test_bigquery_hook`** — Calls `get_credentials_and_project_id()` (same
path `BigQueryHook` uses), then runs `SELECT 1` on BigQuery with explicit
project
3. **`test_gcs_hook`** — Calls `get_credentials_and_project_id()` (same path
`GCSHook` uses), then lists buckets with explicit project
---
### BEFORE Fix (Regression on `main`)
All three DAGs **fail** with `AirflowException` from
`_get_credentials_using_adc()` before the explicit `project_id` can be applied:
**1. `test_secret_manager_backend` — FAILED**
```
Initializing CloudSecretManagerBackend with
project_id='airflow-pr-test-61654'...
Getting connection using `google.auth.default()` since no explicit
credentials are provided.
WARNING - No project ID could be determined.
AirflowException: Project ID could not be determined from default
credentials.
Please provide `key_secret_project_id` parameter.
DagRun Finished: dag_id=test_secret_manager_backend, state=failed,
run_duration=6.53s
```
**2. `test_bigquery_hook` — FAILED**
```
Calling get_credentials_and_project_id() with ADC (no default project)...
AirflowException: Project ID could not be determined from default
credentials.
Please provide `key_secret_project_id` parameter.
DagRun Finished: dag_id=test_bigquery_hook, state=failed, run_duration=6.55s
```
**3. `test_gcs_hook` — FAILED**
```
Calling get_credentials_and_project_id() with ADC (no default project)...
AirflowException: Project ID could not be determined from default
credentials.
Please provide `key_secret_project_id` parameter.
DagRun Finished: dag_id=test_gcs_hook, state=failed, run_duration=6.50s
```
---
### AFTER Fix (PR applied)
All three DAGs **succeed** — the explicit `project_id` is properly honored:
**1. `test_secret_manager_backend` — SUCCESS**
```
Initializing CloudSecretManagerBackend with
project_id='airflow-pr-test-61654'...
Getting connection using `google.auth.default()` since no explicit
credentials are provided.
WARNING - No project ID could be determined.
SUCCESS: Backend initialized. project_id = airflow-pr-test-61654
SUCCESS: Secret retrieved, length=32 chars
DagRun Finished: dag_id=test_secret_manager_backend, state=success,
run_duration=6.84s
```
**2. `test_bigquery_hook` — SUCCESS**
```
Calling get_credentials_and_project_id() with ADC (no default project)...
ADC returned project_id: ''
Using effective project_id: airflow-pr-test-61654
Running BigQuery query: SELECT 1 AS test_value, 'airflow-pr-61654' AS source
SUCCESS: Query returned 1 row(s): {'test_value': 1, 'source':
'airflow-pr-61654'}
DagRun Finished: dag_id=test_bigquery_hook, state=success, run_duration=7.50s
```
**3. `test_gcs_hook` — SUCCESS**
```
Calling get_credentials_and_project_id() with ADC (no default project)...
ADC returned project_id: ''
Using effective project_id: airflow-pr-test-61654
Listing buckets in project 'airflow-pr-test-61654'...
SUCCESS: Found 0 bucket(s)
DagRun Finished: dag_id=test_gcs_hook, state=success, run_duration=5.96s
```
---
### Summary
| DAG | Before Fix | After Fix |
|-----|-----------|-----------|
| `test_secret_manager_backend` | **FAILED** — `AirflowException` raised
before explicit `project_id` applied | **SUCCESS** — Secret retrieved from real
GCP Secret Manager |
| `test_bigquery_hook` | **FAILED** — `AirflowException` raised before
explicit `project_id` applied | **SUCCESS** — BigQuery query executed on real
GCP |
| `test_gcs_hook` | **FAILED** — `AirflowException` raised before explicit
`project_id` applied | **SUCCESS** — GCS bucket listing completed on real GCP |
All tests ran against the live GCP project `airflow-pr-test-61654` with ADC
configured to return `None` for `project_id`, confirming both the regression
and the fix across multiple GCP operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
