shivaam opened a new pull request, #63080:
URL: https://github.com/apache/airflow/pull/63080
## What
Replace `session.expunge_all()` with `session.expunge(obj)` in
`MetastoreBackend.get_connection()` and `MetastoreBackend.get_variable()`.
## Why
`expunge_all()` removes every object from the SQLAlchemy session — not just
the queried Connection or Variable. Since Airflow uses scoped (thread-local)
sessions, any code sharing the same thread gets its pending objects destroyed.
This causes team-scoped S3DagBundles to silently fail to persist. During
`sync_bundles_to_db`, the call chain is:
1. `session.add(DagBundleModel("team_alpha"))` — pending in session
2. Next iteration calls `_extract_and_sign_template("team_beta")`
3. → `S3DagBundle.view_url_template()` → `S3Hook` →
`AwsGenericHook.conn_config` → `get_connection("aws_default")`
4. → `MetastoreBackend.get_connection()` → `session.expunge_all()`
5. → `team_alpha` DagBundleModel is gone from the session
6. At commit time, nothing is left to persist
Non-team bundles (like `shared_dags`) survive only because they happen to be
processed last — no subsequent `expunge_all()` wipes them.
## How
`expunge(obj)` detaches only the specific queried object, preserving all
other session state. The original intent (return a detached object safe for
callers to use freely) is maintained without the collateral damage.
When the query returns no result (`conn is None`), we skip the expunge
entirely — nothing to detach.
## Testing
### Unit tests (4 new in `test_secrets_metastore.py`)
- `get_connection` with pending session objects — pending objects preserved ✅
- `get_connection` not found with pending session objects — session
untouched ✅
- `get_variable` with pending session objects — pending objects preserved ✅
- `get_variable` not found with pending session objects — session untouched ✅
All 6 existing `test_secrets_backends.py` tests pass unchanged.
### End-to-end on EC2 (PostgreSQL 16, multi-team, S3DagBundles)
- Fresh DB reset, 2 teams, 3 S3 bundles (2 team-scoped + 1 shared)
- All bundles persisted to `dag_bundle` table (previously only shared
survived)
- Dag-processor logs confirmed `session.new=1 (preserved)` during
MetastoreBackend lookups
- `airflow connections get aws_default` — all fields returned correctly
- `airflow variables set/get` — round-trip works
- No `DetachedInstanceError` or session errors in any service logs
closes: #62244
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Kiro (Claude Opus 4.6)
Generated-by: Kiro (Claude Opus 4.6) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
