alseerx opened a new pull request, #42079:
URL: https://github.com/apache/superset/pull/42079
### SUMMARY
Fixes the "deque mutated during iteration" error (apache/superset#40903)
caused by dynamically adding/removing SQLAlchemy `connect` event listeners on
shared cached engines.
**Root cause:** `get_sqla_engine()` registered a temporary `connect` event
listener for prequeries (e.g. `SET search_path`) on the cached engine, then
removed it in a `finally` block. Under concurrent access, multiple threads
mutated the engine's internal listener deque simultaneously, triggering
`RuntimeError: deque mutated during iteration`.
**Fix:** Move prequery execution from event listeners to direct cursor
execution in `get_raw_connection()`, which runs after obtaining each
connection. This eliminates all dynamic listener mutation on shared engines
while preserving identical prequery behavior.
```python
# Before (in get_sqla_engine) — race condition on shared engine:
sqla.event.listen(engine, "connect", run_prequeries)
try:
yield engine
finally:
sqla.event.remove(engine, "connect", run_prequeries)
# After (in get_raw_connection) — no shared state mutation:
with closing(engine.raw_connection()) as conn:
if prequeries:
cursor = conn.cursor()
for prequery in prequeries:
cursor.execute(prequery)
yield conn
```
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A — backend-only change, no UI impact.
### TESTING INSTRUCTIONS
1. All pre-commit checks pass (mypy, ruff, pylint, etc.)
2. Updated unit tests verify:
- No event listeners registered in `get_sqla_engine()`
- Prequeries execute directly in `get_raw_connection()`
- Cursor cleanup on both success and exception paths
### ADDITIONAL INFORMATION
- [x] Has associated issue: Fixes #21
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
Link to Devin session:
https://app.devin.ai/sessions/90ed846778f942dc89855b900484e7f6
Requested by: @alseerx
Since @OmarDadabhoy is not responding to requests in his fork to merge the
changes to the main repository. I am "adopting" these changes to create the PR
to the main repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
