lohitkolluri commented on PR #67900:
URL: https://github.com/apache/airflow/pull/67900#issuecomment-4788327564
@pierrejeambrun I was able to reproduce the HTTP 500 on SQLite with this
setup:
**Reproduction method:**
1. Start Airflow with the SQLite backend (default configuration).
2. Connect to the database with SQLite CLI and lower the busy timeout so
lock contention surfaces immediately:
```sql
PRAGMA busy_timeout=0;
```
3. Hold an exclusive write lock from another session:
```sql
BEGIN EXCLUSIVE;
```
4. While the lock is held, send two concurrent `POST /backfills` requests
for the same `dag_id`.
**Result:**
Without the busy-timeout retry window (normally 5000ms by default in
SQLAlchemy's SQLite configuration), the second request hits:
```
sqlite3.OperationalError: database is locked
```
which propagates as an unhandled `OperationalError` → HTTP 500 Internal
Server Error.
**Why the busy_timeout masks this under normal conditions:**
SQLAlchemy sets `PRAGMA busy_timeout=5000` by default for SQLite (`sqlite3`
Python driver's default `timeout` parameter is 5.0 seconds). This gives short
transactions (like a single backfill creation) enough time to complete before
the busy handler fires. This is why the race isn't visible in single-process
testing — the 5-second retry window is nearly always sufficient to absorb the
contention on SQLite's single-writer lock.
When `busy_timeout` is lowered or under higher real-world concurrency (e.g.,
`airflow standalone` shares a single SQLite DB across scheduler, triggerer, API
server, and DAG processor — all writing concurrently), the underlying race
surfaces as an HTTP 500.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
