cooley-hi opened a new issue, #76331:
URL: https://github.com/apache/airflow/issues/76331
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
`apache-airflow-providers-amazon` 9.38.0 and 9.39.0 (`hooks/duckdb.py` is
unchanged on `main`). 9.37.0 is not affected because it has no
`hooks/duckdb.py`.
### Apache Airflow version
2.11.2
### Operating System
Debian bookworm (`apache/airflow:2.11.2-python3.12`), linux/amd64
### Deployment
Other Docker-based deployment (AWS ECS Fargate, `airflow webserver` with the
default gunicorn `--preload`)
### Deployment details
- `duckdb==1.4.4` comes from the Airflow 2.11.2 constraints.
- `apache-airflow-providers-duckdb` is **not** installed.
- The webserver task has 4 vCPU.
### What happened
After upgrading the Amazon provider from 9.36.0 to 9.38.0, every gunicorn
webserver worker segfaults when it exits. That covers both the
`worker_refresh_interval` TTIN/TTOU rotation and a normal SIGTERM shutdown. The
master logs `Worker (pid:N) was sent SIGSEGV!`. With core dumps enabled, each
crash wrote a ~0.7 GB core file, which slowly filled the container's ephemeral
storage. The webserver itself stayed healthy, because the master replaces each
crashed worker.
A core file from a crashed worker shows:
- `SIGSEGV`, `si_code=SEGV_MAPERR`, and one thread in the process.
- The stack, from bottom to top: Python finalization (`libpython3.12`), then
`_duckdb.cpython-312-x86_64-linux-gnu.so`, then `libstdc++`
(`std::thread::join`), then glibc `pthread_join`.
The cause is a chain of four steps:
1. `airflow/providers/amazon/aws/hooks/duckdb.py` runs `import duckdb` at
module level. It does this *before* it tries to import
`airflow.providers.duckdb.hooks.duckdb.DuckDBHook`:
```python
try:
import duckdb
from airflow.providers.duckdb.hooks.duckdb import DuckDBHook
except ImportError:
raise AirflowOptionalProviderFeatureException(...)
```
2. The 2.11 webserver force-imports every hook class registered under
`connection-types` (`ProvidersManager._import_info_from_all_hooks()`, used for
`connection_form_widgets` and `field_behaviours`). With gunicorn `--preload`,
this happens in the master before it forks.
- When `apache-airflow-providers-duckdb` is not installed, the
`DuckDBHook` import fails and the hook is skipped as an optional feature.
- By then, `duckdb` is already imported.
3. `import duckdb` (1.4.x and 1.5.x) starts DuckDB's background thread pool
immediately. The pool has `threads - 1` workers, and `threads` follows the
cgroup CPU quota.
4. Forked workers inherit DuckDB's global state but not its threads. At
interpreter exit, DuckDB's destructor joins threads that don't exist in the
child, and the process segfaults.
So the crash also hits deployments that never use DuckDB and don't have the
DuckDB provider installed.
### What you think should happen instead
Importing the hook module should have no side effects beyond defining the
class. In particular, it should not import `duckdb` when the DuckDB provider is
missing and the hook is about to be discarded.
Possible fixes:
- Import `airflow.providers.duckdb.hooks.duckdb` first, or check for it with
`importlib.util.find_spec`, before `import duckdb`. Then the optional-feature
path never loads `duckdb`.
- Move `import duckdb` into the methods that use it, such as
`configure_secrets`, and keep the type-only import under `TYPE_CHECKING`.
Note: `airflow.providers.duckdb.hooks.duckdb` also runs `import duckdb` at
module level. So deployments that *do* install the DuckDB provider likely hit
the same crash through that hook as well. That may be worth a separate look.
### How to reproduce
The minimal repro, with no Airflow involved, shows the DuckDB behavior
behind the crash:
```bash
docker run --rm --cpus 4 --platform linux/amd64
apache/airflow:2.11.2-python3.12 python -c '
import os, sys, signal
import duckdb
print("threads after import:", len(os.listdir("/proc/self/task")))
for _ in range(5):
pid = os.fork()
if pid == 0:
sys.exit(0)
_, status = os.waitpid(pid, 0)
print("child:", signal.Signals(os.WTERMSIG(status)).name if
os.WIFSIGNALED(status) else "exit %d" % os.WEXITSTATUS(status))
'
```
Results with `duckdb==1.4.4`:
| `--cpus` | DuckDB worker threads after `import duckdb` | Children killed
by SIGSEGV |
|---|---|---|
| 1 | 0 | 0 of 5 |
| 2 | 1 | 0 of 5 |
| 3 | 2 | 0 of 50 |
| 4 | 3 | 50 of 50 |
- DuckDB 1.4.5, 1.5.0 and 1.5.6 also crash.
- DuckDB 2.0.0 starts no threads on import, and its children exit cleanly.
To reproduce through the provider: install
`apache-airflow-providers-amazon==9.38.0` without the DuckDB provider, run
`airflow webserver` with at least 4 CPUs, and wait for a worker refresh, or
send SIGTERM to a worker. Running `from airflow.providers.amazon.aws.hooks
import duckdb` before a fork has the same effect as the script above, because
that module runs `import duckdb` first.
### Anything else
- Related: duckdb/duckdb-python#292 ("DuckDB global state is not fork-safe").
- Airflow 3's API server defaults to uvicorn, which starts each worker as a
new process, so the default setup is likely not affected. `[api] server_type =
gunicorn` uses preload plus fork and is likely affected.
- Workaround: pin `apache-airflow-providers-amazon<9.38`, uninstall `duckdb`
from the image, or run the webserver with 3 vCPU or fewer.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
