varadendrasimha511 commented on issue #43764:
URL: https://github.com/apache/superset/issues/43764#issuecomment-5537329791
# Fix: Dataset import timeout (`#43764`)
## Root cause
Importing `datasets.zip` is not a simple file upload. For each dataset in the
archive, Superset:
1. Parses the YAML/JSON definition
2. Looks up whether the dataset (by UUID) already exists in Superset's
metadata database
3. Creates or updates the dataset row, plus every column and metric attached
to it
4. If `overwrite=true`, replaces existing rows instead of skipping them
5. Commits the change to the Superset metadata DB (Postgres/MySQL/etc.)
With 293 datasets in one zip, that's hundreds of sequential metadata-DB
round trips happening inside a **single synchronous HTTP request**, handled
by **one gunicorn worker**. At the default 120s timeout, the worker gets
killed mid-import:
```
[2026-09-01 09:24:20 +0000] [1033] [CRITICAL] WORKER TIMEOUT (pid:1034)
[2026-09-01 09:24:22 +0000] [1034] [INFO] Worker exiting (pid:1034)
```
...and the client sees a 503 with `upstream connect error or
disconnect/reset before headers`.
Manually raising `--timeout` to 300 on the command line "fixed" it once, but
that's not a permanent fix — the flag doesn't survive redeploys, and there's
almost always a reverse proxy in front of gunicorn with its **own**, shorter
timeout that will keep resetting the connection regardless of what gunicorn
is set to.
## Permanent fix (4 parts)
### 1. Bake the gunicorn timeout into startup config, not a CLI flag typed
by hand
Don't rely on someone manually running `-w 10 -k gevent --timeout 300` on
the command line — that's exactly what already got lost once. Put the flags
directly into whatever starts gunicorn in your deployment (Docker `CMD`,
Helm `values.yaml`, systemd `ExecStart`, entrypoint script), so it's
version-controlled and always applied the same way:
```bash
gunicorn \
--workers 10 \
--worker-class gevent \
--worker-connections 1000 \
--timeout 300 \
--graceful-timeout 30 \
--bind 0.0.0.0:8088 \
"superset.app:create_app()"
```
Or, if your image already builds the gunicorn command from environment
variables, set:
```bash
GUNICORN_CMD_ARGS="--workers=10 --worker-class=gevent
--worker-connections=1000 --timeout=300 --graceful-timeout=30"
```
Either way, the key point is: it lives in the Dockerfile/Helm chart/systemd
unit that's checked into version control, not in someone's shell history.
### 2. Raise the reverse proxy's idle/read timeout to match
The `upstream connect error ... reset reason: connection termination`
message is the signature of the **proxy**, not gunicorn, closing the
connection early. Fix whichever of these applies to your stack:
**nginx**
```nginx
location / {
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_pass http://superset_upstream;
}
```
**AWS ALB** — target group attribute:
```
Idle timeout: 300 (seconds)
```
**Kubernetes ingress-nginx** — ingress annotation:
```yaml
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
```
**GCP HTTP(S) Load Balancer** — backend service:
```
timeoutSec: 300
```
If you skip this step, raising only the gunicorn timeout will not fully fix
the 503 — the proxy will still cut the connection first.
### 3. Cap batch size on the client side as a second line of defense
Even with a generous timeout, importing 293 datasets in one request means a
single slow row or DB hiccup anywhere in the sequence stalls (and forces a
retry of) the whole batch. Split `datasets.zip` into chunks of ~25–50
datasets and loop the import call:
```bash
#!/usr/bin/env bash
set -euo pipefail
BATCH_DIR="./dataset_batches"
BATCH_SIZE=25
rm -rf "$BATCH_DIR" && mkdir -p "$BATCH_DIR"
# split the extracted dataset YAML files into batches of $BATCH_SIZE,
# then re-zip each batch
find ./datasets -maxdepth 1 -type f -name '*.yaml' | \
split -l "$BATCH_SIZE" - "$BATCH_DIR/batch_"
i=0
for batch in "$BATCH_DIR"/batch_*; do
i=$((i+1))
zip_file="$BATCH_DIR/datasets_batch_${i}.zip"
while read -r f; do zip -j "$zip_file" "$f"; done < "$batch"
echo "=== Importing batch ${i} (${zip_file}) ==="
curl -k -s -w "%{http_code}" -b cookies.txt -o "batch_${i}_response.txt" \
--url "https://$APP_PREFIX.$DOMAIN/api/v1/dataset/import/"; \
--header 'accept: application/json' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--header "referer: https://$APP_PREFIX.$DOMAIN"; \
--header "x-csrftoken: $CSRF_TOKEN" \
--form "formData=@${zip_file}" \
--form "overwrite=true" \
--form "passwords=$passwords_json"
done
```
This bounds worst-case request duration regardless of the timeout setting,
and a failure only costs one batch, not the whole 293-dataset import.
### 4. (Optional) Check metadata DB capacity if timeouts persist
Each dataset triggers several sequential writes to the Superset metadata
DB. If steps 1–3 don't fully resolve it, the remaining bottleneck is likely
the metadata database itself being under-provisioned or connection-starved
under load. That's a DB-side/ops check (CPU, connection count, slow query
log on the metadata DB) rather than an app config change, so it's not
included here as a code step — flag it separately if it comes up after
verification.
## Verification checklist
- [ ] Gunicorn flags/env vars applied and confirmed via
`ps aux | grep gunicorn` after redeploy
- [ ] Proxy timeout raised (nginx / ALB / ingress / LB — whichever applies)
and confirmed in its config
- [ ] Full 293-dataset import re-run end-to-end via the batching script
- [ ] Worker logs show no `WORKER TIMEOUT` entries
- [ ] HTTP response is `200`, not `503`
- [ ] Total wall-clock time logged as a baseline for future dataset growth
## Answer to the original question
> Although I am uploading the zip file, is it just uploading the data, or is
> it also running database queries each time I upload datasets.zip?
It's running database queries — one dataset import performs multiple
metadata-DB reads/writes per dataset (existence check, create/update of the
dataset row, its columns, and its metrics). It is **not** querying your
underlying source data warehouse; that only happens later when a chart runs
against the dataset, or if "Sync columns from source" is triggered
explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
