sadpandajoe opened a new pull request, #43907: URL: https://github.com/apache/superset/pull/43907
### SUMMARY `superset load_examples` runs on every ephemeral/dev environment (`SUPERSET_LOAD_EXAMPLES=yes` in `docker/docker-init.sh`), so its wall-clock time is paid constantly. This PR trims redundant work on that default path (no `--force`, no `--load-big-data`) and adds timing so the remaining cost is measurable. **1. Dedupe dataset imports in `ImportExamplesCommand._import`** (`superset/commands/importers/v1/examples.py`) Some examples ship the same dataset config under two folders: - `world_health/dataset.yaml` and `misc_charts/datasets/wb_health_population.yaml` → same uuid, same `wb_health_population` table - `sales_dashboard/dataset.yaml` and `featured_charts/datasets/cleaned_sales_data.yaml` → same uuid, same `cleaned_sales_data` table `load_contents()` keys them under distinct paths, and `load_examples_from_configs()` always passes `overwrite=True`, so `import_dataset()` ran twice for each uuid — the second call repeating the full column/metric sync against an identical config. Each uuid is now imported at most once per run. Note: the parquet *data* was not being loaded twice on this path — `load_parquet_table()` already short-circuits on `has_table()` when `force=False`. The duplicated cost was confined to the YAML metadata import, which is what this fixes. **2. Right-size `to_sql` chunksize by column count** (`superset/examples/generic_loader.py`) `chunksize` was a flat 500 rows regardless of width. With `method="multi"`, bound params per batch = rows × columns; on the widest example table (328 columns) that's ~164k params per statement, which is near some backends' limits and slower to plan than a right-sized batch. Chunksize is now `50_000 // num_cols`, clamped to `[50, 500]` — narrow tables keep the previous 500, wide ones drop to a sane batch. **3. Timing instrumentation** (`superset/cli/examples.py`) Logs the duration of each dataset loader, the YAML config import phase, and the total `load_examples` run. This path had no timing before, so regressions/improvements here were guesswork. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF N/A — backend/CLI only. ### TESTING INSTRUCTIONS Unit tests (new `test_import_dedupes_datasets_with_same_uuid` plus existing coverage): ```bash pytest tests/unit_tests/commands/importers/v1/examples_test.py tests/unit_tests/examples/ ``` Manual: run `superset load_examples` (or bring up the docker-compose dev stack with `SUPERSET_LOAD_EXAMPLES=yes`) and check the logs — you should see `Finished [<dataset>] in N.NNs` per loader, `Finished [Examples From Configs] in N.NNs`, and `load_examples finished in N.NNs`. The examples database should have exactly one `wb_health_population` and one `cleaned_sales_data` dataset, and all example dashboards/charts should render as before. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
