GitHub user dosubot[bot] added a comment to the discussion: Slow saving changes on exist dataset
<!-- Answer --> This is a well-documented pain point. When you save a virtual dataset, Superset runs a **two-stage synchronous pipeline** [[1]](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/datasets/api.py#L429-L431): 1. **`UpdateDatasetCommand`** — saves your metadata changes to the Superset DB (with several validation queries for column/metric uniqueness, existence, etc.) 2. **`RefreshDatasetCommand`** — **executes your dataset's SQL query** (with `LIMIT 1`) against the underlying data warehouse to sync column metadata [[2]](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/commands/dataset/refresh.py#L44-L48) That second step is the main bottleneck: the HTTP request **blocks** until your warehouse responds. If you're connected to something like Databricks, Snowflake, or BigQuery with cold-start or query compilation overhead, even a trivial save can take a long time [[3]](https://github.com/apache/superset/issues/38012#issuecomment-3908177644). There's currently no async offloading for this step. A few other factors that can compound the slowness: - **Large number of columns/metrics** — the update path loads the full column and metric collections into memory for diffing before persisting changes [[4]](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/daos/dataset.py#L247). - **Frontend freeze with 100+ metrics** — if the editor itself is slow/frozen, [PR #37555](https://github.com/apache/superset/pull/37555) added pagination to the dataset editor tables to fix a 6+ minute browser hang [[5]](https://github.com/apache/superset/pull/37555). Make sure you're on a version that includes this fix. **What you can try:** - **Increase gunicorn workers** so that one blocked save doesn't starve other requests. - **Check your underlying database performance** — the save speed is directly tied to how fast that warehouse can execute a `SELECT ... LIMIT 1` version of your dataset SQL. - **Upgrade Superset** if you're on an older version — several performance fixes have landed (consolidated double PUT calls in [#33523](https://github.com/apache/superset/pull/33523), N+1 fix in [#37899](https://github.com/apache/superset/pull/37899), editor pagination in [#37555](https://github.com/apache/superset/pull/37555)). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=brand-link).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/d08910e7-8175-445c-9f86-ecdb9046bddd?feedback_type=great_response&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/d08910e7-8175-445c-9f86-ecdb9046bddd?feedback_type=irrelevant_answer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/d08910e7-8175-445c-9f86-ecdb9046bddd?feedback_type=incorrect_sources&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/d08910e7-8175-445c-9f86-ecdb9046bddd?feedback_type=too_verbose&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d08910e7-8175-44 5c-9f86-ecdb9046bddd?feedback_type=hallucination&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/d08910e7-8175-445c-9f86-ecdb9046bddd?feedback_type=bug_report&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-bug_report) | [Other](https://app.dosu.dev/response-feedback/d08910e7-8175-445c-9f86-ecdb9046bddd?feedback_type=other&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-other)</sup> [](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=chat-badge) [](https://cursor.com/link/prompt?text=This%20is%20a%20well-documented%20pain%20point.%20When%20you%20save%20a%20virtual%20dataset%2C%20Superset%20runs%20a%20%2A%2Atwo-stage%20synchronous%20pipeline%2A%2A%20%5B%5B1%5D%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/datasets/api.py%23L429-L431%29%3A%0A%0A1.%20%2A%2A%60UpdateDatasetCommand%60%2A%2A%20%E2%80%94%20saves%20your%20metadata%20changes%20to%20the%20Superset%20DB%20%28with%20several%20validation%20queries%20for%20column/metric%20uniqueness%2C%20existence%2C%20etc.%29%0A2.%20%2A%2A%60RefreshDatasetCommand%60%2A%2A%20%E2%80%94%20%2A%2Aexecutes%20your%20dataset%27s%20SQL%20query%2A%2A%20%28with%20%60LIMIT%201%60% 29%20against%20the%20underlying%20data%20warehouse%20to%20sync%20column%20metadata%20%5B%5B2%5D%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/commands/dataset/refresh.py%23L44-L48%29%0A%0AThat%20second%20step%20is%20the%20main%20bottleneck%3A%20the%20HTTP%20request%20%2A%2Ablocks%2A%2A%20until%20your%20warehouse%20responds.%20If%20you%27re%20connected%20to%20something%20like%20Databricks%2C%20Snowflake%2C%20or%20BigQuery%20with%20cold-start%20or%20query%20compilation%20overhead%2C%20even%20a%20trivial%20save%20can%20take%20a%20long%20time%20%5B%5B3%5D%5D%28https%3A//github.com/apache/superset/issues/38012%23issuecomment-3908177644%29.%20There%27s%20currently%20no%20async%20offloading%20for%20this%20step.%0A%0AA%20few%20other%20factors%20that%20can%20compound%20the%20slowness%3A%0A%0A-%20%2A%2ALarge%20number%20of%20columns/metrics%2A%2A%20%E2%80%94%20the%20update%20path%20loads%20the%20full%20column%20and%20metric%20collections%20int o%20memory%20for%20diffing%20before%20persisting%20changes%20%5B%5B4%5D%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/daos/dataset.py%23L247%29.%0A-%20%2A%2AFrontend%20freeze%20with%20100%2B%20metrics%2A%2A%20%E2%80%94%20if%20the%20editor%20itself%20is%20slow/frozen%2C%20%5BPR%20%2337555%5D%28https%3A//github.com/apache/superset/pull/37555%29%20added%20pagination%20to%20the%20dataset%20editor%20tables%20to%20fix%20a%206%2B%20minute%20browser%20hang%20%5B%5B5%5D%5D%28https%3A//github.com/apache/superset/pull/37555%29.%20Make%20sure%20you%27re%20on%20a%20version%20that%20includes%20this%20fix.%0A%0A%2A%2AWhat%20you%20can%20try%3A%2A%2A%0A%0A-%20%2A%2AIncrease%20gunicorn%20workers%2A%2A%20so%20that%20one%20blocked%20save%20doesn%27t%20starve%20other%20requests.%0A-%20%2A%2ACheck%20your%20underlying%20database%20performance%2A%2A%20%E2%80%94%20the%20save%20speed%20is%20directly%20tied%20to%20how%20fast%20that%20warehouse%20can%20execute %20a%20%60SELECT%20...%20LIMIT%201%60%20version%20of%20your%20dataset%20SQL.%0A-%20%2A%2AUpgrade%20Superset%2A%2A%20if%20you%27re%20on%20an%20older%20version%20%E2%80%94%20several%20performance%20fixes%20have%20landed%20%28consolidated%20double%20PUT%20calls%20in%20%5B%2333523%5D%28https%3A//github.com/apache/superset/pull/33523%29%2C%20N%2B1%20fix%20in%20%5B%2337899%5D%28https%3A//github.com/apache/superset/pull/37899%29%2C%20editor%20pagination%20in%20%5B%2337555%5D%28https%3A//github.com/apache/superset/pull/37555%29%29.) [](https://go.dosu.dev/discord-bot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=join-discord) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/discussions/38975) GitHub link: https://github.com/apache/superset/discussions/38975#discussioncomment-16395479 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
