codeant-ai-for-open-source[bot] commented on code in PR #41133: URL: https://github.com/apache/superset/pull/41133#discussion_r3545202478
########## superset/tasks/export_dashboard_excel.py: ########## @@ -0,0 +1,345 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Celery task that exports every chart on a dashboard to a single multi-sheet +``.xlsx`` file, uploads it to S3, and emails the requesting user a pre-signed +download link. + +In ``"data"`` mode the task re-runs each chart's saved query context under the +requesting user, applies the live dashboard filter state, and streams the results +row-by-row into a constant-memory workbook so large dashboards never load all +data at once. In ``"images"`` mode non-table charts are instead rendered to +images (through the same headless path as scheduled reports, reflecting the live +filters) and embedded, while table-like charts stay tabular. +""" + +from __future__ import annotations + +import logging +import os +import tempfile +from datetime import datetime, timedelta, timezone +from typing import Any + +from celery.exceptions import SoftTimeLimitExceeded +from flask import current_app, g + +from superset import db, security_manager +from superset.charts.data.dashboard_filter_context import ( + apply_dashboard_filter_context, + get_dashboard_filter_context, +) +from superset.charts.schemas import ChartDataQueryContextSchema +from superset.commands.chart.data.get_data_command import ChartDataCommand +from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType +from superset.dashboards.excel_export import email +from superset.dashboards.excel_export.layout import get_charts_in_layout_order +from superset.dashboards.excel_export.screenshot import render_chart_image +from superset.extensions import cache_manager, celery_app +from superset.utils import json, s3 +from superset.utils.core import override_user +from superset.utils.excel_streaming import StreamingXlsxWriter + +logger = logging.getLogger(__name__) + +# Export modes: "data" streams every chart's tabular result (the default, +# unchanged behavior); "images" embeds non-table charts as rendered images and +# keeps only table-like charts tabular. +EXPORT_MODE_DATA = "data" +EXPORT_MODE_IMAGES = "images" + +# Viz types kept as tabular data in image mode; everything else is rendered as an +# image. Operators can override the set via ``EXCEL_EXPORT_TABLE_VIZ_TYPES``. +TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"} + +EXPORT_SOFT_TIME_LIMIT = 600 +EXPORT_HARD_TIME_LIMIT = 660 +# TTL for the per-user+dashboard in-flight lock set by the API before enqueue. +# It outlives the hard time limit so a worker killed at that limit (which skips +# the ``finally`` cleanup) cannot hold the lock forever; the delete in +# ``finally`` is the fast path that frees it as soon as the task settles. +EXPORT_INFLIGHT_CACHE_TTL = EXPORT_HARD_TIME_LIMIT + 60 + + +class _ChartSkippedError(Exception): + """Signals a chart that could not be exported and should be listed as skipped.""" + + +def _chart_label(chart: Any) -> str: + """Human-readable label for a chart in the skipped-charts list.""" + return f"{chart.id} - {chart.slice_name or ''}".strip() + + +def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]: + return [record.get(col) for col in colnames] + + +def _table_viz_types() -> set[str]: + """Viz types kept tabular in image mode (config override or built-in default).""" + return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES Review Comment: **Suggestion:** The fallback uses truthiness instead of an explicit `None` check, so configuring `EXCEL_EXPORT_TABLE_VIZ_TYPES` to an empty set (to force all charts to render as images) is ignored and silently falls back to the built-in defaults. This breaks the documented config contract; only `None` should trigger fallback. [incorrect condition logic] <details> <summary><b>Severity Level:</b> Major ⚠️</summary> ```mdx - ⚠️ Image-mode exports ignore empty table-viz configuration. - ⚠️ Operators cannot force all charts rendered as images. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. An operator configures `EXCEL_EXPORT_TABLE_VIZ_TYPES` to an empty set in `superset/superset/config.py` (documented at lines 9-12: "Set to None to fall back to the built-in default.") intending that in image-mode exports all chart viz types be rendered as images instead of tabular sheets. 2. The dashboard image-mode export endpoint `/api/v1/dashboard/<id>/export_xlsx/` in `superset/superset/dashboards/api.py` (lines 24-31 in the snippet) is called with JSON `{"mode": "images"}`, which enqueues the Celery task `export_dashboard_excel` with `mode="images"` for that dashboard. 3. Inside the task, `_renders_as_image` in `superset/tasks/export_dashboard_excel.py` (lines 17-19) calls `_table_viz_types()` (lines 12-14), which currently returns `current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES`. 4. Because the configured empty set is falsy, `_table_viz_types()` falls back to the built-in `TABLE_VIZ_TYPES` instead of using the operator-provided empty set, so viz types `"table"`, `"pivot_table_v2"`, and `"pivot_table"` are treated as tabular data and exported as sheets rather than images, breaking the documented configuration contract and the operator’s expectation that all charts render as images in image mode. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=17a8dc791fbb41d3a4a4c496c9ab672c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=17a8dc791fbb41d3a4a4c496c9ab672c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** superset/tasks/export_dashboard_excel.py **Line:** 93:93 **Comment:** *Incorrect Condition Logic: The fallback uses truthiness instead of an explicit `None` check, so configuring `EXCEL_EXPORT_TABLE_VIZ_TYPES` to an empty set (to force all charts to render as images) is ignored and silently falls back to the built-in defaults. This breaks the documented config contract; only `None` should trigger fallback. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=882e2a549b2c108132e462eee9ce9bb1d04669c7466b52649f4deba59dd922ed&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=882e2a549b2c108132e462eee9ce9bb1d04669c7466b52649f4deba59dd922ed&reaction=dislike'>👎</a> ########## superset/tasks/export_dashboard_excel.py: ########## @@ -0,0 +1,345 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Celery task that exports every chart on a dashboard to a single multi-sheet +``.xlsx`` file, uploads it to S3, and emails the requesting user a pre-signed +download link. + +In ``"data"`` mode the task re-runs each chart's saved query context under the +requesting user, applies the live dashboard filter state, and streams the results +row-by-row into a constant-memory workbook so large dashboards never load all +data at once. In ``"images"`` mode non-table charts are instead rendered to +images (through the same headless path as scheduled reports, reflecting the live +filters) and embedded, while table-like charts stay tabular. +""" + +from __future__ import annotations + +import logging +import os +import tempfile +from datetime import datetime, timedelta, timezone +from typing import Any + +from celery.exceptions import SoftTimeLimitExceeded +from flask import current_app, g + +from superset import db, security_manager +from superset.charts.data.dashboard_filter_context import ( + apply_dashboard_filter_context, + get_dashboard_filter_context, +) +from superset.charts.schemas import ChartDataQueryContextSchema +from superset.commands.chart.data.get_data_command import ChartDataCommand +from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType +from superset.dashboards.excel_export import email +from superset.dashboards.excel_export.layout import get_charts_in_layout_order +from superset.dashboards.excel_export.screenshot import render_chart_image +from superset.extensions import cache_manager, celery_app +from superset.utils import json, s3 +from superset.utils.core import override_user +from superset.utils.excel_streaming import StreamingXlsxWriter + +logger = logging.getLogger(__name__) + +# Export modes: "data" streams every chart's tabular result (the default, +# unchanged behavior); "images" embeds non-table charts as rendered images and +# keeps only table-like charts tabular. +EXPORT_MODE_DATA = "data" +EXPORT_MODE_IMAGES = "images" + +# Viz types kept as tabular data in image mode; everything else is rendered as an +# image. Operators can override the set via ``EXCEL_EXPORT_TABLE_VIZ_TYPES``. +TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"} + +EXPORT_SOFT_TIME_LIMIT = 600 +EXPORT_HARD_TIME_LIMIT = 660 +# TTL for the per-user+dashboard in-flight lock set by the API before enqueue. +# It outlives the hard time limit so a worker killed at that limit (which skips +# the ``finally`` cleanup) cannot hold the lock forever; the delete in +# ``finally`` is the fast path that frees it as soon as the task settles. +EXPORT_INFLIGHT_CACHE_TTL = EXPORT_HARD_TIME_LIMIT + 60 + + +class _ChartSkippedError(Exception): + """Signals a chart that could not be exported and should be listed as skipped.""" + + +def _chart_label(chart: Any) -> str: + """Human-readable label for a chart in the skipped-charts list.""" + return f"{chart.id} - {chart.slice_name or ''}".strip() + + +def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]: + return [record.get(col) for col in colnames] + + +def _table_viz_types() -> set[str]: + """Viz types kept tabular in image mode (config override or built-in default).""" + return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES + + +def _renders_as_image(chart: Any, mode: str) -> bool: + """Whether this chart is embedded as an image rather than streamed as data.""" + return mode == EXPORT_MODE_IMAGES and chart.viz_type not in _table_viz_types() + + +def _write_chart_image_sheet( + writer: StreamingXlsxWriter, + chart: Any, + dashboard_id: int, + active_data_mask: dict[str, Any], + user: Any, +) -> None: + """ + Render a single chart to an image and embed it as its own sheet. + + :raises _ChartSkippedError: if the chart could not be rendered + """ + image = render_chart_image(chart, dashboard_id, active_data_mask, user) + if image is None: + raise _ChartSkippedError + writer.add_image_sheet(_chart_label(chart), image) + + +def _write_chart_sheets( + writer: StreamingXlsxWriter, + chart: Any, + dashboard_id: int, + active_data_mask: dict[str, Any], +) -> None: + """ + Run a single chart's query and stream its result(s) into the workbook. + + Charts may yield more than one query (e.g. mixed-series charts); each becomes + its own sheet. Raises if the chart cannot be exported, so the caller can skip + it and note it in the email. + """ + json_body = json.loads(chart.query_context) + # Override any stale saved values: we always want full JSON results. + json_body["result_format"] = ChartDataResultFormat.JSON + json_body["result_type"] = ChartDataResultType.FULL + json_body.pop("force", None) + + filter_context = get_dashboard_filter_context( + dashboard_id=dashboard_id, + chart_id=chart.id, + active_data_mask=active_data_mask, + ) + if filter_context.extra_form_data: + apply_dashboard_filter_context(json_body, filter_context.extra_form_data) + + # Jinja macros resolve form data from g.form_data; expose the saved context. + g.form_data = json_body + + query_context = ChartDataQueryContextSchema().load(json_body) + command = ChartDataCommand(query_context) + command.validate() + result = command.run() + + for index, query in enumerate(result["queries"]): + colnames = query.get("colnames") or [] + data = query.get("data") or [] + if index == 0: + name = f"{chart.id} - {chart.slice_name or ''}" + else: + name = f"{chart.id}.{index} - {chart.slice_name or ''}" + writer.add_sheet( + name, + colnames, + (_record_to_row(record, colnames) for record in data), + ) + + +def _build_workbook( + path: str, + dashboard: Any, + active_data_mask: dict[str, Any], + job_id: str, + mode: str, + user: Any, +) -> dict[str, list[str]]: + """Build the workbook on disk. + + Return the charts that could not be exported, grouped by the reason they + were omitted (see the ``email.ERROR_*`` reason keys), so the notification + can explain each group separately. + """ + errored: dict[str, list[str]] = {} + writer = StreamingXlsxWriter(path) + try: + for chart in get_charts_in_layout_order(dashboard): + label = _chart_label(chart) + as_image = _renders_as_image(chart, mode) + # Image charts render from their saved params and don't need a query + # context; data (and table) charts still do. + if not as_image and not chart.query_context: + errored.setdefault(email.ERROR_NO_QUERY_CONTEXT, []).append(label) + continue + try: + if as_image: + _write_chart_image_sheet( + writer, chart, dashboard.id, active_data_mask, user + ) + else: + _write_chart_sheets(writer, chart, dashboard.id, active_data_mask) + except SoftTimeLimitExceeded: + logger.warning( + "Chart %s timed out in dashboard export %s", chart.id, job_id + ) + errored.setdefault(email.ERROR_TIMEOUT, []).append(label) + except _ChartSkippedError: + logger.warning( + "Skipping chart %s in dashboard export %s (could not render)", + chart.id, + job_id, + ) + errored.setdefault(email.ERROR_GENERAL, []).append(label) + except Exception: # pylint: disable=broad-except + logger.exception( + "Skipping chart %s in dashboard export %s", chart.id, job_id + ) + errored.setdefault(email.ERROR_GENERAL, []).append(label) + + if writer.sheet_count == 0: + flat = [label for labels in errored.values() for label in labels] + writer.add_summary_sheet( + "Export Summary", + ["No chart data could be exported.", *flat], + ) + finally: + writer.close() + return errored + + +def _send_failure_email( + user: Any, dashboard_title: str, requested_at: datetime +) -> None: + if not (user and getattr(user, "email", None)): + return + try: + email.send_export_email( + user.email, + email.build_subject(dashboard_title, success=False), + email.build_failure_email(dashboard_title, requested_at), + ) + except Exception: # pylint: disable=broad-except + logger.exception("Failed to send export failure email") + + +@celery_app.task( + name="export_dashboard_excel", + bind=True, + soft_time_limit=EXPORT_SOFT_TIME_LIMIT, + time_limit=EXPORT_HARD_TIME_LIMIT, + max_retries=0, +) +def export_dashboard_excel( + self: Any, # pylint: disable=unused-argument + dashboard_id: int, + user_id: int, + active_data_mask: dict[str, Any], + job_id: str, + mode: str = EXPORT_MODE_DATA, + inflight_key: str | None = None, +) -> None: + """ + Export a dashboard's charts to an ``.xlsx`` and email a download link. + + :param dashboard_id: The dashboard to export + :param user_id: The requesting user (the task runs with their permissions) + :param active_data_mask: Live dashboard filter state keyed by native filter id + :param job_id: Correlation id, also the Celery task id and S3 object name + :param mode: ``"data"`` streams every chart's tabular result; ``"images"`` + embeds non-table charts as rendered images and keeps tables tabular + :param inflight_key: Cache key of the per-user+dashboard throttle lock, freed + when the task settles (the lock's TTL is the backstop) + """ + # pylint: disable=import-outside-toplevel + from superset.models.dashboard import Dashboard + + requested_at = datetime.now(tz=timezone.utc) + user = security_manager.get_user_by_id(user_id) + dashboard_title = "" + tmp_path: str | None = None + + try: + with override_user(user, force=False): + dashboard = ( + db.session.query(Dashboard).filter_by(id=dashboard_id).one_or_none() + ) + if dashboard is None: + raise ValueError(f"Dashboard {dashboard_id} not found") + dashboard_title = dashboard.dashboard_title or f"Dashboard {dashboard_id}" + + file_descriptor, tmp_path = tempfile.mkstemp( + suffix=".xlsx", prefix=f"dash-export-{job_id}-" + ) + os.close(file_descriptor) + + errored = _build_workbook( + tmp_path, dashboard, active_data_mask, job_id, mode, user + ) + + bucket = current_app.config["EXCEL_EXPORT_S3_BUCKET"] + key = ( + f"{current_app.config['EXCEL_EXPORT_S3_KEY_PREFIX']}" + f"{dashboard_id}/{job_id}.xlsx" + ) + ttl = current_app.config["EXCEL_EXPORT_LINK_TTL_SECONDS"] + + s3.upload_file_to_s3(tmp_path, bucket, key) + download_url = s3.generate_presigned_url(bucket, key, ttl) + expires_at = datetime.now(tz=timezone.utc) + timedelta(seconds=ttl) + + if user and getattr(user, "email", None): + try: + email.send_export_email( + user.email, + email.build_subject(dashboard_title, success=True), + email.build_success_email( + dashboard_title=dashboard_title, + download_url=download_url, + requested_at=requested_at, + expires_at=expires_at, + ttl_seconds=ttl, + errored=errored, + ), + ) + except Exception: # pylint: disable=broad-except + # The file is already in S3; a send failure should not trigger + # a misleading failure email. + logger.exception("Failed to send export success email") + except SoftTimeLimitExceeded: + logger.warning("Dashboard excel export %s timed out", job_id) + _send_failure_email(user, dashboard_title, requested_at) + raise + except Exception: + logger.exception("Dashboard excel export %s failed", job_id) + _send_failure_email(user, dashboard_title, requested_at) + raise + finally: + if inflight_key: + try: + cache_manager.cache.delete(inflight_key) + except Exception: # pylint: disable=broad-except + # Best-effort: the lock's TTL is the backstop if this fails. + logger.exception( + "Failed to clear in-flight export lock %s", inflight_key + ) Review Comment: **Suggestion:** The in-flight lock is deleted by key only, without verifying that the lock value still belongs to the current job. If two exports for the same user+dashboard overlap (for example after a lock TTL expiry or duplicate enqueue), the earlier task can delete the newer task's lock and allow additional concurrent exports. Store the lock value as `job_id` and only clear it when the cached value matches the current task's `job_id` (compare-and-delete behavior). [race condition] <details> <summary><b>Severity Level:</b> Critical 🚨</summary> ```mdx - ❌ Dashboard Excel export throttle allows overlapping tasks. - ⚠️ Multiple exports strain database and S3 resources. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. In `superset/superset/dashboards/api.py` the export_xlsx handler (lines 12-23 in the snippet) handles `POST /api/v1/dashboard/<id>/export_xlsx/`, computes `inflight_key = f"excel-export-inflight:{g.user.id}:{dashboard.id}"`, generates a `job_id`, and stores the throttle lock with `cache_manager.cache.set(inflight_key, job_id, timeout=EXPORT_INFLIGHT_CACHE_TTL)` before enqueuing `export_dashboard_excel`. 2. The first Celery task `export_dashboard_excel` (defined in `superset/tasks/export_dashboard_excel.py` around lines 251-259) runs for longer than `EXPORT_INFLIGHT_CACHE_TTL` due to a stuck worker or misconfigured time limits, so the cache entry for `inflight_key` expires while the task is still executing. 3. While the first task is still running, the same user calls the export endpoint again; since `cache_manager.cache.get(inflight_key)` now returns falsy (the lock expired), the handler enqueues a second `export_dashboard_excel` task for the same user+dashboard and overwrites the cache value with a new `job_id` via `cache_manager.cache.set(inflight_key, new_job_id, timeout=EXPORT_INFLIGHT_CACHE_TTL)`. 4. When the first task finally reaches its `finally` block in `superset/tasks/export_dashboard_excel.py` lines 336-343, it sees a non-null `inflight_key` and unconditionally executes `cache_manager.cache.delete(inflight_key)`, deleting the lock that now belongs to the second task; a third export_xlsx call sees no lock and enqueues yet another concurrent export, defeating the intended one-export-per-user+dashboard throttle. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1e9713dd44b346c7ae3f52e53576f724&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1e9713dd44b346c7ae3f52e53576f724&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** superset/tasks/export_dashboard_excel.py **Line:** 336:343 **Comment:** *Race Condition: The in-flight lock is deleted by key only, without verifying that the lock value still belongs to the current job. If two exports for the same user+dashboard overlap (for example after a lock TTL expiry or duplicate enqueue), the earlier task can delete the newer task's lock and allow additional concurrent exports. Store the lock value as `job_id` and only clear it when the cached value matches the current task's `job_id` (compare-and-delete behavior). Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=5e557f20b8f92b28fe1abfedad0f07f7358434285ce97676cf57245881b492a5&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=5e557f20b8f92b28fe1abfedad0f07f7358434285ce97676cf57245881b492a5&reaction=dislike'>👎</a> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
