codeant-ai-for-open-source[bot] commented on code in PR #41133: URL: https://github.com/apache/superset/pull/41133#discussion_r3589640921
########## superset/tasks/export_dashboard_excel.py: ########## @@ -0,0 +1,361 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Celery task that exports every chart on a dashboard to a single multi-sheet +``.xlsx`` file, uploads it to S3, and emails the requesting user a pre-signed +download link. + +In ``"data"`` mode the task re-runs each chart's saved query context under the +requesting user, applies the live dashboard filter state, and streams the results +row-by-row into a constant-memory workbook so large dashboards never load all +data at once. In ``"images"`` mode non-table charts are instead rendered to +images (through the same headless path as scheduled reports, reflecting the live +filters) and embedded, while table-like charts stay tabular. +""" + +from __future__ import annotations + +import logging +import os +import tempfile +from datetime import datetime, timedelta, timezone +from typing import Any + +from celery.exceptions import SoftTimeLimitExceeded +from flask import current_app, g + +from superset import db, security_manager +from superset.charts.data.dashboard_filter_context import ( + apply_dashboard_filter_context, + get_dashboard_filter_context, +) +from superset.charts.schemas import ChartDataQueryContextSchema +from superset.commands.chart.data.get_data_command import ChartDataCommand +from superset.commands.distributed_lock.release import ReleaseDistributedLock +from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType +from superset.dashboards.excel_export import email +from superset.dashboards.excel_export.layout import get_charts_in_layout_order +from superset.dashboards.excel_export.screenshot import render_chart_image +from superset.extensions import celery_app +from superset.utils import json, s3 +from superset.utils.core import override_user +from superset.utils.excel_streaming import StreamingXlsxWriter + +logger = logging.getLogger(__name__) + +# Export modes: "data" streams every chart's tabular result (the default, +# unchanged behavior); "images" embeds non-table charts as rendered images and +# keeps only table-like charts tabular. +EXPORT_MODE_DATA = "data" +EXPORT_MODE_IMAGES = "images" + +# Viz types kept as tabular data in image mode; everything else is rendered as an +# image. Operators can override the set via ``EXCEL_EXPORT_TABLE_VIZ_TYPES``. +TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"} + +EXPORT_SOFT_TIME_LIMIT = 600 +EXPORT_HARD_TIME_LIMIT = 660 + +# Namespace + TTL for the per-user+dashboard in-flight lock the API acquires +# before enqueue and this task releases when it settles. The lock uses the +# shared, atomic DistributedLock backend (Redis when configured, the metadata +# DB otherwise) so it actually synchronizes across the web server and workers — +# unlike a plain cache, which is a no-op under the default ``NullCache``. +# The TTL outlives the hard time limit so a worker killed at that limit (which +# skips the ``finally`` release) cannot hold the lock forever; the release in +# ``finally`` is the fast path that frees it as soon as the task settles. +EXPORT_LOCK_NAMESPACE = "excel_export" +EXPORT_LOCK_TTL_SECONDS = EXPORT_HARD_TIME_LIMIT + 60 + + +def export_lock_params(user_id: int, dashboard_id: int) -> dict[str, int]: + """Key parameters identifying the per-user+dashboard in-flight lock.""" + return {"user_id": user_id, "dashboard_id": dashboard_id} + + +class _ChartSkippedError(Exception): + """Signals a chart that could not be exported and should be listed as skipped.""" + + +def _chart_label(chart: Any) -> str: + """Human-readable label for a chart in the skipped-charts list.""" + return f"{chart.id} - {chart.slice_name or ''}".strip() + + +def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]: + return [record.get(col) for col in colnames] + + +def _table_viz_types() -> set[str]: + """Viz types kept tabular in image mode (config override or built-in default).""" + return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES + + +def _renders_as_image(chart: Any, mode: str) -> bool: + """Whether this chart is embedded as an image rather than streamed as data.""" + return mode == EXPORT_MODE_IMAGES and chart.viz_type not in _table_viz_types() + + +def _write_chart_image_sheet( + writer: StreamingXlsxWriter, + chart: Any, + dashboard_id: int, + active_data_mask: dict[str, Any], + user: Any, +) -> None: + """ + Render a single chart to an image and embed it as its own sheet. + + :raises _ChartSkippedError: if the chart could not be rendered + """ + image = render_chart_image(chart, dashboard_id, active_data_mask, user) + if image is None: + raise _ChartSkippedError + writer.add_image_sheet(_chart_label(chart), image) + + +def _write_chart_sheets( + writer: StreamingXlsxWriter, + chart: Any, + dashboard_id: int, + active_data_mask: dict[str, Any], +) -> None: + """ + Run a single chart's query and stream its result(s) into the workbook. + + Charts may yield more than one query (e.g. mixed-series charts); each becomes + its own sheet. Raises if the chart cannot be exported, so the caller can skip + it and note it in the email. + """ + json_body = json.loads(chart.query_context) + # Override any stale saved values: we always want full JSON results. + json_body["result_format"] = ChartDataResultFormat.JSON + json_body["result_type"] = ChartDataResultType.FULL + json_body.pop("force", None) + + filter_context = get_dashboard_filter_context( + dashboard_id=dashboard_id, + chart_id=chart.id, + active_data_mask=active_data_mask, + ) Review Comment: **Suggestion:** This call happens once per chart and `get_dashboard_filter_context` re-queries the dashboard and reparses its metadata each time, creating an avoidable N+1 pattern on dashboards with many charts. Precompute the dashboard filter metadata/context once per export (or pass the already-loaded dashboard structure) to avoid repeated DB and JSON work in the chart loop. [performance] <details> <summary><b>Severity Level:</b> Critical 🚨</summary> ```mdx - ❌ Export task issues redundant dashboard queries per chart. - ⚠️ Large dashboard exports slower, more likely to timeout. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. Create or identify a dashboard with many charts and native filters in Superset; this dashboard will be served via the Dashboard API in superset/dashboards/api.py. 2. Trigger an Excel export by calling POST /api/v1/dashboard/{pk}/export_xlsx/ implemented in superset/dashboards/api.py lines 22-30 and 1656-66. The endpoint validates the payload, checks permissions, and enqueues the Celery task export_dashboard_excel.apply_async with kwargs including dashboard_id, user_id, active_data_mask, job_id, and mode (lines 48-57). 3. A Celery worker executes export_dashboard_excel (superset/tasks/export_dashboard_excel.py lines 266-308). Inside the task, it queries the Dashboard once with db.session.query(Dashboard).filter_by(id=dashboard_id).one_or_none() (lines 293-297) and then calls _build_workbook(tmp_path, dashboard, active_data_mask, job_id, mode, user) (lines 306-308). 4. In _build_workbook (superset/tasks/export_dashboard_excel.py lines 180-241), the code iterates over get_charts_in_layout_order(dashboard) and, for each chart that is not rendered as an image, calls _write_chart_sheets(writer, chart, dashboard.id, active_data_mask) (lines 197-211). _write_chart_sheets (lines 131-177) calls get_dashboard_filter_context(dashboard_id=dashboard_id, chart_id=chart.id, active_data_mask=active_data_mask) for every chart. get_dashboard_filter_context in superset/charts/data/dashboard_filter_context.py (lines 269-312) performs its own db.session.query(Dashboard).filter_by(id=dashboard_id).one_or_none(), checks access, and parses dashboard.json_metadata and dashboard.position_json on each invocation. This per-chart call pattern produces an N+1 query and metadata parsing pattern on dashboards with many charts, increasing CPU, database load, and export runtime for every use of the export_xlsx endpoint. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=8eef80ab92b9414e947620f1ef80a4ff&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=8eef80ab92b9414e947620f1ef80a4ff&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** superset/tasks/export_dashboard_excel.py **Line:** 150:154 **Comment:** *Performance: This call happens once per chart and `get_dashboard_filter_context` re-queries the dashboard and reparses its metadata each time, creating an avoidable N+1 pattern on dashboards with many charts. Precompute the dashboard filter metadata/context once per export (or pass the already-loaded dashboard structure) to avoid repeated DB and JSON work in the chart loop. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=c2f39dfde16157a904f6728e8d624015d520a4236204bed94e943377e161dad1&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=c2f39dfde16157a904f6728e8d624015d520a4236204bed94e943377e161dad1&reaction=dislike'>👎</a> ########## superset/utils/excel_streaming.py: ########## @@ -0,0 +1,254 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Streaming XLSX writer for multi-sheet dashboard exports. + +Unlike :mod:`superset.utils.excel`, which builds an in-memory DataFrame per +sheet and hands the whole thing to ``xlsxwriter`` at once, this writer opens the +workbook in ``constant_memory`` mode and writes rows one at a time, so +``xlsxwriter`` keeps at most one row per sheet buffered on the writer side. The +source records may still be materialized upstream (e.g. by the chart query +response); this bounds only the writer's own footprint, not the caller's. +""" + +from __future__ import annotations + +import math +import numbers +import re +from collections.abc import Iterable, Sequence +from datetime import date, datetime +from decimal import Decimal +from io import BytesIO +from typing import Any + +import xlsxwriter + +from superset.utils.excel import NEUTRAL_DOCUMENT_PROPERTIES + +# Excel limits a sheet name to 31 characters and forbids these characters. +MAX_SHEET_NAME_LEN = 31 +_INVALID_SHEET_CHARS_RE = re.compile(r"[\[\]:*?/\\]") +# Excel reserves the sheet name "History" (case-insensitive). +_RESERVED_SHEET_NAME = "history" + +# A worksheet holds at most 1,048,576 rows; one is reserved for the header. +MAX_DATA_ROWS_PER_SHEET = 1_048_576 - 1 + +# Leading characters that turn a cell into a formula in spreadsheet apps. Mirrors +# superset.utils.excel.quote_formulas so streamed exports get the same guard. +_FORMULA_PREFIXES = {"=", "+", "-", "@"} + +# Excel cannot represent integers beyond 10**15 without precision loss. +_MAX_EXCEL_INT = 10**15 + + +def _quote_if_formula(text: str) -> str: + """ + Prefix formula-like text with an apostrophe so spreadsheet apps treat it as + literal text (defense against formula injection). + + Leading whitespace is ignored when detecting a formula, because spreadsheet + apps still evaluate a cell whose formula prefix is preceded by spaces or + tabs (e.g. ``" =cmd"`` or ``"\\t=cmd"``). + """ + stripped = text.lstrip() + return f"'{text}" if stripped and stripped[0] in _FORMULA_PREFIXES else text + + +def _coerce_float_cell(value: Any) -> Any: + """ + Convert a ``Decimal``/real value to something ``xlsxwriter`` accepts. + + ``float()`` on a non-finite ``Decimal`` ("NaN"/"Infinity") yields a value + xlsxwriter rejects, and an over-large value can raise ``OverflowError``; + blank the former and stringify the latter, and stringify magnitudes Excel + cannot represent precisely. + """ + try: + number = float(value) + except (OverflowError, ValueError): + return str(value) + if not math.isfinite(number): + return "" + return str(number) if abs(number) > _MAX_EXCEL_INT else number + + +def sanitize_sheet_name(raw: str, used: set[str]) -> str: + """ + Produce a valid, unique Excel sheet name from ``raw``. + + Replaces forbidden characters, strips surrounding apostrophes/whitespace, + avoids the reserved name "History", truncates to 31 characters, and + disambiguates case-insensitive collisions with ``~2``/``~3`` suffixes. + The chosen name (lower-cased) is added to ``used``. + + :param raw: The desired sheet name (e.g. ``"42 - Sales by Region"``) + :param used: Lower-cased names already taken; mutated with the result + :returns: A sanitized, unique sheet name no longer than 31 characters + """ + name = _INVALID_SHEET_CHARS_RE.sub("_", raw or "") + name = name.strip().strip("'").strip() + if not name: + name = "Sheet" + if name.lower() == _RESERVED_SHEET_NAME: + name = f"{name}_" + name = name[:MAX_SHEET_NAME_LEN] + + if name.lower() not in used: + used.add(name.lower()) + return name + + suffix = 2 + while True: + marker = f"~{suffix}" + candidate = name[: MAX_SHEET_NAME_LEN - len(marker)] + marker + if candidate.lower() not in used: + used.add(candidate.lower()) + return candidate + suffix += 1 + + +def _sanitize_cell(value: Any) -> Any: + """ + Coerce a single cell value into something safe for ``xlsxwriter``. + + Quotes formula-like strings (defense against formula injection), stringifies + integers/floats Excel cannot represent precisely, renders temporal values as + ISO strings (timezones are not natively supported), and blanks out ``None`` + and non-finite floats. + """ + if value is None: + return "" + # bool is a subclass of int; preserve it before the numeric branches. + if isinstance(value, bool): + return value + if isinstance(value, str): + return _quote_if_formula(value) + if isinstance(value, (datetime, date)): + return value.isoformat() + if isinstance(value, Decimal): + return _coerce_float_cell(value) + if isinstance(value, numbers.Integral): + number = int(value) + return str(number) if abs(number) > _MAX_EXCEL_INT else number + if isinstance(value, numbers.Real): + return _coerce_float_cell(value) + # Anything else (lists, dicts, custom objects) is stringified, still guarding + # against formula injection on the resulting text. + return _quote_if_formula(str(value)) + + +class StreamingXlsxWriter: + """ + A thin wrapper over ``xlsxwriter`` in constant-memory mode that writes one + sheet per chart, row by row. + + Sheet names are sanitized and de-duplicated, cell values are sanitized for + safety/compatibility, and per-sheet row counts are capped at Excel's limit. + Always call :meth:`close` (e.g. in a ``finally`` block) to finalize the file. + """ + + def __init__(self, path: str) -> None: + self._workbook = xlsxwriter.Workbook(path, {"constant_memory": True}) + # Reset document properties so the file carries no identifying details. + self._workbook.set_properties(NEUTRAL_DOCUMENT_PROPERTIES) + self._used_sheet_names: set[str] = set() + self.sheet_count = 0 + + def add_sheet( + self, + name: str, + columns: Sequence[Any], + rows: Iterable[Sequence[Any]], + ) -> int: + """ + Write a header row followed by data rows into a new sheet. + + :param name: Desired sheet name (sanitized/de-duplicated automatically) + :param columns: Column headers + :param rows: Iterable of row sequences, streamed one at a time + :returns: The number of data rows actually written (capped just below + Excel's per-sheet limit; when the data is larger a final notice row + is appended and the dropped rows are not counted) + """ + sheet_name = sanitize_sheet_name(name, self._used_sheet_names) + worksheet = self._workbook.add_worksheet(sheet_name) + worksheet.write_row(0, 0, [_sanitize_cell(col) for col in columns]) + + # Reserve the final row for a truncation notice, so when the data + # exceeds the sheet's capacity the user can see rows were dropped + # instead of silently losing them. + row_cap = MAX_DATA_ROWS_PER_SHEET - 1 + written = 0 + truncated = False + for row in rows: + if written >= row_cap: + truncated = True + break Review Comment: **Suggestion:** The row-cap calculation is off by one: `MAX_DATA_ROWS_PER_SHEET` already excludes the header row, but another `- 1` is applied before truncation checks. This drops one extra data row and marks sheets as truncated even when the input exactly fits Excel’s real per-sheet data capacity. [off-by-one] <details> <summary><b>Severity Level:</b> Major ⚠️</summary> ```mdx - ⚠️ Last valid row dropped near Excel row limit. - ⚠️ Truncation notice misleading when data fits sheet capacity. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. Trigger any dashboard Excel export using POST /api/v1/dashboard/{pk}/export_xlsx/ (superset/dashboards/api.py lines 22-30 and 1656-66), which enqueues export_dashboard_excel and ultimately constructs a StreamingXlsxWriter in _build_workbook (superset/tasks/export_dashboard_excel.py line 195) and calls writer.add_sheet for each chart result (lines 166-177). 2. Prepare a chart whose query returns exactly 1,048,575 data rows, the maximum number of data rows Excel can hold when one row is reserved for the header; this dataset is passed as the rows iterable into StreamingXlsxWriter.add_sheet. 3. In StreamingXlsxWriter.add_sheet (superset/utils/excel_streaming.py lines 189-214), MAX_DATA_ROWS_PER_SHEET is defined as 1_048_576 - 1 with the comment “A worksheet holds at most 1,048,576 rows; one is reserved for the header” (lines 49-50). However, row_cap is then computed as MAX_DATA_ROWS_PER_SHEET - 1 (line 196), giving a cap of 1_048_574 data rows instead of the full 1_048_575 capacity. 4. As the for loop over rows executes (lines 199-205), once written reaches 1_048_574, the condition written >= row_cap becomes true, truncated is set to True, and the loop breaks, so only 1_048_574 data rows are written. The subsequent truncation notice logic (lines 206-211) writes a “[Truncated: only first 1,048,574 rows exported]” message, even though Excel could have stored all 1,048,575 rows without truncation. The last valid row is never written, and the sheet is incorrectly marked as truncated for an input that exactly fits Excel’s per-sheet data capacity. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=27c87c2bb6c742bb9da65b97c2ef3923&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=27c87c2bb6c742bb9da65b97c2ef3923&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** superset/utils/excel_streaming.py **Line:** 196:202 **Comment:** *Off By One: The row-cap calculation is off by one: `MAX_DATA_ROWS_PER_SHEET` already excludes the header row, but another `- 1` is applied before truncation checks. This drops one extra data row and marks sheets as truncated even when the input exactly fits Excel’s real per-sheet data capacity. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=aecae5568679fc5e18c195d32800608cfdc341a2695eda8e05b525c7333070c7&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41133&comment_hash=aecae5568679fc5e18c195d32800608cfdc341a2695eda8e05b525c7333070c7&reaction=dislike'>👎</a> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
