aminghadersohi commented on code in PR #45348:
URL: https://github.com/apache/superset/pull/45348#discussion_r4291021227
##########
superset/sql/execution/executor.py:
##########
@@ -404,6 +408,118 @@ def execute_sql_with_cursor(
return results
+def _decode_async_results_payload(blob: bytes) -> dict[str, Any]:
+ """
+ Decode a results backend blob written by the async SQL execution task.
+
+ The task stores a zlib-compressed payload serialized either as MessagePack
+ or as UTF-8 JSON, depending on ``RESULTS_BACKEND_USE_MSGPACK``. The format
+ is detected from the payload itself (a JSON object starts with ``{``, which
+ is never the first byte of a MessagePack map), so results stay readable if
+ the setting differs between the writing worker and the reading process.
+
+ :param blob: Compressed payload read from the results backend
+ :returns: The deserialized payload
+ """
+ import msgpack
+
+ from superset.utils import json
+
+ decompressed = cast(bytes, utils.zlib_decompress(blob, decode=False))
+ if decompressed.lstrip()[:1] == b"{":
+ return json.loads(decompressed)
+ return msgpack.loads(decompressed, raw=False)
+
+
+def _decode_async_statement_data(stmt_data: dict[str, Any]) -> pd.DataFrame |
None:
+ """
+ Convert one stored statement's data into a DataFrame.
+
+ SELECT results are Arrow IPC bytes (MessagePack) or Base64 strings (JSON).
+ Older JSON payloads contain records; DML statements store ``None``. A
+ SELECT returning no rows yields an empty DataFrame that keeps its columns,
+ matching the synchronous execution path.
+
+ :param stmt_data: Statement entry from the stored payload
+ :returns: DataFrame of the statement rows, or ``None`` for DML
+ """
+ data = stmt_data.get("data")
+ if data is None:
+ return None
+
+ if isinstance(data, str):
+ data = base64.b64decode(data)
+ if isinstance(data, (bytes, bytearray, memoryview)):
+ from superset.result_set import SupersetResultSet
+
+ table = pa.ipc.open_stream(pa.BufferReader(data)).read_all()
+ df = SupersetResultSet.convert_table_to_df(table)
+ # Older sidecars contain untyped values; do not guess lost types.
+ nested_columns = {
+ **stmt_data.get("nested_columns", {}),
+
**deserialize_nested_columns(stmt_data.get("nested_columns_msgpack", {})),
+ }
+ for name, values in nested_columns.items():
+ df[name] = values
Review Comment:
Fixed the remaining malformed-Decimal gap in
807fa02847106d703d7ef4be5d66e1294e5fca5d: the per-column fallback catches
DecimalException (including InvalidOperation), keeping the Arrow column and
independently decoded sibling columns.
Regression:
tests/unit_tests/sql/execution/test_async_results.py::test_bad_nested_sidecar_keeps_arrow_and_other_columns[tag4-True]
and [tag4-False] failed before this commit and pass after it. The expanded
test covers all seven tag decoders, timedelta unpack/shape/type/overflow
errors, non-ASCII Decimal bytes, unknown tags and both transports. Focused
before: 3 failed / 28 passed (including the separate docs regression); final
two-module run: 408 passed / 2 skipped.
Search: `rg -n
'deserialize_nested_columns|_decode_nested_scalar|_decode_async_statement_data|nested_columns_msgpack'
superset`.
Siblings checked:
- superset/sql/execution/executor.py:432 — fixed in
807fa02847106d703d7ef4be5d66e1294e5fca5d
- superset/sql/execution/executor.py:1502 — fixed in
807fa02847106d703d7ef4be5d66e1294e5fca5d
- superset/sql/execution/serialization.py:72 — not affected: tag 1 int
raises ValueError, already caught per column.
- superset/sql/execution/serialization.py:74 — not affected: tag 2 ASCII/ISO
datetime errors are ValueError subclasses, already caught.
- superset/sql/execution/serialization.py:76 — not affected: tag 3 UUID gets
bytes from MessagePack and rejects incorrect lengths with ValueError, already
caught.
- superset/sql/execution/serialization.py:78 — fixed in
807fa02847106d703d7ef4be5d66e1294e5fca5d
- superset/sql/execution/serialization.py:80 — not affected: tag 5 ASCII/ISO
date errors are ValueError subclasses, already caught.
- superset/sql/execution/serialization.py:82 — not affected: tag 6 ASCII/ISO
time errors are ValueError subclasses, already caught.
- superset/sql/execution/serialization.py:84 — not affected: tag 7 malformed
MessagePack, unpack shape, timedelta types and overflow raise
UnpackException/ValueError/TypeError/OverflowError, already caught.
- superset/sql/execution/serialization.py:118 — not affected:
Base64/MessagePack format and recursive-container failures use existing caught
exception families; unknown tags remain rejected by the low-level decoder.
- superset/sql/execution/celery_task.py:171 — not affected: this is the
sidecar writer, not a reader.
- superset/sql/execution/executor.py:1198 — not affected: cache hits rebuild
original records without decoding sidecars.
- superset/result_set.py:369 — not affected: synchronous results restore
original nested values without decoding sidecars.
##########
docs/developer_docs/extensions/mcp.md:
##########
@@ -28,6 +28,34 @@ under the License.
Model Context Protocol (MCP) integration allows extensions to register custom
AI agent capabilities that integrate seamlessly with Superset's MCP service.
Extensions can provide both **tools** (executable functions) and **prompts**
(interactive guidance) that AI agents can discover and use.
+## Python SQL query results
Review Comment:
Fixed the remaining section-scope gap in
807fa02847106d703d7ef4be5d66e1294e5fca5d: the Python SQL query results
subsection follows the existing Backend APIs endpoint text and complete
DatasetReferencesAPI example, before Automatic Context Detection.
Regression:
tests/unit_tests/sql/execution/test_async_results.py::test_python_sql_results_documented_under_backend_apis
failed before this commit and passes after it. It explicitly asserts the
/extensions namespace text and DatasetReferencesAPI example precede, and are
outside, the SQL-results subsection. Final two-module run: 408 passed / 2
skipped.
Search: `rg -n 'Python SQL query results|DatasetReferencesAPI|Extension
endpoints are registered' docs/developer_docs/extensions`.
Siblings checked:
- docs/developer_docs/extensions/development.md:209 — fixed in
807fa02847106d703d7ef4be5d66e1294e5fca5d
- docs/developer_docs/extensions/development.md:223 — fixed in
807fa02847106d703d7ef4be5d66e1294e5fca5d
- docs/developer_docs/extensions/development.md:254 — fixed in
807fa02847106d703d7ef4be5d66e1294e5fca5d
- docs/developer_docs/extensions/development.md:286 — not affected:
Automatic Context Detection remains a separate level-three section.
- docs/developer_docs/extensions/mcp.md — not affected: the generic Python
results section remains absent, asserted by the regression test.
- superset-core/src/superset_core/queries/types.py — not affected: no Python
result-handle contract changes in this documentation correction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
