sadpandajoe commented on code in PR #45348:
URL: https://github.com/apache/superset/pull/45348#discussion_r4288802779
##########
superset/sql/execution/celery_task.py:
##########
@@ -298,33 +307,31 @@ def _store_results_in_backend(
def _serialize_result_set(
result_set: SupersetResultSet,
-) -> tuple[bytes | list[Any], list[Any]]:
+) -> tuple[bytes | str, list[Any]]:
"""
Serialize result set based on RESULTS_BACKEND_USE_MSGPACK config.
- When msgpack is enabled, uses Apache Arrow IPC format for efficiency.
- Otherwise, falls back to JSON-serializable records.
+ Both transports use Apache Arrow IPC to retain the result types. JSON
+ wraps the IPC bytes in Base64; MessagePack carries them directly.
:param result_set: Query result set to serialize
:returns: Tuple of (serialized_data, columns)
"""
from superset import results_backend_use_msgpack
- from superset.dataframe import df_to_records
- if results_backend_use_msgpack:
- if has_app_context():
- stats_logger = app.config["STATS_LOGGER"]
- with stats_timing(
- "sqllab.query.results_backend_pa_serialization", stats_logger
- ):
- data: bytes | list[Any] = write_ipc_buffer(
- result_set.pa_table
- ).to_pybytes()
- else:
- data = write_ipc_buffer(result_set.pa_table).to_pybytes()
+ if has_app_context():
+ stats_logger = app.config["STATS_LOGGER"]
+ with stats_timing(
+ "sqllab.query.results_backend_pa_serialization", stats_logger
+ ):
+ arrow_data = write_ipc_buffer(result_set.pa_table).to_pybytes()
else:
- df = result_set.to_pandas_df()
- data = df_to_records(df, convert_decimals=True) or []
+ arrow_data = write_ipc_buffer(result_set.pa_table).to_pybytes()
+ data = (
+ arrow_data
+ if results_backend_use_msgpack
+ else base64.b64encode(arrow_data).decode("ascii")
Review Comment:
With `RESULTS_BACKEND_USE_MSGPACK = False`, the payload is now
Base64-wrapped Arrow IPC, which is about 33% larger than the raw bytes.
`_store_results_in_backend` checks `SQLLAB_PAYLOAD_MAX_MB` against this
inflated pre-compression size. A text-heavy result that was accepted before
(e.g. a single ~8 MiB string column serialized as JSON records, ~8.0 MiB) now
measures ~10.7 MiB and fails with `RESULT_TOO_LARGE_ERROR` under a 10 MB limit.
Was this size increase for JSON-mode deployments intentional, or should the
limit check account for the encoding overhead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
