mistercrunch commented on a change in pull request #8069: [SQL Lab] Async query
results serialization with MessagePack and PyArrow
URL:
https://github.com/apache/incubator-superset/pull/8069#discussion_r315505524
##########
File path: superset/sql_lab.py
##########
@@ -224,6 +234,38 @@ def execute_sql_statement(sql_statement, query,
user_name, session, cursor):
return dataframe.SupersetDataFrame(data, cursor_description,
db_engine_spec)
+def _serialize_payload(payload, use_msgpack=False):
+ logging.debug("Serializing to msgpack: {}".format(use_msgpack))
+ if use_msgpack:
+ return msgpack.dumps(payload, default=json_iso_dttm_ser,
use_bin_type=True)
+ else:
+ return json.dumps(payload, default=json_iso_dttm_ser, ignore_nan=True)
+
+
+def _serialize_and_expand_data(cdf, db_engine_spec, use_msgpack=False):
+ selected_columns = cdf.columns or []
+
+ if use_msgpack:
+ with stats_timing(
+ "sqllab.query.results_backend_pa_serialization", stats_logger
+ ):
+ data = (
+ pa.default_serialization_context()
+ .serialize(cdf.raw_df)
+ .to_buffer()
+ .to_pybytes()
+ )
+ # expand when loading data from results backend
+ all_columns, expanded_columns = (selected_columns, [])
+ else:
+ data = cdf.data or []
+ all_columns, data, expanded_columns = db_engine_spec.expand_data(
Review comment:
Mmmmh now I'm catching the complexity of expand_data which is applied on
deser with webpack and on serialization with json... This makes things a bit
twisted and unintuitive. I get why it's that way, but makes me think we should
just force the msgpak + arrow approach which is superior. I'm thinking that a
try-catch block wrapping deserialization and reraise a proper message `raise
SupersetDeserException("Deserializer mismatch, please re-run your query")`
should be sufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
