Hyukjin Kwon created SPARK-61110:
------------------------------------
Summary: Add DataFrame._repr_html_, the HTML table of eager
evaluation in notebooks, in the Python client as PySpark does
Key: SPARK-61110
URL: https://issues.apache.org/jira/browse/SPARK-61110
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{DataFrame._repr_html_}}, which Jupyter calls to display a
DataFrame, works as follows:
* It reads {{spark.sql.repl.eagerEval.enabled}}, {{maxNumRows}} and
{{truncate}}.
* If eager evaluation is enabled, it returns the HTML table of the DataFrame.
The server formats the table, of an {{HtmlString}} relation.
* Otherwise it returns None.
* Once it has been called, {{__repr__}} of the DataFrame is its
{{DataFrame[...]}} string, so a notebook does not show the table twice.
The Python client lacked {{_repr_html_}}. Attribute access took it as a column,
so {{df._repr_html_()}} raised a {{TypeError}} ("'Column' object is not
callable"). A notebook with eager evaluation showed the text table of
{{__repr__}} instead of the HTML table.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
