KevinGG commented on a change in pull request #11020: [BEAM-7926] Update Data
Visualization
URL: https://github.com/apache/beam/pull/11020#discussion_r387281998
##########
File path:
sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
##########
@@ -238,31 +322,57 @@ def _display_dive(self, data, update=None):
display(HTML(html))
def _display_overview(self, data, update=None):
+ if (not data.empty and self._include_window_info and
+ all(column in data.columns
+ for column in ('event_time', 'windows', 'pane_info'))):
+ data = data.drop(['event_time', 'windows', 'pane_info'], axis=1)
+
gfsg = GenericFeatureStatisticsGenerator()
proto = gfsg.ProtoFromDataFrames([{'name': 'data', 'table': data}])
protostr = base64.b64encode(proto.SerializeToString()).decode('utf-8')
if update:
script = _OVERVIEW_SCRIPT_TEMPLATE.format(
- display_id=update, protostr=protostr)
+ display_id=update._overview_display_id, protostr=protostr)
display_javascript(Javascript(script))
else:
html = _OVERVIEW_HTML_TEMPLATE.format(
display_id=self._overview_display_id, protostr=protostr)
display(HTML(html))
def _display_dataframe(self, data, update=None):
- if update:
- table_id = 'table_{}'.format(update)
- html = _DATAFRAME_PAGINATION_TEMPLATE.format(
- dataframe_html=data.to_html(notebook=True, table_id=table_id),
- table_id=table_id)
- update_display(HTML(html), display_id=update)
+ table_id = 'table_{}'.format(
+ update._df_display_id if update else self._df_display_id)
+ columns = [{
+ 'title': ''
+ }] + [{
+ 'title': str(column)
+ } for column in data.columns]
+ format_window_info_in_dataframe(data)
+ rows = data.applymap(lambda x: str(x)).to_dict('split')['data']
Review comment:
First, we get all the string `data` from the `split`
[orient](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_dict.html)
of `dataframe.to_dict`.
Now the `rows` is a `list` of `row`s of values.
Each `row` looks like `[column_1_val, column_2_val, ...]`
Then we are going to add datatable column index for the values in each `row`.
The index starts from 1 because we are also going to add a column `0`
later., so we have `{k+1: v}`.
Each `row` now becomes `{1: column_1_val, 2: column_2_val, ...}`
Then we add column `0` (`row[0] = k`) of the datatable with values of int
based index (which will be the default order column just as the original
dataframe).
Each `row` now becomes `{1: column_1_val, 2: column_2_val, ..., 0:
int_index_in_dataframe}`
Then the list of above `row`s get supplied as string in the Javascript to
load the data into the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
