[
https://issues.apache.org/jira/browse/BEAM-11629?focusedWorklogId=542467&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-542467
]
ASF GitHub Bot logged work on BEAM-11629:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jan/21 22:19
Start Date: 26/Jan/21 22:19
Worklog Time Spent: 10m
Work Description: dmkozh commented on a change in pull request #13739:
URL: https://github.com/apache/beam/pull/13739#discussion_r564872946
##########
File path: sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
##########
@@ -655,9 +660,13 @@ def process(
t=beam.DoFn.TimestampParam):
yield test_stream.WindowedValueHolder(WindowedValue(e, t, [w], p))
+ extended_target = pcoll
+ if self._is_streaming_pcollection(pcoll):
Review comment:
This is why I was interested in figuring out whether pcollection is not
streaming, so that windowing wouldn't matter. I don't fully understand the
code, but the comment to Reify suggests that wrapping is also needed outside
the visualization ('When it detects one, it puts the element into the correct
window then emits the value to downstream transforms').
It seems nice to have this just depend on the flag, but are we sure it would
work pcollections with non-trivial windowing? I also can imagine a case where
ib.collect is called with `include_window_info=False` at first and then with
`include_window_info=True`, which probably would result in an error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 542467)
Time Spent: 1h 20m (was: 1h 10m)
> Optimize the cache storage for InteractiveRunner
> ------------------------------------------------
>
> Key: BEAM-11629
> URL: https://issues.apache.org/jira/browse/BEAM-11629
> Project: Beam
> Issue Type: Improvement
> Components: runner-py-interactive
> Reporter: Dmytro Kozhevin
> Assignee: Dmytro Kozhevin
> Priority: P2
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently InteractiveRunner wraps every record of the cached PCollection into
> WindowedValue. There is 2 problems about this:
> 1) The windowing information is unnecessary for the batch-mode runs
> (everything is in the same global window).
> 2) Since the cache is stored as text, we pickle the WindowedValue, which adds
> ~500 bytes of data to every record (e.g. a cache of just 1000000 integers
> would take ~500MB instead of ~4MB).
> These issues significantly slow down the interactive runs for data with lots
> of small rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
