Dmytro Kozhevin created BEAM-11629:
--------------------------------------
Summary: Optimize the cache storage for InteractiveRunner
Key: BEAM-11629
URL: https://issues.apache.org/jira/browse/BEAM-11629
Project: Beam
Issue Type: Improvement
Components: runner-py-interactive
Reporter: Dmytro Kozhevin
Currently InteractiveRunner wraps every record of the cached PCollection into
WindowedValue. There is 2 problems about this:
1) The windowing information is unnecessary for the batch-mode runs (everything
is in the same global window).
2) Since the cache is stored as text, we pickle the WindowedValue, which adds
~500 bytes of data to every record (e.g. a cache of just 1000000 integers would
take ~500MB instead of ~4MB).
These issues significantly slow down the interactive runs for data with lots of
small rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
