[
https://issues.apache.org/jira/browse/BEAM-11629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416258#comment-17416258
]
Beam JIRA Bot commented on BEAM-11629:
--------------------------------------
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Optimize the cache storage for InteractiveRunner
> ------------------------------------------------
>
> Key: BEAM-11629
> URL: https://issues.apache.org/jira/browse/BEAM-11629
> Project: Beam
> Issue Type: Improvement
> Components: runner-py-interactive
> Reporter: Dmytro Kozhevin
> Priority: P2
> Labels: stale-P2
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently InteractiveRunner wraps every record of the cached PCollection into
> WindowedValue. There is 2 problems about this:
> 1) The windowing information is unnecessary for the batch-mode runs
> (everything is in the same global window).
> 2) Since the cache is stored as text, we pickle the WindowedValue, which adds
> ~500 bytes of data to every record (e.g. a cache of just 1000000 integers
> would take ~500MB instead of ~4MB).
> These issues significantly slow down the interactive runs for data with lots
> of small rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
