[
https://issues.apache.org/jira/browse/BEAM-7568?focusedWorklogId=438328&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438328
]
ASF GitHub Bot logged work on BEAM-7568:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/May/20 16:17
Start Date: 28/May/20 16:17
Worklog Time Spent: 10m
Work Description: lukecwik merged pull request #11823:
URL: https://github.com/apache/beam/pull/11823
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 438328)
Time Spent: 50m (was: 40m)
> Java dataflow harness re-encodes value state cells even if they haven't
> changed
> -------------------------------------------------------------------------------
>
> Key: BEAM-7568
> URL: https://issues.apache.org/jira/browse/BEAM-7568
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.13.0
> Reporter: Steve Niemitz
> Assignee: Steve Niemitz
> Priority: P2
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The java dataflow worker seems to re-encode ValueState cells after every work
> item, even they weren't modified.
> You can see here
> [https://github.com/apache/beam/blob/a71bfda77df36aa1531f01533c372233cfba0dd9/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L413]
> that the value is always encoded (and used to weight the cache entry) even
> if it won't be persisted back to windmill.
> This can have some large performance implications if they values being stored
> are expensive/large to encode, and infrequently modified. Ideally, the
> weight would be also cached, and the value would only need to be modified if
> it was changed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
