[
https://issues.apache.org/jira/browse/BEAM-7568?focusedWorklogId=437623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437623
]
ASF GitHub Bot logged work on BEAM-7568:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/20 04:15
Start Date: 27/May/20 04:15
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #11823:
URL: https://github.com/apache/beam/pull/11823#discussion_r430836474
##########
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java
##########
@@ -410,14 +416,18 @@ protected WorkItemCommitRequest
persistDirectly(WindmillStateCache.ForKey cache)
return WorkItemCommitRequest.newBuilder().buildPartial();
}
- ByteString.Output stream = ByteString.newOutput();
- if (value != null) {
- coder.encode(value, stream, Coder.Context.OUTER);
+ ByteString encoded = null;
+ if (cachedSize == -1 || modified) {
Review comment:
This logic is really frustrating that we are encoding because we aren't
returning the size of what we read as part of the state future from
WindmillStateReader.
This could be better but this is definitely an improvement to not re-encode
the value needlessly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 437623)
Time Spent: 0.5h (was: 20m)
> Java dataflow harness re-encodes value state cells even if they haven't
> changed
> -------------------------------------------------------------------------------
>
> Key: BEAM-7568
> URL: https://issues.apache.org/jira/browse/BEAM-7568
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.13.0
> Reporter: Steve Niemitz
> Assignee: Steve Niemitz
> Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The java dataflow worker seems to re-encode ValueState cells after every work
> item, even they weren't modified.
> You can see here
> [https://github.com/apache/beam/blob/a71bfda77df36aa1531f01533c372233cfba0dd9/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L413]
> that the value is always encoded (and used to weight the cache entry) even
> if it won't be persisted back to windmill.
> This can have some large performance implications if they values being stored
> are expensive/large to encode, and infrequently modified. Ideally, the
> weight would be also cached, and the value would only need to be modified if
> it was changed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
