[
https://issues.apache.org/jira/browse/BEAM-7568?focusedWorklogId=437619&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437619
]
ASF GitHub Bot logged work on BEAM-7568:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/20 04:14
Start Date: 27/May/20 04:14
Worklog Time Spent: 10m
Work Description: steveniemitz commented on a change in pull request
#11823:
URL: https://github.com/apache/beam/pull/11823#discussion_r430837481
##########
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java
##########
@@ -410,14 +416,18 @@ protected WorkItemCommitRequest
persistDirectly(WindmillStateCache.ForKey cache)
return WorkItemCommitRequest.newBuilder().buildPartial();
}
- ByteString.Output stream = ByteString.newOutput();
- if (value != null) {
- coder.encode(value, stream, Coder.Context.OUTER);
+ ByteString encoded = null;
+ if (cachedSize == -1 || modified) {
Review comment:
agreed, it took me awhile to reason this out w/ the different code paths
that can affect the values here. I'm reasonably sure you don't need the
`cachedSize == -1` branch but, it's been working like this for ~1 year so I
didn't want to tempt fate.
There's also the case where this is the first write, but `valueIsKnown =
false` (because `isNewKey = false`), so we end up with a read that doesn't
return anything.
Thanks for the quick reviews!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 437619)
Time Spent: 20m (was: 10m)
> Java dataflow harness re-encodes value state cells even if they haven't
> changed
> -------------------------------------------------------------------------------
>
> Key: BEAM-7568
> URL: https://issues.apache.org/jira/browse/BEAM-7568
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.13.0
> Reporter: Steve Niemitz
> Assignee: Steve Niemitz
> Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The java dataflow worker seems to re-encode ValueState cells after every work
> item, even they weren't modified.
> You can see here
> [https://github.com/apache/beam/blob/a71bfda77df36aa1531f01533c372233cfba0dd9/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L413]
> that the value is always encoded (and used to weight the cache entry) even
> if it won't be persisted back to windmill.
> This can have some large performance implications if they values being stored
> are expensive/large to encode, and infrequently modified. Ideally, the
> weight would be also cached, and the value would only need to be modified if
> it was changed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
