[
https://issues.apache.org/jira/browse/BEAM-11707?focusedWorklogId=557856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557856
]
ASF GitHub Bot logged work on BEAM-11707:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Feb/21 10:57
Start Date: 25/Feb/21 10:57
Worklog Time Spent: 10m
Work Description: scwhittle commented on pull request #13862:
URL: https://github.com/apache/beam/pull/13862#issuecomment-785808463
I'm not exactly sure why this isn't triggering already but I believe the
issue can occur with the IdTracker possibly writing empty maps here:
https://github.com/apache/beam/blob/3bb232fb098700de408f574585dfe74bbaff7230/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L767
Then if a read is performed and it is not cached, there is a non-null
encoded value (mapcoder encodes size) and then an EmptyMap is coming from
MapCoder:
https://github.com/apache/beam/blob/7c43ab6a8df9b23caa7321fddff9a032a71908f6/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java#L111
Since the creation of the maps only looks if null is returned, the EmptyMap
is possibly inserted into
https://github.com/apache/beam/blob/3bb232fb098700de408f574585dfe74bbaff7230/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L702
I think the IdTracker should delete the values if the maps are empty instead
of caching empty maps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 557856)
Time Spent: 2h 20m (was: 2h 10m)
> Optimize WindmillStateCache CPU usage
> -------------------------------------
>
> Key: BEAM-11707
> URL: https://issues.apache.org/jira/browse/BEAM-11707
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> From profiling nexmark Query11 which has many unique tags per key, I observed
> that the WindmillStateCache cpu usage was 6% of CPU.
> The usage appears to be due to the invalidation set maintenance as well as
> many reads/inserts.
> The invalidation set is maintained so that if a key encounters an error
> processing or the cache token changes, we can invalidate all the entries for
> a key. Currently this is done by removing all entries for the key from the
> cache. Another alternative which appears much more CPU efficient is to
> instead leave the entries in the cache but make them unreachable. This can
> be done by having a per-key object that uses object equality as part of the
> cache lookup. Then to discard entries for the key, we start using a new
> per-key object. Cleanup of per-key objects can be done with a weak reference
> map.
> Another cost to the cache is that objects are grouped by window so that they
> are kept/evicted all at once. However currently when reading items into the
> cache, we fetch the window set and then lookup each tag in it. This could be
> cached for the key to avoid multiple cache lookups. Similarly for putting
> objects we lookup and insert each tag separately and then update the cache to
> update the weight for the per-window set. This could be done once after all
> updates for the window have been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
