[
https://issues.apache.org/jira/browse/BEAM-5933?focusedWorklogId=186235&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186235
]
ASF GitHub Bot logged work on BEAM-5933:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jan/19 09:12
Start Date: 17/Jan/19 09:12
Worklog Time Spent: 10m
Work Description: janotav commented on issue #6909: BEAM-5933: avoid
memory allocation in hashCode call
URL: https://github.com/apache/beam/pull/6909#issuecomment-455097247
I have encountered this as a part of larger investigation into the
performance aspects of our pipelines. The invocations were frequent enough to
stand out in the memory profiler. I found it interesting that hash calculations
needs to allocate memory and considering the high number of occurrences I
thought this would be worth "fixing". Unfortunately I was not aware of this
"hidden contract" that assumes particular hash implementation is necessary.
Btw the overall improvement we were able to achieve in our pipelines was
about 15% (7mins down to 6mins), but this can be predominantly (if not
completely) attributed to removing the beam metrics collection in the
DirectRunner. The load that beam metrics (in DirectRunner) put on the heap is
massive (that extra minute basically went to GC activity), however, knowing
that anything performance related is implicitly considered a non-issue I didn't
report it. This high resource consumption is probably why FlinkRunner
eventually added the option to disable metrics collection.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 186235)
Time Spent: 1h (was: 50m)
> PCollectionViews$SimplePCollectionView.hashCode allocates memory
> ----------------------------------------------------------------
>
> Key: BEAM-5933
> URL: https://issues.apache.org/jira/browse/BEAM-5933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Affects Versions: 2.8.0
> Reporter: Vojtech Janota
> Assignee: Vojtech Janota
> Priority: Trivial
> Fix For: 2.9.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I'm currently profiling memory consumption of our Beam pipeline and have
> noticed that
>
> org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.hashCode()
> makes noticeable heap allocations. The implementation is:
> return Objects.hash(tag);
> That itself translates to:
> return Arrays.hashCode(values);
> Which performs implicit array creation in order to call:
> public static int Arrays.hashCode(Object a[]);
> Instead of the helper call, doing simple:
> tag.hashCode();
> Seems more appropriate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
