[
https://issues.apache.org/jira/browse/BEAM-5933?focusedWorklogId=187159&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187159
]
ASF GitHub Bot logged work on BEAM-5933:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Jan/19 23:31
Start Date: 18/Jan/19 23:31
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #6909: BEAM-5933: avoid
memory allocation in hashCode call
URL: https://github.com/apache/beam/pull/6909#issuecomment-455720806
@janotav you are quite right that this hidden contract is very suspicious. I
have looked into the type hierarchy to investigate.
The issue is that there are two desires in conflict: (1) a runner can
deserialize a protobuf PCollectionView using just the tag, into whatever its
runner-specific representation and (2) you can use PCollectionView as a key to
retrieve values. Together, these force any subclass of PCollectionView should
be equal (and equal hashcode) if their tags are equal, since runner's create
proxy views or whatever. IMO this contract is broken, since the same tag but
different `ViewFn` should not ever be equal.
But if you want to gain the performance back, I bet you can roll forward and
also just change here to match:
https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RunnerPCollectionView.java#L108
Even better would be to port things to use the tag as the key into any
implementation map.
There is not even equals and hashcode on these subclasses in the Dataflow
worker so I think that implies the tag is used directly:
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowPortabilityPCollectionView.java
and
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FetchAndFilterStreamingSideInputsOperation.java#L99
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 187159)
Time Spent: 1h 40m (was: 1.5h)
> PCollectionViews$SimplePCollectionView.hashCode allocates memory
> ----------------------------------------------------------------
>
> Key: BEAM-5933
> URL: https://issues.apache.org/jira/browse/BEAM-5933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Affects Versions: 2.8.0
> Reporter: Vojtech Janota
> Assignee: Vojtech Janota
> Priority: Trivial
> Fix For: 2.9.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> I'm currently profiling memory consumption of our Beam pipeline and have
> noticed that
>
> org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.hashCode()
> makes noticeable heap allocations. The implementation is:
> return Objects.hash(tag);
> That itself translates to:
> return Arrays.hashCode(values);
> Which performs implicit array creation in order to call:
> public static int Arrays.hashCode(Object a[]);
> Instead of the helper call, doing simple:
> tag.hashCode();
> Seems more appropriate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
