[
https://issues.apache.org/jira/browse/CRUNCH-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902941#comment-13902941
]
Chao Shi commented on CRUNCH-346:
---------------------------------
I think ByteBuffer is mutable and it directly reference to the buffer
BytesWritable owns. We can also make PTypes#protos use NoOpDeepCopier as
protobuf message is immutable and all internal data structures are copied from
the original buffer (e.g ByteString). I'm not sure about thrift message, but it
seems to be mutable.
> Don't deep-copy immutable Writable PTypes
> -----------------------------------------
>
> Key: CRUNCH-346
> URL: https://issues.apache.org/jira/browse/CRUNCH-346
> Project: Crunch
> Issue Type: Improvement
> Reporter: Chao Shi
> Attachments: CRUNCH-346.patch
>
>
> I found getDetachedValue() appears quite often when jstack on one of my
> pipeline. A piece of stacktrace is shown below. In the pipeline, most of
> types we used are immutable (e.g. java primitives, strings, protobuf). I
> think we can avoid deep-copy overhead here.
> "main" prio=10 tid=0x00007f0de801d800 nid=0x7ef runnable [0x00007f0dee66c000]
> java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:52)
> at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:46)
> at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:63)
> at
> org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:36)
> at
> org.apache.crunch.types.writable.WritableType.getDetachedValue(WritableType.java:125)
> at
> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:54)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
