[
https://issues.apache.org/jira/browse/CRUNCH-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903374#comment-13903374
]
Gabriel Reid commented on CRUNCH-346:
-------------------------------------
Thanks for catching the ByteBuffer issue [~stepinto].
About the PTypes#protos, agreed that the no-op deep copier should be used. The
only issue is the API with which it should be done. Right now the
NoOpDeepCopier is used for all derived types (i.e. everything created within
PTypes), but that needs to be changed.
I'm was thinking about the possibility of adding a "derivedImmutable" method
(in addition to the normal "derived") method to allow creating PTypes for
derived types that are immutable. Any thoughts on that?
> Don't deep-copy immutable Writable PTypes
> -----------------------------------------
>
> Key: CRUNCH-346
> URL: https://issues.apache.org/jira/browse/CRUNCH-346
> Project: Crunch
> Issue Type: Improvement
> Reporter: Chao Shi
> Attachments: CRUNCH-346.patch
>
>
> I found getDetachedValue() appears quite often when jstack on one of my
> pipeline. A piece of stacktrace is shown below. In the pipeline, most of
> types we used are immutable (e.g. java primitives, strings, protobuf). I
> think we can avoid deep-copy overhead here.
> "main" prio=10 tid=0x00007f0de801d800 nid=0x7ef runnable [0x00007f0dee66c000]
> java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:52)
> at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:46)
> at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:63)
> at
> org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:36)
> at
> org.apache.crunch.types.writable.WritableType.getDetachedValue(WritableType.java:125)
> at
> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:54)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
