[
https://issues.apache.org/jira/browse/TEZ-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545450#comment-16545450
]
Jonathan Eagles commented on TEZ-3942:
--------------------------------------
Thanks, [~nishantdash] for taking up this work. Haven't had much time to look
at this recently. Please assign this jira to yourself and feel free to use any
or all patches to base your work on.
> RPC getTask writable optimization invalid in hadoop 2.8+
> --------------------------------------------------------
>
> Key: TEZ-3942
> URL: https://issues.apache.org/jira/browse/TEZ-3942
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Major
> Attachments: TEZ-3942.001.patch, TEZ-3942.bench.patch,
> TEZ-3942.test.patch
>
>
> TEZ-3140 added an optimization to improve performance of RPC writable.
> HADOOP-13426 added in hadoop 2.8 has invalidated the assumption of the added
> optimization by changing the underlying output buffer.
> {noformat}
> "IPC Server handler 25 on 35274" #85 daemon prio=5 os_prio=0
> tid=0x00000000022c0000 nid=0x1b40f runnable [0x00002ba1a6627000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> - locked <0x000000072fe9ac68> (a
> org.apache.hadoop.ipc.ResponseBuffer$FramedBuffer)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> - locked <0x000000072fe9ac48> (a org.apache.hadoop.ipc.ResponseBuffer)
> at
> org.apache.tez.dag.api.EntityDescriptor.write(EntityDescriptor.java:121)
> at org.apache.tez.runtime.api.impl.InputSpec.write(InputSpec.java:66)
> at org.apache.tez.runtime.api.impl.TaskSpec.write(TaskSpec.java:174)
> at org.apache.tez.common.ContainerTask.write(ContainerTask.java:77)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:202)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:128)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:82)
> at
> org.apache.hadoop.ipc.RpcWritable$WritableWrapper.writeTo(RpcWritable.java:75)
> at
> org.apache.hadoop.ipc.Server.setupResponseForWritable(Server.java:2807)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2792)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2766)
> at org.apache.hadoop.ipc.Server.access$100(Server.java:138)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:905)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:810)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1949)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2523)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
