Thanks a lot for the input.
From my debug log below, the problem should be because
ipc.maximum.data.length is too small.
2015-05-26 10:10:48,376 INFO [Socket Reader #1 for port 52017]
ipc.Server: Socket Reader #1 for port 52017: readAndProcess from client
10.10.255.241 threw exception [java.io.IOException: Requested data
length 166822274 is longer than maximum configured RPC length 67108864.
RPC came from 10.10.255.241]
java.io.IOException: Requested data length 166822274 is longer than
maximum configured RPC length 67108864. RPC came from 10.10.255.241
at
org.apache.hadoop.ipc.Server$Connection.checkDataLength(Server.java:1459)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1521)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:762)
at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:636)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607)
However I am curious why the data request size 166822274 is bigger than
my HDFS max block size (128 MB). Do you have an idea?
BR,
Patcharee
On 22. mai 2015 19:58, Johannes Zillmann wrote:
Hey Patcharee,
i sometimes faced that in case the DAG or the properties/object it contains
become quite big. Pumping up ipc.maximum.data.length, e.g.
ipc.maximum.data.length=134217728 usually helped!
best
Johannes
On 22 May 2015, at 19:49, Rohini Palaniswamy <[email protected]> wrote:
Can you check if the hadoop version on your cluster and the version of hadoop
jars on your pig classpath same? Also is the tez jars on pig classpath and the
tez jars installed in hdfs are of the same version?
-Rohini
On Fri, May 22, 2015 at 10:26 AM, Hitesh Shah <[email protected]> wrote:
Hello Patcharee
Could you start with sending a mail to users@pig to see if they have come
across this issue first? Also, can you check the application master logs to see
if there are any errors ( might be useful to enable DEBUG level logging to get
more information )?
thanks
— Hitesh
On May 22, 2015, at 5:50 AM, patcharee <[email protected]> wrote:
Hi,
I ran a pig script on tez and got the EOFException. Check at
http://wiki.apache.org/hadoop/EOFException I have no ideas at all how I can fix
it. However I did not get the exception when I executed this pig script on MR.
I am using HadoopVersion: 2.6.0.2.2.4.2-2, PigVersion: 0.14.0.2.2.4.2-2,
TezVersion: 0.5.2.2.2.4.2-2
I will appreciate any suggestions. Thanks.
2015-05-22 14:44:13,638 [PigTezLauncher-0] ERROR
org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG -
Application id: application_1432237888868_0133
org.apache.tez.dag.api.TezException: com.google.protobuf.ServiceException: java.io.EOFException:
End of File Exception between local host is: "compute-10-0.local/10.10.255.241";
destination host is: "compute-10-3.local":47111; : java.io.EOFException; For more details
see: http://wiki.apache.org/hadoop/EOFException
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:415)
at org.apache.tez.client.TezClient.submitDAG(TezClient.java:351)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:162)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.protobuf.ServiceException: java.io.EOFException: End of File Exception
between local host is: "compute-10-0.local/10.10.255.241"; destination host is:
"compute-10-3.local":47111; : java.io.EOFException; For more details see:
http://wiki.apache.org/hadoop/EOFException
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:246)
at com.sun.proxy.$Proxy31.submitDAG(Unknown Source)
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:408)
... 8 more
Caused by: java.io.EOFException: End of File Exception between local host is:
"compute-10-0.local/10.10.255.241"; destination host is:
"compute-10-3.local":47111; : java.io.EOFException; For more details see:
http://wiki.apache.org/hadoop/EOFException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
at org.apache.hadoop.ipc.Client.call(Client.java:1400)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
... 10 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:967)
