Tobias Schlottke created PIG-3231:
-------------------------------------
Summary: Problems with pig (TRUNK, 0.11) after upgrading to
CDH4.2(yarn) using avro input
Key: PIG-3231
URL: https://issues.apache.org/jira/browse/PIG-3231
Project: Pig
Issue Type: Bug
Affects Versions: 0.11
Environment: CDH4.2, yarn, avro
Reporter: Tobias Schlottke
Hi there,
we've got a strange issue after switching to a new cluster with cdh4.2 (from
cdh3):
Pig seems to create temporary avro files for its map reduce jobs, which it
either deletes or never creates.
Pig fails with the "no error returned by hadoop"-message, but in nn-logs I
found something interesting.
The actual exception from nn-log is:
a
{code}
2013-03-01 12:59:30,858 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
192.168.1.28:37814: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/user/metrigo/event_logger/compact_log/2013/01/14/_temporary/1/_temporary/attempt_1362133122980_0017_m_000007_0/part-m-00007.avro
File does not exist. Holder
DFSClient_attempt_1362133122980_0017_m_000007_0_1992466008_1 does not have any
open files.
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/user/metrigo/event_logger/compact_log/2013/01/14/_temporary/1/_temporary/attempt_1362133122980_0017_m_000007_0/part-m-00007.avro
File does not exist. Holder
DFSClient_attempt_1362133122980_0017_m_000007_0_1992466008_1 does not have any
open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2396)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2387)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2183)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:481)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1695)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1691)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1689)
{code}
Please note that we're analyzing a bunch of files (~200 files, we're using glob
matchers), some of them are small.
We made it work once without the small files.
Any Idea on how to find the reason for this?
Best,
Tobias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
