So. After some internal discussions with Anoop, here is a summary of the situation.
An hbase-0.94.0 jar file was included in the MR job client file. But also, this MR client file was stored into the Master lib directory. And only in the master and the RS hosted on the same host. Not in any of the other RS nodes. Removing this file from the client, recompiling HBase 0.94.12-SNAPSHOT and redeploying everything fixed the issue. What does it mean. I think there is something between Hbase 0.94.0 and HBase 0.94.12 which is not compatible. It's not related to the TableSplit class. This class is like that since 0.94.5. It's most probably related to a more recent modification which is breaking the compatibility between HBase 0.94.0 and last HBase 0.94 branch. The MR job on my server was running for months without any issue, with this 0.94.0 jar included. Which mean the compatibility has been broken recently. Something like between 0.94.10 and 0.94.12 (I guess). Now, even if 0.94.12 is not compatible with HBase version < 0.94.5. Is this something we want to investigate further? Or 0.94.5 versions are already too old and if there is some break of the compatibility we can live with that? JM 2013/9/8 Jean-Marc Spaggiari <[email protected]> > FYI, > > I just faced the exact same exception with version 0.94.12SNAPSHOT... All > tasks failed with the same exception > > $ bin/hbase hbck > Version: 0.94.12-SNAPSHOT > .... > 0 inconsistencies detected. > Status: OK > > I will update, rebuild and retry tomorrow morning... > > java.lang.NegativeArraySizeException: -1 > at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148) > > at > org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) > at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:396) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:728) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > > > 2013/9/4 Jean-Marc Spaggiari <[email protected]> > >> That's interesting. Can you please tell u sa bit more about the context? >> What kind of table are you using for you job? Is it an empty one? Anything >> special? Have yoy run HBCK? >> >> Also, can you please double check your HBase version? I looked at the >> code for 0.94.9 and it doesn't seems to be in sync with the stack trace you >> have provided. readFields is calling readByteArray many times, so we need >> to figure which one exactly failed. >> >> Thanks, >> >> JM >> >> >> 2013/9/4 Job Thomas <[email protected]> >> >>> I am using Hbase 0.94.9 >>> >>> >>> >>> Best Regards, >>> Job M Thomas >>> >>> ________________________________ >>> >>> From: Job Thomas [mailto:[email protected]] >>> Sent: Wed 9/4/2013 11:08 AM >>> To: [email protected] >>> Subject: java.lang.NegativeArraySizeException: -1 in hbase >>> >>> >>> >>> >>> Hi All, >>> >>> I am getting the following error while runnig a simple hbase-mapreduce >>> job to get one table data and write back to another table . >>> >>> 13/09/04 10:24:03 INFO mapred.JobClient: map 0% reduce 0% >>> >>> 13/09/04 10:24:22 INFO mapred.JobClient: Task Id : >>> attempt_201309031846_0023_m_000000_0, Status : FAILED >>> >>> java.lang.NegativeArraySizeException: -1 >>> >>> at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:147) >>> >>> at >>> org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133) >>> >>> at >>> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) >>> >>> at >>> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) >>> >>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:396) >>> >>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:728) >>> >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>> >>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>> >>> at java.security.AccessController.doPrivileged(Native Method) >>> >>> at javax.security.auth.Subject.doAs(Subject.java:416) >>> >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>> >>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>> >>> >>> >>> >>> Best Regards, >>> Job M Thomas >>> Suntec Business Solution >>> >>> >>> >> >
