Both clusters were are running the same version of hbase, and hadoop, with
matching compile dates and checksums. Also, `hbase hbck` showed no
inconsistencies in the source hbase instance.
Stack trace:
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of
TaskAttempt attempt_1470780584817_0448_m_000346_3 is : 0.17224467
2016-11-20 06:06:01,200 FATAL [IPC Server handler 28 on 39202]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1470780584817_0448_m_000346_3 - exited :
org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0710356613704525352\x00\x80\x00\x00\x00\x85B\xBCU/IN:crdAt/1462937192582/Put/vlen=12/seqid=0
doesn't match the original one
\x00\x00\x0710156613704525352\x00\x80\x00\x00\x00\x85B\xBCU
at org.apache.hadoop.hbase.client.Put.add(Put.java:321)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.addPutToKv(Import.java:215)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.processKV(Import.java:195)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:158)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:143)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:126)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-11-20 06:06:01,200 INFO [IPC Server handler 28 on 39202]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
from attempt_1470780584817_0448_m_000346_3: Error:
org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0710356613704525352\x00\x80\x00\x00\x00\x85B\xBCU/IN:crdAt/1462937192582/Put/vlen=12/seqid=0
doesn't match the original one
\x00\x00\x0710156613704525352\x00\x80\x00\x00\x00\x85B\xBCU
at org.apache.hadoop.hbase.client.Put.add(Put.java:321)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.addPutToKv(Import.java:215)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.processKV(Import.java:195)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:158)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:143)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:126)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-11-20 06:06:01,200 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
Diagnostics report from attempt_1470780584817_0448_m_000346_3: Error:
org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0710356613704525352\x00\x80\x00\x00\x00\x85B\xBCU/IN:crdAt/1462937192582/Put/vlen=12/seqid=0
doesn't match the original one
\x00\x00\x0710156613704525352\x00\x80\x00\x00\x00\x85B\xBCU
at org.apache.hadoop.hbase.client.Put.add(Put.java:321)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.addPutToKv(Import.java:215)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.processKV(Import.java:195)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:158)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:143)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:126)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
On Mon, Nov 21, 2016 at 4:58 PM, Ted Yu <[email protected]> wrote:
> Can you give the whole stack trace for WrongRowIOException ?
>
> Was the cluster running Export using the same version of hbase (
> 1.0.0-cdh5.5.2) ?
>
> Thanks
>
> On Mon, Nov 21, 2016 at 4:35 PM, Julian Jaffe <[email protected]>
> wrote:
>
> > Hbase Version: 1.0.0-cdh5.5.2
> >
> > We're importing the data using `hbase
> > org.apache.hadoop.hbase.mapreduce.Import 'table.name' /path/to/backup`
> > (The data was exported from an HBase instance on another cluster using
> > `hbase org.apache.hadoop.hbase.mapreduce.Export` and then distcp'd
> between
> > the clusters).
> >
> > On Mon, Nov 21, 2016 at 4:29 PM, Ted Yu <[email protected]> wrote:
> >
> > > I did a quick search - there was no relevant JIRA or discussion thread
> at
> > > first glance.
> > >
> > > Which hbase release are you using ?
> > >
> > > How do you import the data ?
> > >
> > > More details would be helpful.
> > >
> > > Thanks
> > >
> > > On Mon, Nov 21, 2016 at 2:48 PM, Julian Jaffe <
> [email protected]>
> > > wrote:
> > >
> > > > When importing data into a fresh HBase instance, after some time the
> > > import
> > > > throws the following exception:
> > > >
> > > > Error: org.apache.hadoop.hbase.client.WrongRowIOException: The row
> in
> > > > \x00\x00\x0767341283611_10153807927108612\x00\x80\x00\
> > > > x00\x00\x84)L\xA7/IN:nme/1461847340445/Put/vlen=42/seqid=0
> > > > doesn't match the original one
> > > > \x00\x00\x0767341283611_10153805927108612\x00\x80\x00\
> > x00\x00\x84)L\xA7
> > > >
> > > > (The non-matching row differs on different runs).
> > > >
> > > > If the import is allowed to run to completion, the row count of the
> > data
> > > > imported is less than the row count of the source data.
> > > >
> > > > Googling for this error only turns up the source code that generates
> > the
> > > > error, so it doesn't seem to be a common problem.
> > > >
> > > > Can anyone provide any guidance?
> > > >
> > > > Julian Jaffe
> > > >
> > >
> >
>
