[
https://issues.apache.org/jira/browse/HBASE-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736179#comment-13736179
]
Aleksandr Shulman commented on HBASE-8619:
------------------------------------------
I'm seeing this occur in my testing. It's a race condition so it's hard to
repro, unfortunately.
The latest stack trace where I saw it looked like this:
{code} Exception in thread "main"
java.lang.reflect.UndeclaredThrowableException
20:30:00 at $Proxy7.getHTableDescriptors(Unknown Source)
20:30:00 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1863)
20:30:00 at
org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:440)
20:30:00 at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureCompression(HFileOutputFormat.java:458)
20:30:00 at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:375)
20:30:00 at
org.apache.hadoop.hbase.mapreduce.ImportTsv.createSubmittableJob(ImportTsv.java:280)
20:30:00 at
org.apache.hadoop.hbase.mapreduce.ImportTsv.main(ImportTsv.java:424)
20:30:00 Caused by: java.io.IOException: Call to
hbase-rolling-6.ent.cloudera.com/10.20.186.99:22001 failed on local exception:
java.io.IOException: Connection reset by peer
20:30:00 at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1033)
20:30:00 at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1002)
20:30:00 at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
20:30:00 ... 7 more
20:30:00 Caused by: java.io.IOException: Connection reset by peer
20:30:00 at sun.nio.ch.FileDispatcher.read0(Native Method)
20:30:00 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
20:30:00 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
20:30:00 at sun.nio.ch.IOUtil.read(IOUtil.java:171)
20:30:00 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
20:30:00 at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:56)
20:30:00 at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
20:30:00 at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
20:30:00 at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
20:30:00 at java.io.FilterInputStream.read(FilterInputStream.java:116)
20:30:00 at java.io.FilterInputStream.read(FilterInputStream.java:116)
20:30:00 at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:373)
20:30:00 at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
20:30:00 at
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
20:30:00 at java.io.DataInputStream.readInt(DataInputStream.java:370)
20:30:00 at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
20:30:00 at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
{code}
> [HBase Client]: Improve client to retry if master is down when requesting
> getHTableDescriptor
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-8619
> URL: https://issues.apache.org/jira/browse/HBASE-8619
> Project: HBase
> Issue Type: Bug
> Components: Client
> Affects Versions: 0.94.8
> Reporter: Aleksandr Shulman
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.94.12
>
>
> In our rolling upgrade testing, running ImportTsv failed in the job
> submission phase when the master was down. This was when the master was
> failing over to the backup master. In this case, a retry would have been
> helpful and made sure that the job would get submitted.
> A good solution would be to refresh the master information before placing the
> call to getHTableDescriptor.
> Command:
> {code} sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c1,c2,c3
> -Dimporttsv.bulk.output=/user/hbase/storeFiles2_2/import2_table1369439156
> import2_table1369439156 /user/hbase/tsv2{code}
> Here is the stack trace:
> {code} 13/05/24 16:55:49 INFO compress.CodecPool: Got brand-new compressor
> [.deflate]
> 16:45:44 Exception in thread "main"
> java.lang.reflect.UndeclaredThrowableException
> 16:45:44 at $Proxy7.getHTableDescriptors(Unknown Source)
> 16:45:44 at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1861)
> 16:45:44 at
> org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:440)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureCompression(HFileOutputFormat.java:458)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:375)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.ImportTsv.createSubmittableJob(ImportTsv.java:280)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.ImportTsv.main(ImportTsv.java:424)
> 16:45:44 Caused by: java.io.IOException: Call to
> hbase-rolling-6.ent.cloudera.com/10.20.186.99:22001 failed on local
> exception: java.io.EOFException
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1030)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:999)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
> 16:45:44 ... 7 more
> 16:45:44 Caused by: java.io.EOFException
> 16:45:44 at java.io.DataInputStream.readInt(DataInputStream.java:375)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
