[
https://issues.apache.org/jira/browse/HBASE-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768289#comment-17768289
]
Duo Zhang commented on HBASE-28106:
-----------------------------------
I think there are mainly two problems...
First is the patch in HBASE-28101 is incorrect, we should not throw
EOFException out there, as we have already removed the call from the map, so
throwing exception out can not finish call, it will hang there and finally we
will get a tmeout exception...
Second is the UT itself, I tried to print out the DoNotRetryIOException, it is
TableNotFound, obviously this is incorrect... The server just closes the
connection when auth failed, it should not be considered as TableNotFound... I
think this is exactly what we want to fix in HBASE-28101...
Let me reopen HBASE-28101 and apply an addendum about the first problem, and
then let's change the test code in this issue.
Thanks.
> TestShadeSaslAuthenticationProvider fails for branch-2.x
> --------------------------------------------------------
>
> Key: HBASE-28106
> URL: https://issues.apache.org/jira/browse/HBASE-28106
> Project: HBase
> Issue Type: Bug
> Reporter: Nihal Jain
> Priority: Blocker
> Fix For: 2.6.0, 2.4.18, 2.5.6
>
>
> {{org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider}}
> is failing consistently in branch-2.x
> This could be a code bug as well!
>
> For branch-2:
> * Nightly failing:
> [https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/885/testReport/]
> For branch-2.5
> * PR failing:
> [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5422/3/testReport/]
> * Nightly failing:
> [https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/405/testReport/]
> For branch-2.4
> * PR failing:[
> [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5423/3/testReport/]
> * Nightly failing:
> [https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/619/testReport/]
>
> StackTrace copied from failed test case:
> {code:java}
> java.lang.Exception: Unexpected exception,
> expected<org.apache.hadoop.hbase.DoNotRetryIOException> but
> was<org.apache.hadoop.hbase.client.RetriesExhaustedException>
> at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:30)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=4, exceptions:
> 2023-09-21T17:53:07.490Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=4, exceptions:
> 2023-09-21T17:53:06.869Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Call to address=6a87b11f0f42:38383 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection
> closed
> 2023-09-21T17:53:06.975Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:07.182Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:07.489Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:08.215Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=4, exceptions:
> 2023-09-21T17:53:07.597Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:07.703Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:07.909Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:08.215Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.057Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=4, exceptions:
> 2023-09-21T17:53:08.422Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:08.530Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Call to address=6a87b11f0f42:38383 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection
> closed
> 2023-09-21T17:53:08.748Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.056Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Call to address=6a87b11f0f42:38383 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection
> closed
> 2023-09-21T17:53:09.990Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
> maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=4, exceptions:
> 2023-09-21T17:53:09.362Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.469Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.683Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.990Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:141)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:390)
> at org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:363)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:363)
> at
> org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider$3.run(TestShadeSaslAuthenticationProvider.java:235)
> at
> org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider$3.run(TestShadeSaslAuthenticationProvider.java:230)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at
> org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider.testNegativeAuthentication(TestShadeSaslAuthenticationProvider.java:230)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
> ... 18 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=4, exceptions:
> 2023-09-21T17:53:09.362Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.469Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.683Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> 2023-09-21T17:53:09.990Z,
> RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
> maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383
> failed on local exception: java.io.IOException: Connection reset by peer
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:141)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:390)
> at org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:363)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:363)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1141)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:742)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:214)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
> ... 37 more
> Caused by: java.io.IOException: Call to address=6a87b11f0f42:38383 failed on
> local exception: java.io.IOException: Connection reset by peer
> at
> java.base/jdk.internal.reflect.GeneratedConstructorAccessor69.newInstance(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:212)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.exceptionCaught(NettyRpcDuplexHandler.java:229)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> Caused by: java.io.IOException: Connection reset by peer
> at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
> at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
> at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
> at
> java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
> at
> org.apache.hbase.thirdparty.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259)
> at
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
> at
> org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
> ... 8 more
> {code}
>
> Also note the test class code is not same between master and branch-2.x
> *
> [https://github.com/apache/hbase/blob/master/hbase-examples/src/test/java/org/apache/hadoop/hbase/security/provider/example/TestShadeSaslAuthenticationProvider.java]
> *
> [https://github.com/apache/hbase/blob/branch-2/hbase-examples/src/test/java/org/apache/hadoop/hbase/security/provider/example/TestShadeSaslAuthenticationProvider.java]
>
> This issue could be related to HBASE-28101
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
