[
https://issues.apache.org/jira/browse/HBASE-28101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767710#comment-17767710
]
Nihal Jain commented on HBASE-28101:
------------------------------------
Hi [~zhangduo] I suspect this change is causing 2 test cases to fail
consistently in branch-2.5 and branch-2.4.
*
org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider.testNegativeAuthentication
*
org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider
Branch-2.5
* PR failing:
[https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5422/3/testReport/]
* Nightly failing:
[https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/405/testReport/]
Branch-2.4
* PR failing:[
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5423/3/testReport/|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5423/3/testReport/]
* Nightly failing:
[https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/619/testReport/]
StackTrace:
{code:java}
java.lang.Exception: Unexpected exception,
expected<org.apache.hadoop.hbase.DoNotRetryIOException> but
was<org.apache.hadoop.hbase.client.RetriesExhaustedException>
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:30)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=4, exceptions:
2023-09-21T17:53:07.490Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=4, exceptions:
2023-09-21T17:53:06.869Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
Call to address=6a87b11f0f42:38383 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
2023-09-21T17:53:06.975Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.182Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.489Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:08.215Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=4, exceptions:
2023-09-21T17:53:07.597Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.703Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.909Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:08.215Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.057Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=4, exceptions:
2023-09-21T17:53:08.422Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:08.530Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
Call to address=6a87b11f0f42:38383 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
2023-09-21T17:53:08.748Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.056Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
Call to address=6a87b11f0f42:38383 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
2023-09-21T17:53:09.990Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100,
maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=4, exceptions:
2023-09-21T17:53:09.362Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.469Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.683Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.990Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:141)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:390)
at org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:363)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:363)
at
org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider$3.run(TestShadeSaslAuthenticationProvider.java:235)
at
org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider$3.run(TestShadeSaslAuthenticationProvider.java:230)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at
org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider.testNegativeAuthentication(TestShadeSaslAuthenticationProvider.java:230)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
... 18 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=4, exceptions:
2023-09-21T17:53:09.362Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.469Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.683Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.990Z,
RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100,
maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed
on local exception: java.io.IOException: Connection reset by peer
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:141)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:390)
at org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:363)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:363)
at
org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1141)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:742)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:214)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
... 37 more
Caused by: java.io.IOException: Call to address=6a87b11f0f42:38383 failed on
local exception: java.io.IOException: Connection reset by peer
at
java.base/jdk.internal.reflect.GeneratedConstructorAccessor69.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:239)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:212)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.exceptionCaught(NettyRpcDuplexHandler.java:229)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
at
org.apache.hbase.thirdparty.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259)
at
org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at
org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
at
org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
... 8 more
{code}
> Should check the return value of protobuf Message.mergeDelimitedFrom
> --------------------------------------------------------------------
>
> Key: HBASE-28101
> URL: https://issues.apache.org/jira/browse/HBASE-28101
> Project: HBase
> Issue Type: Bug
> Components: Protobufs, rpc
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Critical
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
>
>
> Found this problem while upgrading thirdparty dependencies.
> The javadoc of mergeDelimitedFrom says
> {code}
> /**
> * Like {@link #mergeFrom(InputStream)}, but does not read until EOF.
> Instead, the size of the
> * message (encoded as a varint) is read first, then the message data.
> Use {@link
> * MessageLite#writeDelimitedTo(OutputStream)} to write messages in this
> format.
> *
> * @return true if successful, or false if the stream is at EOF when the
> method starts. Any
> * other error (including reaching EOF during parsing) causes an
> exception to be thrown.
> * @throws InvalidProtocolBufferException the bytes read are not
> syntactically correct
> * according to the protobuf wire format specification. The data is
> corrupt, incomplete,
> * or was never a protobuf in the first place.
> * @throws IOException an I/O error reading from the stream
> */
> {code}
> So we should not ignore the return value, otherwise we may get an incomplete
> message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
