[
https://issues.apache.org/jira/browse/HDDS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464486#comment-17464486
]
Sadanand Shenoy commented on HDDS-5731:
---------------------------------------
Tried doing _ozone admin pipeline list_ and found that the datanodes in every
RATIS THREE pipeline didn’t have the datastream port information. Closing these
pipelines and disabling creation of ONE factor ratis pipelines brought up new
RATIS THREE pipelines in a while with the datastream port information and
writes are working. Still not sure what caused it though!
> Ozone key put call doesn’t succeed and gets stuck while using streaming code
> path
> ---------------------------------------------------------------------------------
>
> Key: HDDS-5731
> URL: https://issues.apache.org/jira/browse/HDDS-5731
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Assignee: Tsz-wo Sze
> Priority: Major
> Attachments: datanode-jstack.txt, ozone-client-jstack.txt
>
>
> On executing ozone sh key put for a keysize > 4mb , the key put gets stuck
> forever without any exception reported on DN.
> On the stderr ,this is found
> {code:java}
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServerTransport
> notifyTerminated
> INFO: Transport failed
> org.apache.ratis.thirdparty.io.netty.handler.codec.http2.Http2Exception:
> HTTP/2 client preface string missing or corrupt. Hex dump for received bytes:
> 00000000000000b0000000170a150a10bafb674a520448a7
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:108)
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:306)
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
> at
> org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
> at
> org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
> at
> org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> cc : [~szetszwo] [~shashikant]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
