Hrm, the errors here <http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/quot-BlockingRecv-error-quot-when-inserting-rows-into-KUDU-1-2/m-p/52484#M2685?eid=1&aid=1> seem similar to the ones you're seeing. Does that give you any clues? Are there any other processes running on your machines?
Andrew On Thu, Nov 30, 2017 at 7:32 PM, Zhen Zhang <[email protected]> wrote: > Hi all, I found some rpc negotiation errors on both master and client > side. I'm not sure what cause this, on both client and master server, I see > no network errors using ifconfig. Do you know what may cause this error > occur? > > I'm using Kudu1.3.0, and the security configuration is disabled: > --rpc_encryption=disabled > --rpc_authentication=disabled > > The master side error: > W1201 09:16:05.300465 71450 negotiation.cc:290] Failed RPC negotiation. > Trace: > 1201 09:15:50.300130 (+ 0us) reactor.cc:373] Submitting negotiation > task for server connection from 10.68.11.135:42847 > 1201 09:15:50.300446 (+ 316us) server_negotiation.cc:118] Beginning > negotiation > 1201 09:15:50.300448 (+ 2us) server_negotiation.cc:282] Waiting for > connection header > 1201 09:16:05.300393 (+14999945us) negotiation.cc:281] Negotiation > complete: Timed out: Server connection negotiation failed: server > connection from 10.68.11.135:42847 > Metrics: {"negotiator.queue_time_us":237,"thread_start_us":213," > threads_started":1} > > > The client side error: > 2017-12-01,09:16:46,063 ERROR org.apache.kudu.client.TabletClient: [Peer > master-sgp1-hadoop-prc-kudu01.awssgp:18600] Unexpected exception from > downstream on [id: 0x21836b16, /10.68.11.135:42847 => > sgp1-hadoop-prc-kudu01.awssgp/10.68.1.73:18600] > java.io.IOException: Broken pipe > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:51) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo( > SocketSendBufferPool.java:203) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > socket.nio.NioClientSocketPipelineSink.eventSunk( > NioClientSocketPipelineSink.java:84) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream( > DefaultChannelPipeline.java:779) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > Channels.write(Channels.java:725) > at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec. > oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) > at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec. > oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > Channels.write(Channels.java:704) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > Channels.write(Channels.java:671) > at org.apache.kudu.client.Negotiator.sendSaslMessage( > Negotiator.java:218) > at org.apache.kudu.client.Negotiator.sendNegotiateMessage( > Negotiator.java:211) > at org.apache.kudu.client.Negotiator.sendHello( > Negotiator.java:178) > at org.apache.kudu.client.TabletClient.channelConnected( > TabletClient.java:586) > at org.apache.kudu.client.shaded.org.jboss.netty.channel. > SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler. > java:100) > at org.apache.kudu.client.TabletClient.handleUpstream( > TabletClient.java:595) > > Thanks, > Zhen > -- Andrew Wong
