Szehon Ho created HIVE-20789:
--------------------------------
Summary: HiveServer2 should have Timeouts against clients that
never close sockets
Key: HIVE-20789
URL: https://issues.apache.org/jira/browse/HIVE-20789
Project: Hive
Issue Type: Bug
Reporter: Szehon Ho
We have had a scenario that health checks sending 0 bytes to HiveServer2
sockets would DDOS the HiveServer2, if they dont send TCP FIN then they will
continually cause all HiveServer2 thrift thread-pool threads to block at this
stack (we are running an older version of Hive here, so ignore the lines)
{noformat}
"HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <23781b74> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
at
org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
Eventually HiveServer2 has no more free threads left.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
