[
https://issues.apache.org/jira/browse/THRIFT-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated THRIFT-5435:
---------------------------------
Description:
Portion of a blog called "NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN
HIVESERVER2 LOG" by Eric Liu are below. NOTE: The web link causes the Apache
bot to remove it. You will need to Google it yourself.
It addresses HIve Server 2 crashes our customers have seen and filed customer
service complaints on. The Apache Thrift library raises a fatal
RuntimeException when in my opinion it should raise a non-fatal Exception
instead.
Portion of the blog:
-----------------------------------------------------------------------------------------------------
"I have seen lots of users complain about seeing lots of “No data or no sasl
data in the stream” errors in the HiveServer2 server log, yet they have not
noticed any performance impact nor query failure for Hive. So I think it would
be good to write a blog about the possible reason behind this to clarify and
remove the concerns that users have. The following shows the full error message
and stacktrace taken from HiveServer2 log:
ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool:
Thread-533556]: Error occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in
the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:765)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:762)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1687)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:762)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
The likely cause is below:
# You have kerberos enabled
# You have multiple HiveServer2 hosts
# You have Load Balancer enabled in front of all HS2 servers that have such
errors
If you have above setup, the error message you saw in HiveServer2 is harmless
and can be safely ignored. This just indicated that SASL negotiation failed for
one particular Hive client, which in this case would be the Load Balancer who
pings regularly to those HiveServer2’s to check for connectivity. Those pings
from LB were trying with PLAIN TCP connection, hence those messages.
-----------------------------------------------------------------------------------------------------
This is similar exception information from Hive Server 2 that crashed on our
customer's machine:
ExceptionClassName
java.lang.RuntimeException
ExceptionStackTrace
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in
the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
was:
These portion of a blog called "NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN
HIVESERVER2 LOG" by Eric Liu. NOTE: The web link causes the Apache bot to
remove it. You will need to Google it yourself.
It addresses HIve Server 2 crashes our customers have seen and filed customer
service complaints on. The Apache Thrift library raises a fatal
RuntimeException when In my opinion it should raise a non-fatal Exception
instead.
Portion of the blog:
-----------------------------------------------------------------------------------------------------
"I have seen lots of users complain about seeing lots of “No data or no sasl
data in the stream” errors in the HiveServer2 server log, yet they have not
noticed any performance impact nor query failure for Hive. So I think it would
be good to write a blog about the possible reason behind this to clarify and
remove the concerns that users have. The following shows the full error message
and stacktrace taken from HiveServer2 log:
ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool:
Thread-533556]: Error occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in
the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:765)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:762)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1687)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:762)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
The likely cause is below:
# You have kerberos enabled
# You have multiple HiveServer2 hosts
# You have Load Balancer enabled in front of all HS2 servers that have such
errors
If you have above setup, the error message you saw in HiveServer2 is harmless
and can be safely ignored. This just indicated that SASL negotiation failed for
one particular Hive client, which in this case would be the Load Balancer who
pings regularly to those HiveServer2’s to check for connectivity. Those pings
from LB were trying with PLAIN TCP connection, hence those messages.
-----------------------------------------------------------------------------------------------------
This is similar exception information from Hive Server 2 that crashed on our
customer's machine:
ExceptionClassName
java.lang.RuntimeException
ExceptionStackTrace
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in
the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
> Hive Server 2 crashes when Thrift library encounters particular security error
> ------------------------------------------------------------------------------
>
> Key: THRIFT-5435
> URL: https://issues.apache.org/jira/browse/THRIFT-5435
> Project: Thrift
> Issue Type: Improvement
> Reporter: Matt McCline
> Priority: Major
>
> Portion of a blog called "NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN
> HIVESERVER2 LOG" by Eric Liu are below. NOTE: The web link causes the Apache
> bot to remove it. You will need to Google it yourself.
> It addresses HIve Server 2 crashes our customers have seen and filed customer
> service complaints on. The Apache Thrift library raises a fatal
> RuntimeException when in my opinion it should raise a non-fatal Exception
> instead.
> Portion of the blog:
> -----------------------------------------------------------------------------------------------------
> "I have seen lots of users complain about seeing lots of “No data or no sasl
> data in the stream” errors in the HiveServer2 server log, yet they have not
> noticed any performance impact nor query failure for Hive. So I think it
> would be good to write a blog about the possible reason behind this to
> clarify and remove the concerns that users have. The following shows the full
> error message and stacktrace taken from HiveServer2 log:
>
> ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool:
> Thread-533556]: Error occurred during processing of message.
> java.lang.RuntimeException:
> org.apache.thrift.transport.TSaslTransportException: No data or no sasl data
> in the stream
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:765)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:762)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1687)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:762)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TSaslTransportException: No data or
> no sasl data in the stream
> The likely cause is below:
> # You have kerberos enabled
> # You have multiple HiveServer2 hosts
> # You have Load Balancer enabled in front of all HS2 servers that have such
> errors
> If you have above setup, the error message you saw in HiveServer2 is harmless
> and can be safely ignored. This just indicated that SASL negotiation failed
> for one particular Hive client, which in this case would be the Load Balancer
> who pings regularly to those HiveServer2’s to check for connectivity. Those
> pings from LB were trying with PLAIN TCP connection, hence those messages.
> -----------------------------------------------------------------------------------------------------
> This is similar exception information from Hive Server 2 that crashed on our
> customer's machine:
> ExceptionClassName
> java.lang.RuntimeException
> ExceptionStackTrace
> java.lang.RuntimeException:
> org.apache.thrift.transport.TSaslTransportException: No data or no sasl data
> in the stream
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
> sasl data in the stream
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 10 more
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
