So if your not able to connect with beeline with that trustore info you have a typo. Either a typo in your connection string.(unlikely) or the trustore isn't valid.(a different type of typo)
Can you rules either of them out as being a typo? (Does using the zookeeper namespace work connection string for hive work). You know the one that's in Ambari, on the hive summary page? Hive does actually right the connection string it wants you to use to the zookeeper namespace. Can you connect to zookeeper and check what string it wants you to use? (Send me a reply and I'll dig up how to do this if you don't know how to do it. If you have more than 1 hive instance their will be more than 1 entry... I'm not going to respond until tomorrow just so you know ). On Tue., Dec. 17, 2019, 12:00 Jan Hentschel, < [email protected]> wrote: > We already tried the SSL configuration in the Beeline connection string > with a similar result. Hive is in binary mode with TLS. Below the output > when trying to use the Truststore parameters in the connection string > (resulting in the same output in the Thriftserver logs): > > > > spark@master01:/$ /usr/hdp/current/spark2-client/bin/beeline -u > 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > ' -e 'show databases' > > Connecting to jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:20 INFO Utils: Supplied authorities: master01:10016 > > 19/11/01 10:35:20 INFO Utils: Resolved authority: master01:10016 > > 19/11/01 10:35:20 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 19/11/01 10:35:20 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: > binary > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit: > Invalid status 21 (state=08S01,code=0) > > 19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016 > > 19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016 > > 19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: > binary > > No current connection > > 19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016 > > 19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016 > > 19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected] > ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit > > 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: > binary > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit: > Invalid status 21 (state=08S01,code=0) > > > > *From: *Matt Andruff <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Friday, December 13, 2019 at 8:17 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Problem with Spark2 Thriftserver Alert in Ambari > > > > It's clear from the Ambati alert the thrift server connection it's trying > to use binary connection (and missing the trustore). Are you using hive in > binary with SSL or https? > > > > On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <[email protected]> > wrote: > > What happens when you try to connect via beeline with: > > > > > jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password> > > > > ? > > > > Or are you just using the zookeeper namespace to connect? > > > > On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, < > [email protected]> wrote: > > Hello everybody, > > > > we're having problems with the Ambari alert for the Spark Thriftserver, > complaining about the Beeline connection to the Thriftserver (see the alert > below): > > > > Connection failed on host master01:10016 (Traceback (most recent call > last): > > File > "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py", > line 147, in execute > > Execute(cmd, user=sparkuser, path=[beeline_cmd], > timeout=CHECK_COMMAND_TIMEOUT_DEFAULT) > > File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line > 166, in __init__ > > self.env.run() > > File > "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line > 160, in run > > self.run_action(resource, action) > > File > "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line > 124, in run_action > > provider_action() > > File > "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", > line 263, in action_run > > returns=self.resource.returns) > > File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line > 72, in inner > > result = function(command, **kwargs) > > File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line > 102, in checked_call > > tries=tries, try_sleep=try_sleep, > timeout_kill_strategy=timeout_kill_strategy, returns=returns) > > File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line > 150, in _call_wrapper > > result = _call(command, **kwargs_copy) > > File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line > 314, in _call > > raise ExecutionFailed(err_msg, code, out, err) > > ExecutionFailed: Execution of '! > /usr/hdp/current/spark2-client/bin/beeline -u > 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ' returned 1. Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: > java.net.ConnectException: Connection refused (Connection refused) > (state=08S01,code=0) > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: > java.net.ConnectException: Connection refused (Connection refused) > (state=08S01,code=0) > > ) > > > > Running it with a user the same problem comes up: > > > > dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$ > ! /usr/hdp/current/spark2-client/bin/beeline -u > 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary' > -e '' > > Connecting to jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016 > > 19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016 > > 19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: > binary > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: > java.net.SocketException: Connection reset (state=08S01,code=0) > > Beeline version 1.21.2.3.0.1.0-187 by Apache Hive > > 19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016 > > 19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016 > > 19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: > binary > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: > java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0) > > > > When trying to run the same with the Spark user from the command line, a > similar issues arises: > > > > spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u > 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary' > -e 'show databases' > > Connecting to jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016 > > 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016 > > 19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: > binary > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: > Invalid status 21 (state=08S01,code=0) > > 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016 > > 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016 > > 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: > binary > > No current connection > > 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016 > > 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016 > > 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport > with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ > [email protected];transportMode=binary > > 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: > binary > > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: > Invalid status 21 (state=08S01,code=0) > > > > Looking at the Thriftserver logs reveals the following: > > > > 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during > processing of message. > > java.lang.RuntimeException: > org.apache.thrift.transport.TTransportException: > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? > > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739) > > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:360) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736) > > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.thrift.transport.TTransportException: > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? > > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) > > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > > at > org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) > > at > org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) > > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) > > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > > ... 10 more > > Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext > connection? > > at > sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) > > at sun.security.ssl.InputRecord.read(InputRecord.java:527) > > at > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) > > at > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) > > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) > > ... 16 more > > 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during > processing of message. > > java.lang.RuntimeException: > org.apache.thrift.transport.TTransportException: > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? > > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739) > > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:360) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736) > > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.thrift.transport.TTransportException: > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? > > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) > > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > > at > org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) > > at > org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) > > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) > > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > > ... 10 more > > Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext > connection? > > at > sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) > > at sun.security.ssl.InputRecord.read(InputRecord.java:527) > > at > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) > > at > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) > > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) > > ... 16 more > > > > TLS is configured on the Hive side, but still it seems that instead of a > TLS connection Beeline or Spark tries to establish a plaintext connection. > Does anyone know how to resolve this problem? Besides the points > mentioned above the Thriftserver, f.e. when connecting to it via ODBC. > > > >
