[
https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691444#comment-16691444
]
Peter Vary commented on HIVE-20845:
-----------------------------------
There are strange errors in the hive.log when trying to connect to an
active-passive HS2 with DriverManager.getConnection(...).
{code:java}
2018-11-19T00:00:17,408 INFO [Thread-9] jdbc.ZooKeeperHiveClientHelper: Found
HS2 Active Host: localhost Port: 50666 Identity:
2ca89854-eecd-447b-b5b3-8882af5f44ac Mode: binary
2018-11-19T00:00:17,408 DEBUG [Thread-9] jdbc.ZooKeeperHiveClientHelper:
Configurations applied to JDBC connection params.
{hive.server2.instance.uri=localhost:50666, hive.server2.authentication=NONE,
hive.server2.transport.mode=binary, hive.server2.thrift.sasl.qop=auth,
hive.server2.thrift.bind.host=localhost, hive.server2.thrift.port=50666,
hive.server2.use.SSL=false,
registry.unique.id=2ca89854-eecd-447b-b5b3-8882af5f44ac,
hive.server2.webui.port=50665}
2018-11-19T00:00:17,408 DEBUG [Thread-9] jdbc.Utils: Resolved authority:
localhost:50666
2018-11-19T00:00:18,501 INFO [Thread-9] jdbc.HiveConnection: Connected to
localhost:50666
2018-11-19T00:00:18,671 ERROR [HiveServer2-Handler-Pool: Thread-130]
server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
Invalid status 80
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
[libthrift-0.9.3.jar:0.9.3]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
Caused by: org.apache.thrift.transport.TTransportException: Invalid status 80
at
org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
~[libthrift-0.9.3.jar:0.9.3]
... 4 more
2018-11-19T00:00:18,681 ERROR [Thread-9] jdbc.HiveConnection: Error opening
session
org.apache.thrift.transport.TTransportException:
org.apache.http.client.ClientProtocolException
at
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170)
~[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162)
~[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:809)
[hive-jdbc-4.0.0-SNAPSHOT.jar:?]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:309)
[hive-jdbc-4.0.0-SNAPSHOT.jar:?]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
[hive-jdbc-4.0.0-SNAPSHOT.jar:?]
at java.sql.DriverManager.getConnection(DriverManager.java:664)
[?:1.8.0_172]
at java.sql.DriverManager.getConnection(DriverManager.java:270)
[?:1.8.0_172]
at
org.apache.hive.jdbc.miniHS2.MiniHS2.waitForStartup(MiniHS2.java:622)
[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:402)
[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized(TestActivePassiveHA.java:406)
[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
[junit-4.11.jar:?]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[junit-4.11.jar:?]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
[junit-4.11.jar:?]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[junit-4.11.jar:?]
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[junit-4.11.jar:?]
Caused by: org.apache.http.client.ClientProtocolException
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
~[libthrift-0.9.3.jar:0.9.3]
... 22 more
Caused by: org.apache.http.ProtocolException: The server failed to respond with
a valid HTTP response
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:151)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
~[httpcore-4.4.4.jar:4.4.4]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
~[httpcore-4.4.4.jar:4.4.4]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
~[httpcore-4.4.4.jar:4.4.4]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
~[httpcore-4.4.4.jar:4.4.4]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
~[httpclient-4.5.2.jar:4.5.2]
at
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
~[libthrift-0.9.3.jar:0.9.3]
... 22 more
2018-11-19T00:00:18,683 WARN [Thread-9] jdbc.HiveConnection: Failed to connect
to localhost:50666
{code}
And later:
{code:java}
2018-11-19T00:00:38,026 INFO [Thread-9]
server.HS2ActivePassiveHARegistryClient: Returning cached registry client for
namespace: hs2ActivePassiveHATest-unsecure
2018-11-19T00:00:38,026 INFO [Thread-9] jdbc.ZooKeeperHiveClientHelper: Found
HS2 Active Host: localhost Port: 50668 Identity:
8b4ca1b4-ade6-47d9-ba00-f3e5f6dd4ece Mode: http:/clidriverTest
2018-11-19T00:00:38,026 DEBUG [Thread-9] jdbc.ZooKeeperHiveClientHelper:
Configurations applied to JDBC connection params.
{hive.server2.instance.uri=localhost:50668, hive.server2.authentication=NONE,
hive.server2.transport.mode=http, hive.server2.thrift.http.path=clidriverTest,
hive.server2.thrift.http.port=50668, hive.server2.thrift.bind.host=localhost,
hive.server2.use.SSL=false,
registry.unique.id=8b4ca1b4-ade6-47d9-ba00-f3e5f6dd4ece,
hive.server2.webui.port=50669}
2018-11-19T00:00:38,026 INFO [Thread-9] jdbc.Utils: Selected HiveServer2
instance with uri:
jdbc:hive2://localhost:50668/default;serviceDiscoveryMode=zooKeeperHA;zooKeeperNamespace=hs2ActivePassiveHATest;transportMode=http;httpPath=cliservice;
2018-11-19T00:00:38,026 WARN [Thread-9] jdbc.HiveConnection: Could not open
client transport with JDBC Uri:
jdbc:hive2://localhost:50668/default;serviceDiscoveryMode=zooKeeperHA;zooKeeperNamespace=hs2ActivePassiveHATest;transportMode=http;httpPath=cliservice;:
Could not establish connection to
jdbc:hive2://localhost:50668/default;serviceDiscoveryMode=zooKeeperHA;zooKeeperNamespace=hs2ActivePassiveHATest;transportMode=http;httpPath=cliservice;:
HTTP Response code: 405 Retrying 0 of 1
2018-11-19T00:00:38,028 INFO [Thread-9] jdbc.HiveConnection: Connected to
localhost:50668
2018-11-19T00:00:38,030 ERROR [Thread-9] jdbc.HiveConnection: Error opening
session
org.apache.thrift.transport.TTransportException: HTTP Response code: 405
at
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:262)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170)
~[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162)
~[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:809)
[hive-jdbc-4.0.0-SNAPSHOT.jar:?]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:309)
[hive-jdbc-4.0.0-SNAPSHOT.jar:?]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
[hive-jdbc-4.0.0-SNAPSHOT.jar:?]
at java.sql.DriverManager.getConnection(DriverManager.java:664)
[?:1.8.0_172]
at java.sql.DriverManager.getConnection(DriverManager.java:270)
[?:1.8.0_172]
at
org.apache.hive.jdbc.miniHS2.MiniHS2.waitForStartup(MiniHS2.java:622)
[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:402)
[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized(TestActivePassiveHA.java:406)
[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
[junit-4.11.jar:?]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[junit-4.11.jar:?]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
[junit-4.11.jar:?]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[junit-4.11.jar:?]
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[junit-4.11.jar:?]
2018-11-19T00:00:38,030 WARN [Thread-9] jdbc.HiveConnection: Failed to connect
to localhost:50668
{code}
Might indicate some bug in HA implementation? Any thoughts [~sershe]?
For the tests I just fell back to using the original method for checking the
status of the MiniHS2 instances in HA mode.
> Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
> --------------------------------------------
>
> Key: HIVE-20845
> URL: https://issues.apache.org/jira/browse/HIVE-20845
> Project: Hive
> Issue Type: Test
> Reporter: Peter Vary
> Assignee: Peter Vary
> Priority: Major
> Attachments: HIVE-20845.2.patch, HIVE-20845.3.patch, HIVE-20845.patch
>
>
> Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and
> TestJdbcWithDBTokenStore
> Found out the issue is that we do not wait enough for HS2 to come up.
> Need to fix in MiniHS2.waitForStartup()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
