[ https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417940#comment-16417940 ]
Matt Burgess commented on NIFI-2575: ------------------------------------ Early testing of the NiFi Hive 3 bundle against Hive 3.0 indicates that this issue is indeed fixed for these version(s). > HiveQL Processors Fail due to invalid JDBC URI resolution when using > Zookeeper URI > ---------------------------------------------------------------------------------- > > Key: NIFI-2575 > URL: https://issues.apache.org/jira/browse/NIFI-2575 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Reporter: Yolanda M. Davis > Priority: Major > > When configuring a HiveQL processor using the Zookeeper URL (e.g. > jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2), > it appears that the JDBC driver does not properly build the the uri in the > expected format. This is because HS2 is storing JDBC parameters in ZK > (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the > driver to be able to parse and use those values to configure the connection. > However it appears the driver is expecting zookeeper to simply return the > host:port and subsequently building an invalid URI. > This problem has result in two variation of errors. The following was > experienced by [~mattyb149] > {noformat} > 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2] > org.apache.hive.jdbc.Utils Resolved authority: > hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com > 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2] > org.apache.hive.jdbc.HiveConnection Will try to open client transport with > JDBC Uri: > jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2 > 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2] > org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC > Uri: > jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2 > 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2] > o.a.c.f.imps.CuratorFrameworkImpl Starting > 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread] > o.a.c.f.state.ConnectionStateManager State change: CONNECTED > 2016-08-15 12:45:14,182 INFO [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting > 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2] > o.a.nifi.processors.hive.SelectHiveQL > SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] > SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due > to java.lang.reflect.UndeclaredThrowableException; rolling back session: > java.lang.reflect.UndeclaredThrowableException > 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2] > o.a.nifi.processors.hive.SelectHiveQL > java.lang.reflect.UndeclaredThrowableException: null > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > ~[na:na] > at > org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255) > ~[na:na] > at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown > Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_65] > at java.lang.reflect.Method.invoke(Method.java:497) > ~[na:1.8.0_65] > at > org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na] > at > org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_65] > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_65] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_65] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_65] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create > PoolableConnectionFactory (Could not open client transport for any of the > Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper) > at > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) > ~[na:na] > {noformat} > The following error was experienced by [~YolandaMDavis] > {noformat} > 2016-08-15 19:22:27,338 INFO [Timer-Driven Process Thread-7] > org.apache.hive.jdbc.Utils Supplied authorities: > ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181 > 2016-08-15 19:22:27,340 INFO [Timer-Driven Process Thread-7] > o.a.c.f.imps.CuratorFrameworkImpl Starting > 2016-08-15 19:22:27,360 INFO [Timer-Driven Process Thread-7-EventThread] > o.a.c.f.state.ConnectionStateManager State change: CONNECTED > 2016-08-15 19:22:27,365 INFO [Timer-Driven Process Thread-7] > o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with > uri: > hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false > 2016-08-15 19:22:27,365 INFO [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting > 2016-08-15 19:22:27,371 INFO [Timer-Driven Process Thread-7] > org.apache.hive.jdbc.Utils Resolved authority: > hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false > 2016-08-15 19:22:27,374 INFO [Timer-Driven Process Thread-7] > org.apache.hive.jdbc.HiveConnection Will try to open client transport with > JDBC Uri: > jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 > 2016-08-15 19:22:27,374 ERROR [Timer-Driven Process Thread-7] > o.apache.nifi.processors.hive.PutHiveQL > PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] > PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process due to > java.lang.NullPointerException; rolling back session: > java.lang.NullPointerException > 2016-08-15 19:22:27,383 ERROR [Timer-Driven Process Thread-7] > o.apache.nifi.processors.hive.PutHiveQL > java.lang.NullPointerException: null > at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[na:na] > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266) > ~[na:na] > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > ~[na:na] > at > org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204) > ~[na:na] > at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) > ~[na:na] > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[na:na] > at > org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) > ~[na:na] > at > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) > ~[na:na] > at > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) > ~[na:na] > at > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) > ~[na:na] > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) > ~[na:na] > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > ~[na:na] > at > org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:264) > ~[na:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_101] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_101] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_101] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101] > at > org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174) > ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at com.sun.proxy.$Proxy120.getConnection(Unknown Source) ~[na:na] > at > org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:152) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > 2016-08-15 19:22:27,384 ERROR [Timer-Driven Process Thread-7] > o.apache.nifi.processors.hive.PutHiveQL > PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] > PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process session > due to java.lang.NullPointerException: java.lang.NullPointerException > {noformat} > This problem appears to be resolved in the Apache 1.3 release which is not > yet available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)