[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681658#comment-16681658
]
Zack Atkinson commented on NIFI-2575:
-------------------------------------
[~mattyb149], I see HDF NiFi 1.5-1.7.1 supports Hive2 queries using Zookeeper.
Is there any plans to update Apache NiFi with this fix?
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> ----------------------------------------------------------------------------------
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Yolanda M. Davis
> Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/[email protected]
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/[email protected]/default;principal=hive/[email protected];serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/[email protected]/default;principal=hive/[email protected];serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown
> Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497)
> ~[na:1.8.0_65]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_65]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
> PoolableConnectionFactory (Could not open client transport for any of the
> Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
> at
> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
> ~[na:na]
> {noformat}
> The following error was experienced by [~YolandaMDavis]
> {noformat}
> 2016-08-15 19:22:27,338 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.Utils Supplied authorities:
> ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181
> 2016-08-15 19:22:27,340 INFO [Timer-Driven Process Thread-7]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 19:22:27,360 INFO [Timer-Driven Process Thread-7-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 19:22:27,365 INFO [Timer-Driven Process Thread-7]
> o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with
> uri:
> hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
> 2016-08-15 19:22:27,365 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 19:22:27,371 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
> 2016-08-15 19:22:27,374 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 19:22:27,374 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process due to
> java.lang.NullPointerException; rolling back session:
> java.lang.NullPointerException
> 2016-08-15 19:22:27,383 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> java.lang.NullPointerException: null
> at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[na:na]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> ~[na:na]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> ~[na:na]
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
> ~[na:na]
> at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
> ~[na:na]
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[na:na]
> at
> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
> ~[na:na]
> at
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:264)
> ~[na:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_101]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_101]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_101]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy120.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:152)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-15 19:22:27,384 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process session
> due to java.lang.NullPointerException: java.lang.NullPointerException
> {noformat}
> This problem appears to be resolved in the Apache 1.3 release which is not
> yet available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
