[jira] [Comment Edited] (NIFI-2575) HiveQL Processors Fail due to invalid JDBC URI resolution when using Zookeeper URI
[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681658#comment-16681658
]
Zack Atkinson edited comment on NIFI-2575 at 11/9/18 5:50 PM:
--
[~mattyb149], I see HDF NiFi 1.5-1.7.1 supports Hive2 queries using Zookeeper.
Is there any plans to update Apache NiFi with this fix? I am still seeing this
error all the way through Apache NiFi v1.8 when querying Hive against Zookeeper
using Kerberos.
was (Author: zack08):
[~mattyb149], I see HDF NiFi 1.5-1.7.1 supports Hive2 queries using Zookeeper.
Is there any plans to update Apache NiFi with this fix?
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> --
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Yolanda M. Davis
>Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown
> Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497)
> ~[na:1.8.0_65]
> at
>
[jira] [Comment Edited] (NIFI-2575) HiveQL Processors Fail due to invalid JDBC URI resolution when using Zookeeper URI
[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356883#comment-16356883
]
Gardella Juan Pablo edited comment on NIFI-2575 at 2/9/18 12:06 PM:
I had the same issue. It happens in version Nifi 1.3.0 and also in Nifi 1.5.0.
The problem is the driver, I've updated to 2.1.0 and it works. Another simple
workaround is using Hortonworks'
[nifi-hive-nar|http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/nifi/nifi-hive-nar/1.5.0.3.2.0.0-4/],
it seems Hortonworks is using a patched version.
The root cause I believe is https://issues.apache.org/jira/browse/HIVE-11875,
so to solve the issue it is required to upgrade Hive driver which include that
fix.
Notice that affects any processor that uses HiveConnectionPool.
was (Author: gardellajuanpablo):
I had the same issue. It happens in version Nifi 1.3.0 and also in Nifi 1.5.0.
The problem is the driver, I've updated to 2.1.0 and it works. Another simple
workaround is using Hortonworks'
[nifi-hive-nar|http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/nifi/nifi-hive-nar/1.5.0.3.2.0.0-4/],
it seems Hortonworks is using a patched version.
The root cause I believe is https://issues.apache.org/jira/browse/HIVE-11875,
so to solve the issue it is required to upgrade Hive driver which include that
fix.
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> --
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Yolanda M. Davis
>Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
>
[jira] [Comment Edited] (NIFI-2575) HiveQL Processors Fail due to invalid JDBC URI resolution when using Zookeeper URI
[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356883#comment-16356883
]
Gardella Juan Pablo edited comment on NIFI-2575 at 2/8/18 12:28 PM:
I had the same issue. It happens in version Nifi 1.3.0 and also in Nifi 1.5.0.
The problem is the driver, I've updated to 2.1.0 and it works. Another simple
workaround is using Hortonworks'
[nifi-hive-nar|http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/nifi/nifi-hive-nar/1.5.0.3.2.0.0-4/],
it seems Hortonworks is using a patched version.
The root cause I believe is https://issues.apache.org/jira/browse/HIVE-11875,
so to solve the issue it is required to upgrade Hive driver which include that
fix.
was (Author: gardellajuanpablo):
I had the same issue. It happens in version Nifi 1.3.0 and also in Nifi 1.5.0.
The problem is the driver, I've updated to 2.1.0 and it works. Another simple
workaround is using [Hortonworks'
nifi-hive-nar|[http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/nifi/nifi-hive-nar/1.5.0.3.2.0.0-4/]],
it seems Hortonworks is using a patched version.
The root cause I believe is https://issues.apache.org/jira/browse/HIVE-11875,
so to solve the issue it is required to upgrade Hive driver which include that
fix.
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> --
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Yolanda M. Davis
>Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process
