Re: Hive connection Pool error

2018-05-30 Thread Jeff 
Vishal Dutt,

Your issue relates to an existing JIRA [1] and as luck would have it, it's
already resolved! :) The fix for that JIRA [1] has been merged to master
and will be in the next NiFi release.

[1] https://issues.apache.org/jira/browse/NIFI-5134
[2] https://github.com/apache/nifi/pull/2667

On Wed, May 30, 2018 at 3:42 AM Pierre Villard 
wrote:

> Hi,
>
> Could you share additional details about the processor/CS configuration as
> well?
>
> Thanks
>
> 2018-05-30 7:03 GMT+02:00 Koji Kawamura :
>
>> Hello,
>>
>> Although I encountered various Kerberos related error, I haven't
>> encountered that one.
>> I tried to reproduce the same error by changing Kerberos related
>> configuration, but to no avail.
>> I recommend enabling Kerberos debug option for further debugging.
>>
>> You can add the option at nifi/conf/bootstrap.conf:
>> java.arg.19=-Dsun.security.krb5.debug=true
>>
>> Then debug logs are written to nifi/logs/nifi-bootstap.log
>>
>> Thanks,
>> Koji
>>
>> On Tue, May 29, 2018 at 10:31 PM, Vishal Dutt 
>> wrote:
>> > Hi ,
>> >
>> >
>> >
>> > We  are getting below error on randomly for few minutes and then goes
>> away,
>> > its coming in PUThiveql
>> >
>> >
>> >
>> >
>> >
>> > 2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95]
>> > org.apache.hive.jdbc.HiveConnection Will try to open client transport
>> with
>> > JDBC Uri:
>> > jdbc:hive2://
>> ctcl-hdpmaster2.msoit.com:1/default;principal=hive/_h...@msoit.com
>> >
>> > 2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95]
>> > o.apache.thrift.transport.TSaslTransport SASL negotiation failure
>> >
>> > javax.security.sasl.SaslException: GSS initiate failed
>> >
>> > at
>> >
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>> >
>> > at
>> >
>> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
>> >
>> > at
>> > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>> >
>> > at
>> >
>> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>> >
>> > at
>> >
>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>> >
>> > at
>> >
>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>> >
>> > at
>> >
>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>> >
>> > at
>> >
>> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
>> >
>> > at
>> > org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
>> >
>> > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>> >
>> > at
>> >
>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>> >
>> > at
>> >
>> org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>> >
>> > at
>> >
>> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)
>> >
>> > at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
>> >
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >
>> > at java.lang.reflect.Method.invoke(Method.java:498)
>> >
>> > at
>> >
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
>> >
>> > at com.sun.proxy.$Proxy97.getConnection(Unknown Source)
>> >
>> > at
>> >
>> org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191)
>> >
>> > at
>> org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96)
>> >
>> > at
>> >
>> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274)
>> >
>> > at
>> >
>>

Re: Hive connection Pool error

2018-05-30 Thread Pierre Villard 
Hi,

Could you share additional details about the processor/CS configuration as
well?

Thanks

2018-05-30 7:03 GMT+02:00 Koji Kawamura :

> Hello,
>
> Although I encountered various Kerberos related error, I haven't
> encountered that one.
> I tried to reproduce the same error by changing Kerberos related
> configuration, but to no avail.
> I recommend enabling Kerberos debug option for further debugging.
>
> You can add the option at nifi/conf/bootstrap.conf:
> java.arg.19=-Dsun.security.krb5.debug=true
>
> Then debug logs are written to nifi/logs/nifi-bootstap.log
>
> Thanks,
> Koji
>
> On Tue, May 29, 2018 at 10:31 PM, Vishal Dutt 
> wrote:
> > Hi ,
> >
> >
> >
> > We  are getting below error on randomly for few minutes and then goes
> away,
> > its coming in PUThiveql
> >
> >
> >
> >
> >
> > 2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95]
> > org.apache.hive.jdbc.HiveConnection Will try to open client transport
> with
> > JDBC Uri:
> > jdbc:hive2://ctcl-hdpmaster2.msoit.com:1/default;
> principal=hive/_h...@msoit.com
> >
> > 2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95]
> > o.apache.thrift.transport.TSaslTransport SASL negotiation failure
> >
> > javax.security.sasl.SaslException: GSS initiate failed
> >
> > at
> > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(
> GssKrb5Client.java:211)
> >
> > at
> > org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(
> TSaslClientTransport.java:94)
> >
> > at
> > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> >
> > at
> > org.apache.thrift.transport.TSaslClientTransport.open(
> TSaslClientTransport.java:37)
> >
> > at
> > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$
> 1.run(TUGIAssumingTransport.java:52)
> >
> > at
> > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$
> 1.run(TUGIAssumingTransport.java:49)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> >
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1656)
> >
> > at
> > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.
> open(TUGIAssumingTransport.java:49)
> >
> > at
> > org.apache.hive.jdbc.HiveConnection.openTransport(
> HiveConnection.java:204)
> >
> > at
> > org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
> >
> > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
> >
> > at
> > org.apache.commons.dbcp.DriverConnectionFactory.createConnection(
> DriverConnectionFactory.java:38)
> >
> > at
> > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(
> PoolableConnectionFactory.java:582)
> >
> > at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
> GenericObjectPool.java:1148)
> >
> > at
> > org.apache.commons.dbcp.PoolingDataSource.getConnection(
> PoolingDataSource.java:106)
> >
> > at
> > org.apache.commons.dbcp.BasicDataSource.getConnection(
> BasicDataSource.java:1044)
> >
> > at
> > org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(
> HiveConnectionPool.java:355)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> >
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1656)
> >
> > at
> > org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(
> HiveConnectionPool.java:355)
> >
> > at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
> >
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:498)
> >
> > at
> > org.apache.nifi.controller.service.StandardControllerServiceInvoc
> ationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
> >
> > at com.sun.proxy.$Proxy97.getConnection(Unknown Source)
> >
> > at
> > org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(
> PutHiveQL.java:191)
> >
> > at org.apache.nifi.processor.util.pattern.Put.onTrigger(
> Put.java:96)
> >
> > at
> > org.apache.nifi.processors.hive.PutHiveQL.lambda$
> onTrigger$6(PutHiveQL.java:274)
> >
> > at
> > org.apache.nifi.processor.util.pattern.PartialFunctions.
> onTrigger(PartialFunctions.java:114)
> >
> > at
> > org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(
> RollbackOnFailure.java:184)
> >
> > at
> > org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274)
> >
> > at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1147)
> >
> > at
> > org.apache.nifi.controller.tasks.ConnectableTask.invoke(
>

Re: Hive connection Pool error

2018-05-29 Thread Koji Kawamura 
Hello,

Although I encountered various Kerberos related error, I haven't
encountered that one.
I tried to reproduce the same error by changing Kerberos related
configuration, but to no avail.
I recommend enabling Kerberos debug option for further debugging.

You can add the option at nifi/conf/bootstrap.conf:
java.arg.19=-Dsun.security.krb5.debug=true

Then debug logs are written to nifi/logs/nifi-bootstap.log

Thanks,
Koji

On Tue, May 29, 2018 at 10:31 PM, Vishal Dutt  wrote:
> Hi ,
>
>
>
> We  are getting below error on randomly for few minutes and then goes away,
> its coming in PUThiveql
>
>
>
>
>
> 2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://ctcl-hdpmaster2.msoit.com:1/default;principal=hive/_h...@msoit.com
>
> 2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95]
> o.apache.thrift.transport.TSaslTransport SASL negotiation failure
>
> javax.security.sasl.SaslException: GSS initiate failed
>
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
>
> at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
>
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
>
> at
> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>
> at
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>
> at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
>
> at
> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)
>
> at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
>
> at com.sun.proxy.$Proxy97.getConnection(Unknown Source)
>
> at
> org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191)
>
> at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96)
>
> at
> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274)
>
> at
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>
> at
> org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
>
> at
> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
>
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
>

Hive connection Pool error

2018-05-29 Thread Vishal Dutt 
Hi ,

We  are getting below error on randomly for few minutes and then goes away, its 
coming in PUThiveql


2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95] 
org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC 
Uri: 
jdbc:hive2://ctcl-hdpmaster2.msoit.com:1/default;principal=hive/_h...@msoit.com
2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95] 
o.apache.thrift.transport.TSaslTransport SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at 
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at 
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)
at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy97.getConnection(Unknown Source)
at 
org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191)
at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96)
at 
org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274)
at 
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at 
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at 
org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism 
level: Failed to find any Kerberos tgt)
at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at