Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-29 Thread Jeff 
Peter,

They're in separate NARs, and are isolated by different ClassLoaders, so
their state regarding UGI will be separate.  There shouldn't be a problem
there.  The only way I could think of that might create a problem is if
Atlas JARs were added to HDFS using the Additional Classpath Resources
property (from memory, I don't think the Hive processors have that
property), but that also uses a separate (descendant) ClassLoader, and
shouldn't create a problem either.

On Fri, Jul 27, 2018 at 1:29 PM Peter Wicks (pwicks) 
wrote:

> As an aside, while digging around in the code, I noticed that the Atlas
> Reporting Task has its own Hadoop Kerberos authentication logic
> (org.apache.nifi.atlas.security.Kerberos). I’m not using this, but it made
> me wonder if this could cause trouble if Hive (synchronized) and Atlas
> (separate, unsynchronized) were both trying to login from Keytab at the
> same time.
>
>
>
> --Peter
>
>
>
> *From:* Shawn Weeks [mailto:swe...@weeksconsulting.us]
>
> *Sent:* Friday, July 27, 2018 10:29 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing
> after a week
>
>
>
> If you're using the Hortonworks distribution it's fixed in the latest HDF
> 3.x release I think.
>
>
>
> Thanks
>
> Shawn
>
>
> --
>
> *From:* Peter Wicks (pwicks) 
> *Sent:* Friday, July 27, 2018 10:58 AM
> *To:* users@nifi.apache.org
> *Subject:* RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing
> after a week
>
>
>
> Thanks Shawn. Looks like this was fixed in 1.7.0. Will have to upgrade.
>
>
>
> *From:* Shawn Weeks [mailto:swe...@weeksconsulting.us]
> *Sent:* Friday, July 27, 2018 8:07 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing
> after a week
>
>
>
> See NIFI-5134 as there was a known bug with the Hive Connection Pool that
> made it fail once the Kerberos Tickets expired and you lost your connection
> from Hive. If you don't have this patch in your version once the Kerberos
> Tickets reaches the end of it's lifetime the connection pool won't work
> till you restart NiFi.
>
>
>
> Thanks
>
> Shawn
> ----------
>
> *From:* Peter Wicks (pwicks) 
> *Sent:* Friday, July 27, 2018 8:51:54 AM
> *To:* users@nifi.apache.org
> *Subject:* RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing
> after a week
>
>
>
> I don’t believe that is how this code works. Not to say that might not
> work, but I don’t believe that the Kerberos authentication used by NiFi
> processors relies in any way on the tickets that appear in klist.
>
>
>
> While we are only using a single account on this particular server, many
> of our servers use several Kerberos principals/keytab’s. I don’t think that
> doing kinit’s for all of them would work either.
>
>
>
> Thanks,
>
>   Peter
>
>
>
> *From:* Sivaprasanna [mailto:sivaprasanna...@gmail.com
> ]
> *Sent:* Friday, July 27, 2018 3:12 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] Re: Hive w/ Kerberos Authentication starts failing after
> a week
>
>
>
> Did you try executing 'klist' to see if the tickets are there and renewed?
> If expired, try manual kinit and see if that fixes.
>
>
>
> On Fri, Jul 27, 2018 at 1:51 AM Peter Wicks (pwicks) 
> wrote:
>
> We are seeing frequent failures of our Hive DBCP connections after a week
> of use when using Kerberos with Principal/Keytab. We’ve tried with both the
> Credential Service and without (though in looking at the code, there should
> be no difference).
>
>
>
> It looks like the tickets are expiring and renewal is not happening?
>
>
>
> javax.security.sasl.SaslException: GSS initiate failed
>
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at

RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-27 Thread Peter Wicks (pwicks) 
As an aside, while digging around in the code, I noticed that the Atlas 
Reporting Task has its own Hadoop Kerberos authentication logic 
(org.apache.nifi.atlas.security.Kerberos). I'm not using this, but it made me 
wonder if this could cause trouble if Hive (synchronized) and Atlas (separate, 
unsynchronized) were both trying to login from Keytab at the same time.

--Peter

From: Shawn Weeks [mailto:swe...@weeksconsulting.us]
Sent: Friday, July 27, 2018 10:29 AM
To: users@nifi.apache.org
Subject: Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week


If you're using the Hortonworks distribution it's fixed in the latest HDF 3.x 
release I think.



Thanks

Shawn


From: Peter Wicks (pwicks) 
Sent: Friday, July 27, 2018 10:58 AM
To: users@nifi.apache.org
Subject: RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week


Thanks Shawn. Looks like this was fixed in 1.7.0. Will have to upgrade.



From: Shawn Weeks [mailto:swe...@weeksconsulting.us]
Sent: Friday, July 27, 2018 8:07 AM
To: users@nifi.apache.org
Subject: Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week



See NIFI-5134 as there was a known bug with the Hive Connection Pool that made 
it fail once the Kerberos Tickets expired and you lost your connection from 
Hive. If you don't have this patch in your version once the Kerberos Tickets 
reaches the end of it's lifetime the connection pool won't work till you 
restart NiFi.



Thanks

Shawn



From: Peter Wicks (pwicks) mailto:pwi...@micron.com>>
Sent: Friday, July 27, 2018 8:51:54 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week



I don't believe that is how this code works. Not to say that might not work, 
but I don't believe that the Kerberos authentication used by NiFi processors 
relies in any way on the tickets that appear in klist.



While we are only using a single account on this particular server, many of our 
servers use several Kerberos principals/keytab's. I don't think that doing 
kinit's for all of them would work either.



Thanks,

  Peter



From: Sivaprasanna [mailto:sivaprasanna...@gmail.com]
Sent: Friday, July 27, 2018 3:12 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week



Did you try executing 'klist' to see if the tickets are there and renewed? If 
expired, try manual kinit and see if that fixes.



On Fri, Jul 27, 2018 at 1:51 AM Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:

We are seeing frequent failures of our Hive DBCP connections after a week of 
use when using Kerberos with Principal/Keytab. We've tried with both the 
Credential Service and without (though in looking at the code, there should be 
no difference).



It looks like the tickets are expiring and renewal is not happening?



javax.security.sasl.SaslException: GSS initiate failed

at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)

at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)

at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)

at 
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)

at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

at 
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)

at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)

at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)

at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.S

Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-27 Thread Shawn Weeks 
If you're using the Hortonworks distribution it's fixed in the latest HDF 3.x 
release I think.


Thanks

Shawn



From: Peter Wicks (pwicks) 
Sent: Friday, July 27, 2018 10:58 AM
To: users@nifi.apache.org
Subject: RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week


Thanks Shawn. Looks like this was fixed in 1.7.0. Will have to upgrade.



From: Shawn Weeks [mailto:swe...@weeksconsulting.us]
Sent: Friday, July 27, 2018 8:07 AM
To: users@nifi.apache.org
Subject: Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week



See NIFI-5134 as there was a known bug with the Hive Connection Pool that made 
it fail once the Kerberos Tickets expired and you lost your connection from 
Hive. If you don't have this patch in your version once the Kerberos Tickets 
reaches the end of it's lifetime the connection pool won't work till you 
restart NiFi.



Thanks

Shawn



From: Peter Wicks (pwicks) mailto:pwi...@micron.com>>
Sent: Friday, July 27, 2018 8:51:54 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week



I don’t believe that is how this code works. Not to say that might not work, 
but I don’t believe that the Kerberos authentication used by NiFi processors 
relies in any way on the tickets that appear in klist.



While we are only using a single account on this particular server, many of our 
servers use several Kerberos principals/keytab’s. I don’t think that doing 
kinit’s for all of them would work either.



Thanks,

  Peter



From: Sivaprasanna [mailto:sivaprasanna...@gmail.com]
Sent: Friday, July 27, 2018 3:12 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week



Did you try executing 'klist' to see if the tickets are there and renewed? If 
expired, try manual kinit and see if that fixes.



On Fri, Jul 27, 2018 at 1:51 AM Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:

We are seeing frequent failures of our Hive DBCP connections after a week of 
use when using Kerberos with Principal/Keytab. We’ve tried with both the 
Credential Service and without (though in looking at the code, there should be 
no difference).



It looks like the tickets are expiring and renewal is not happening?



javax.security.sasl.SaslException: GSS initiate failed

at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)

at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)

at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)

at 
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)

at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

at 
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)

at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)

at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)

at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)

at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)

at sun.reflect.GeneratedMethodAccessor515.invoke(Unknown Source)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)



Thanks,

Peter

Re: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-27 Thread Shawn Weeks 
See NIFI-5134 as there was a known bug with the Hive Connection Pool that made 
it fail once the Kerberos Tickets expired and you lost your connection from 
Hive. If you don't have this patch in your version once the Kerberos Tickets 
reaches the end of it's lifetime the connection pool won't work till you 
restart NiFi.


Thanks

Shawn


From: Peter Wicks (pwicks) 
Sent: Friday, July 27, 2018 8:51:54 AM
To: users@nifi.apache.org
Subject: RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a 
week


I don’t believe that is how this code works. Not to say that might not work, 
but I don’t believe that the Kerberos authentication used by NiFi processors 
relies in any way on the tickets that appear in klist.



While we are only using a single account on this particular server, many of our 
servers use several Kerberos principals/keytab’s. I don’t think that doing 
kinit’s for all of them would work either.



Thanks,

  Peter



From: Sivaprasanna [mailto:sivaprasanna...@gmail.com]
Sent: Friday, July 27, 2018 3:12 AM
To: users@nifi.apache.org
Subject: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week



Did you try executing 'klist' to see if the tickets are there and renewed? If 
expired, try manual kinit and see if that fixes.



On Fri, Jul 27, 2018 at 1:51 AM Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:

We are seeing frequent failures of our Hive DBCP connections after a week of 
use when using Kerberos with Principal/Keytab. We’ve tried with both the 
Credential Service and without (though in looking at the code, there should be 
no difference).



It looks like the tickets are expiring and renewal is not happening?



javax.security.sasl.SaslException: GSS initiate failed

at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)

at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)

at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)

at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)

at 
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)

at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

at 
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)

at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)

at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)

at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)

at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)

at sun.reflect.GeneratedMethodAccessor515.invoke(Unknown Source)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)



Thanks,

Peter

RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-27 Thread Peter Wicks (pwicks) 
I don’t believe that is how this code works. Not to say that might not work, 
but I don’t believe that the Kerberos authentication used by NiFi processors 
relies in any way on the tickets that appear in klist.

While we are only using a single account on this particular server, many of our 
servers use several Kerberos principals/keytab’s. I don’t think that doing 
kinit’s for all of them would work either.

Thanks,
  Peter

From: Sivaprasanna [mailto:sivaprasanna...@gmail.com]
Sent: Friday, July 27, 2018 3:12 AM
To: users@nifi.apache.org
Subject: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

Did you try executing 'klist' to see if the tickets are there and renewed? If 
expired, try manual kinit and see if that fixes.

On Fri, Jul 27, 2018 at 1:51 AM Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:
We are seeing frequent failures of our Hive DBCP connections after a week of 
use when using Kerberos with Principal/Keytab. We’ve tried with both the 
Credential Service and without (though in looking at the code, there should be 
no difference).

It looks like the tickets are expiring and renewal is not happening?

javax.security.sasl.SaslException: GSS initiate failed
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at 
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at 
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)
at sun.reflect.GeneratedMethodAccessor515.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Thanks,
Peter