By now I can confirm that according to the documentation it must be the old
NiFi 1.2 according to our admin.
Also, following your suggestion to try 1.5 fails as
[INFO] nifi-poi-processors ................................ FAILURE [
6.617 s]
to build on OsX.
@Juan Pablo Gardella:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="xxx""
principal="dddd"
storeKey=true
* useTicketCache=false*
serviceName="zookeeper";
};
should be located in nifi_jaas.conf ?
Anyway, useTicketCache is already set to false.
Schneider, Jonathan <[email protected]> schrieb am Mi., 10. Jan. 2018 um
17:26 Uhr:
> I don’t have the ability to test 1.5.0 RC at this time. I’m not sure how
> well you could over-lay it over HDF’s flavor of Nifi. If I get a chance,
> I’ll try to load a new server and load 1.5 on it, then load my flows into
> it. The problem I can see having is regarding joining the Ambari managed
> Kerberos realm.
>
>
>
> *Jonathan Schneider*
>
> Hadoop/UNIX Administrator, STSC
>
> SCL Health
>
> 17501 W. 98 <https://maps.google.com/?q=17501+W.+98&entry=gmail&source=g>
> th St, Pillars 25-33
>
> Lenexa, KS 66219
>
> P: 913.895.2999 <(913)%20895-2999>
>
> [email protected] <[email protected]%20%0d>
>
> www.sclhealthsystem.org
>
>
>
> [image: image001.jpg]
>
>
>
> *From:* Matt Burgess [mailto:[email protected]]
> *Sent:* Wednesday, January 10, 2018 10:24 AM
>
>
> *To:* [email protected]
> *Subject:* Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
>
>
>
> To Joe's point, this may not be an issue in the upcoming 1.5.0 release as
> it may have been fixed under [1].
>
>
>
> Regards,
>
> Matt
>
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-3472
>
> On Wed, Jan 10, 2018 at 11:14 AM, Georg Heiler <[email protected]>
> wrote:
>
> Regarding the stack trace I will clarify tomorrow. But it is pretty
> similar and cause from ticket renewal failure.
>
> Georg Heiler <[email protected]> schrieb am Mi. 10. Jan. 2018 um
> 17:13:
>
> No. For sure some 3.0.x, but not entirely sure which one. Just realized
> that this must then be nifi 1.2 :(
>
> Schneider, Jonathan <[email protected]> schrieb am Mi. 10. Jan. 2018 um 17:11:
>
> HDF 3.0.0?
>
>
>
> *Jonathan Schneider*
>
> Hadoop/UNIX Administrator, STSC
>
> SCL Health
>
> 17501 W. 98 <https://maps.google.com/?q=17501+W.+98&entry=gmail&source=g>
> th St, Pillars 25-33
>
> Lenexa, KS 66219
>
> P: 913.895.2999 <(913)%20895-2999>
>
> [email protected] <[email protected]%20%0d>
>
> www.sclhealthsystem.org
>
>
>
>
>
> *From:* Georg Heiler [mailto:[email protected]]
>
> *Sent:* Wednesday, January 10, 2018 10:07 AM
>
>
> *To:* [email protected]
> *Subject:* Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
>
>
>
> Hive is 1.2.1
>
> Joe Witt <[email protected]> schrieb am Mi. 10. Jan. 2018 um 17:04:
>
> Interesting. Not what I thought it might have been.
>
> Can you share the following:
> - NiFi config details for the Hive processors and any controller services
> - Hive version.
>
> And then lets see if someone who knows Hive and our NiFi components
> for it far better than I can chime in :)
>
>
>
> On Wed, Jan 10, 2018 at 8:56 AM, Schneider, Jonathan <[email protected]>
> wrote:
> > For reference, the specific error I get is:
> >
> > 2018-01-10 09:55:55,988 ERROR [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=3a4f82fd-015f-1000-0000-00005aa22fb2] Failed to update Hive
> for
> StandardFlowFileRecord[uuid=7ba71cdb-7557-4eab-bd2d-bd89add1c73f,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1515205062419-12378,
> container=default, section=90], offset=342160,
> length=247],offset=0,name=vp_employmentstat.orc,size=247] due to
> java.sql.SQLException: org.apache.thrift.transport.TTransportException:
> org.apache.http.client.ClientProtocolException; it is possible that
> retrying the operation will succeed, so routing to retry:
> java.sql.SQLException: org.apache.thrift.transport.TTransportException:
> org.apache.http.client.ClientProtocolException
> > java.sql.SQLException: org.apache.thrift.transport.TTransportException:
> org.apache.http.client.ClientProtocolException
> > at
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:308)
> > at
> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
> > at
> org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
> > at
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
> > at
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
> > at
> org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
> > at
> org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
> > at
> org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
> > at
> org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
> > at
> org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
> > at
> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
> > at
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
> > at
> org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
> > at
> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
> > at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
> > at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> > at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> > at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.thrift.transport.TTransportException:
> org.apache.http.client.ClientProtocolException
> > at
> org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
> > at
> org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
> > at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> > at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> > at
> org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
> > at
> org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
> > at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1374)
> > at com.sun.proxy.$Proxy174.ExecuteStatement(Unknown Source)
> > at
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:299)
> > ... 24 common frames omitted
> > Caused by: org.apache.http.client.ClientProtocolException: null
> > at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
> > at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
> > at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> > at
> org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
> > ... 35 common frames omitted
> > Caused by: org.apache.http.HttpException: null
> > at
> org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86)
> > at
> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
> > at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
> > at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> > at
> org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
> > at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> > at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> > ... 38 common frames omitted
> > Caused by: org.apache.http.HttpException: null
> > at
> org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68)
> > at
> org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
> > ... 44 common frames omitted
> > Caused by: java.lang.reflect.UndeclaredThrowableException: null
> > at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884)
> > at
> org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
> > at
> org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
> > ... 45 common frames omitted
> > Caused by: org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)
> > at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> > at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> > at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> > at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> > at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> > at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> > at
> org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
> > at
> org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> > ... 47 common frames omitted
> >
> > Jonathan Schneider
> > Hadoop/UNIX Administrator, STSC
> > SCL Health
> > 17501 W. 98th St
> <https://maps.google.com/?q=17501+W.+98th+St&entry=gmail&source=g>,
> Pillars 25-33
> > Lenexa, KS 66219
> > P: 913.895.2999 <(913)%20895-2999>
> > [email protected]
> > www.sclhealthsystem.org
> >
> >
> >
> >
> > -----Original Message-----
> > From: Joe Witt [mailto:[email protected]]
> > Sent: Wednesday, January 10, 2018 9:55 AM
> > To: [email protected]
> > Subject: Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
> >
> > Cool. This is probably fixed in Apache NiFi 1.5.0 but please share the
> stack dump when it is stuck.
> >
> > bin/nifi.sh dump
> >
> > Then send us the logs dir content.
> >
> > Thanks
> >
> > On Wed, Jan 10, 2018 at 8:54 AM, Schneider, Jonathan <[email protected]>
> wrote:
> >> Joe,
> >>
> >> I can reproduce this easily. Set up a connection to a kerberized Hive
> instance. After 24 hours you will get errors about an expired TGT.
> Restarting the Ni-Fi process is the only way I've found to get it to renew
> the TGT.
> >>
> >> Jonathan Schneider
> >> Hadoop/UNIX Administrator, STSC
> >> SCL Health
> >> 17501 W. 98th St
> <https://maps.google.com/?q=17501+W.+98th+St&entry=gmail&source=g>,
> Pillars 25-33
> >> Lenexa, KS 66219
> >> P: 913.895.2999 <(913)%20895-2999>
> >> [email protected]
> >> www.sclhealthsystem.org
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Joe Witt [mailto:[email protected]]
> >> Sent: Wednesday, January 10, 2018 9:53 AM
> >> To: [email protected]
> >> Subject: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
> >>
> >> *** CAUTION! This email came from outside SCL Health. Do not open
> >> attachments or click links if you do not recognize the sender. ***
> >>
> >> Georg
> >>
> >> We'd need to see what you mean to really understand. Can you please
> share NiFi logs directory content and if the flow is stuck/locked up please
> share a nifi thread dump which will be in the logs if you first run
> bin/nifi.sh dump.
> >>
> >> thanks
> >>
> >> On Wed, Jan 10, 2018 at 8:50 AM, Georg Heiler <
> [email protected]> wrote:
> >>> Hi
> >>> In production I observe problems with ticket renewal for the nifi
> >>> hive processor.
> >>>
> >>> A workaround is to restart the hive service but that doesn't seem
> right.
> >>>
> >>> Is there a real fix for this problem?
> >>>
> >>> Best Georg
> >
>
>
