Hi Arnaud, I created a flow that had PublishKafka_0_10 running with dynamic JAAS properties, and also PutHDFS and ListHDFS talking to a kerberized HDFS, and I wasn't able to reproduce the above error. I also stopped and started all of the processors in different orders, but they continued to work.
Is there any pattern for you that causes the problem? For example, does it always happen if you start the Kafka processors first and then the HDFS processors, and not the other way around? Also, can you confirm that your PutHDFS processor is not using the "Additional Classpath Resources" property, and that there were no additional JARs added to NiFi's lib directory? Just want to ensure nothing else is on the classpath of the PutHDFS processor besides what would be expected. Thanks, Bryan On Wed, May 17, 2017 at 10:24 AM, Arnaud G <[email protected]> wrote: > Hi, > > We didn't fully tested it in 1.1 as we were waiting for the dynamic load of > JAAS in the new Kafka consumer/producer processor. We expected that loading > a JAAS when starting Nifi may had impact on all Kerberos processes. > > Thanks, > > Arnaud > > On Wed, May 17, 2017 at 2:31 PM, Bryan Bende <[email protected]> wrote: >> >> Hello, >> >> Thanks for reporting this, we definitely want to figure out what is >> going on here. >> >> Was this flow working fine before using Apache NiFi 1.1.x (or some >> earlier version) and then stopped working when upgrading to 1.2.0? >> >> Thanks, >> >> Bryan >> >> >> On Wed, May 17, 2017 at 4:49 AM, Arnaud G <[email protected]> wrote: >> > Hi! >> > >> > We are currently facing an issue and we have not yet find a potential >> > solution. >> > >> > We are running 1.2 and we have are facing an interaction between the >> > Kafka >> > 10.2 and PutHDFS processors. >> > >> > We are connecting to an external Kafka server that require SASL/PLAIN >> > authentication. To enable this we are using the new Kafka processor with >> > a >> > JAAS configuration in the processor to authenticate, and this is working >> > fine. >> > >> > However on an unrelated flow the PutHDFS processor stopped to work as >> > the >> > processor is now using the JAAS configuration and ignoring the Kerberos >> > configuration of the processor. (the setup are completely unrelated) >> > >> > We tried a lot of configuration but we have not yet find a way to allow >> > Nifi >> > to authenticate to both Kafka 10.2 and HDFS at the same time, it's >> > either >> > one or the other. >> > >> > Are we missing something? Is there a way to ensure that both processor >> > keep >> > their own configuration? >> > >> > Thanks! >> > >> > >> > For reference PutHDFS failure log: >> > >> > 2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8] >> > o.apache.nifi.processors.hadoop.PutHDFS >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to invoke >> > @OnScheduled method due to java.lang.RuntimeException: Failed while >> > executing one of processor's OnScheduled task.; processor will not be >> > scheduled to run for 30 seconds: java.lang.RuntimeException: Failed >> > while >> > executing one of processor's OnScheduled task. >> > >> > java.lang.RuntimeException: Failed while executing one of processor's >> > OnScheduled task. >> > >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480) >> > >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100) >> > >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301) >> > >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > >> > at >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> > >> > at >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> > >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> > >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> > >> > at java.lang.Thread.run(Thread.java:745) >> > >> > Caused by: java.util.concurrent.ExecutionException: >> > java.lang.reflect.InvocationTargetException >> > >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> > >> > at java.util.concurrent.FutureTask.get(FutureTask.java:206) >> > >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463) >> > >> > ... 9 common frames omitted >> > >> > Caused by: java.lang.reflect.InvocationTargetException: null >> > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > >> > at >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > >> > at >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > >> > at >> > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) >> > >> > at >> > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) >> > >> > at >> > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) >> > >> > at >> > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) >> > >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305) >> > >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301) >> > >> > ... 6 common frames omitted >> > >> > Caused by: java.lang.NullPointerException: null >> > >> > at >> > >> > org.apache.kafka.common.security.plain.PlainSaslServer$PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:162) >> > >> > at >> > >> > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.<init>(SaslRpcServer.java:380) >> > >> > at >> > org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:184) >> > >> > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:577) >> > >> > at >> > >> > org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:418) >> > >> > at >> > >> > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:314) >> > >> > at >> > >> > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:124) >> > >> > at >> > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) >> > >> > at >> > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) >> > >> > at >> > org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) >> > >> > at >> > >> > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:181) >> > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) >> > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) >> > >> > at >> > >> > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) >> > >> > at >> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) >> > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) >> > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) >> > >> > at >> > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:304) >> > >> > at >> > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:301) >> > >> > at java.security.AccessController.doPrivileged(Native Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:422) >> > >> > at >> > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) >> > >> > at >> > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getFileSystemAsUser(AbstractHadoopProcessor.java:301) >> > >> > at >> > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:268) >> > >> > at >> > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:200) >> > >> > at >> > org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191) >> > >> > ... 16 common frames omitted >> > >> > > >
