Hey guys, I also used a flow that has PublishKafka_0_10 and ConsumeKafka_0_10 as well as PutHDFS and ListHDFS all talking to a secured cluster. I've tried various configurations of running vs stopped processors including restarting nifi in each configuration and can't seem to reproduce.
It's strange that you're getting org.apache.kafka.common.securi ty.plain.PlainSaslServer$PlainSaslServerFactory in the list of Sasl service factories. When I put a breakpoint at org.apache.hadoop.security.Sas lRpcServer$FastSaslServerFactory.<init>(SaslRpcServer.java:380) I see several factory instances but the closest to that kafka class is org.apache.hadoop.security.SaslPlainServer$SaslPlainServerFactory. Thanks, Bryan Rosander On Wed, May 17, 2017 at 12:28 PM, Bryan Bende <[email protected]> wrote: > Hi Arnaud, > > I created a flow that had PublishKafka_0_10 running with dynamic JAAS > properties, and also PutHDFS and ListHDFS talking to a kerberized > HDFS, and I wasn't able to reproduce the above error. I also stopped > and started all of the processors in different orders, but they > continued to work. > > Is there any pattern for you that causes the problem? For example, > does it always happen if you start the Kafka processors first and then > the HDFS processors, and not the other way around? > > Also, can you confirm that your PutHDFS processor is not using the > "Additional Classpath Resources" property, and that there were no > additional JARs added to NiFi's lib directory? > > Just want to ensure nothing else is on the classpath of the PutHDFS > processor besides what would be expected. > > Thanks, > > Bryan > > > On Wed, May 17, 2017 at 10:24 AM, Arnaud G <[email protected]> wrote: > > Hi, > > > > We didn't fully tested it in 1.1 as we were waiting for the dynamic load > of > > JAAS in the new Kafka consumer/producer processor. We expected that > loading > > a JAAS when starting Nifi may had impact on all Kerberos processes. > > > > Thanks, > > > > Arnaud > > > > On Wed, May 17, 2017 at 2:31 PM, Bryan Bende <[email protected]> wrote: > >> > >> Hello, > >> > >> Thanks for reporting this, we definitely want to figure out what is > >> going on here. > >> > >> Was this flow working fine before using Apache NiFi 1.1.x (or some > >> earlier version) and then stopped working when upgrading to 1.2.0? > >> > >> Thanks, > >> > >> Bryan > >> > >> > >> On Wed, May 17, 2017 at 4:49 AM, Arnaud G <[email protected]> > wrote: > >> > Hi! > >> > > >> > We are currently facing an issue and we have not yet find a potential > >> > solution. > >> > > >> > We are running 1.2 and we have are facing an interaction between the > >> > Kafka > >> > 10.2 and PutHDFS processors. > >> > > >> > We are connecting to an external Kafka server that require SASL/PLAIN > >> > authentication. To enable this we are using the new Kafka processor > with > >> > a > >> > JAAS configuration in the processor to authenticate, and this is > working > >> > fine. > >> > > >> > However on an unrelated flow the PutHDFS processor stopped to work as > >> > the > >> > processor is now using the JAAS configuration and ignoring the > Kerberos > >> > configuration of the processor. (the setup are completely unrelated) > >> > > >> > We tried a lot of configuration but we have not yet find a way to > allow > >> > Nifi > >> > to authenticate to both Kafka 10.2 and HDFS at the same time, it's > >> > either > >> > one or the other. > >> > > >> > Are we missing something? Is there a way to ensure that both processor > >> > keep > >> > their own configuration? > >> > > >> > Thanks! > >> > > >> > > >> > For reference PutHDFS failure log: > >> > > >> > 2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8] > >> > o.apache.nifi.processors.hadoop.PutHDFS > >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] > >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to invoke > >> > @OnScheduled method due to java.lang.RuntimeException: Failed while > >> > executing one of processor's OnScheduled task.; processor will not be > >> > scheduled to run for 30 seconds: java.lang.RuntimeException: Failed > >> > while > >> > executing one of processor's OnScheduled task. > >> > > >> > java.lang.RuntimeException: Failed while executing one of processor's > >> > OnScheduled task. > >> > > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode. > invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480) > >> > > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode.access$ > 000(StandardProcessorNode.java:100) > >> > > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode$1.run( > StandardProcessorNode.java:1301) > >> > > >> > at > >> > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > >> > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> > > >> > at > >> > > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > >> > > >> > at > >> > > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > >> > > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > >> > > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > >> > > >> > at java.lang.Thread.run(Thread.java:745) > >> > > >> > Caused by: java.util.concurrent.ExecutionException: > >> > java.lang.reflect.InvocationTargetException > >> > > >> > at java.util.concurrent.FutureTask.report(FutureTask. > java:122) > >> > > >> > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > >> > > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode. > invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463) > >> > > >> > ... 9 common frames omitted > >> > > >> > Caused by: java.lang.reflect.InvocationTargetException: null > >> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> > > >> > at > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > >> > > >> > at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > >> > > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > > >> > at > >> > > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations( > ReflectionUtils.java:137) > >> > > >> > at > >> > > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations( > ReflectionUtils.java:125) > >> > > >> > at > >> > > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations( > ReflectionUtils.java:70) > >> > > >> > at > >> > > >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation( > ReflectionUtils.java:47) > >> > > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode$1$1. > call(StandardProcessorNode.java:1305) > >> > > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode$1$1. > call(StandardProcessorNode.java:1301) > >> > > >> > ... 6 common frames omitted > >> > > >> > Caused by: java.lang.NullPointerException: null > >> > > >> > at > >> > > >> > org.apache.kafka.common.security.plain.PlainSaslServer$ > PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:162) > >> > > >> > at > >> > > >> > org.apache.hadoop.security.SaslRpcServer$ > FastSaslServerFactory.<init>(SaslRpcServer.java:380) > >> > > >> > at > >> > org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:184) > >> > > >> > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:577) > >> > > >> > at > >> > > >> > org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtoco > l(NameNodeProxies.java:418) > >> > > >> > at > >> > > >> > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy( > NameNodeProxies.java:314) > >> > > >> > at > >> > > >> > org.apache.hadoop.hdfs.server.namenode.ha. > ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvide > r.java:124) > >> > > >> > at > >> > > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>( > RetryInvocationHandler.java:73) > >> > > >> > at > >> > > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>( > RetryInvocationHandler.java:64) > >> > > >> > at > >> > org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) > >> > > >> > at > >> > > >> > org.apache.hadoop.hdfs.NameNodeProxies.createProxy( > NameNodeProxies.java:181) > >> > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient. > java:678) > >> > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient. > java:619) > >> > > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.initialize( > DistributedFileSystem.java:149) > >> > > >> > at > >> > org.apache.hadoop.fs.FileSystem.createFileSystem( > FileSystem.java:2669) > >> > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) > >> > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) > >> > > >> > at > >> > > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run( > AbstractHadoopProcessor.java:304) > >> > > >> > at > >> > > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run( > AbstractHadoopProcessor.java:301) > >> > > >> > at java.security.AccessController.doPrivileged(Native Method) > >> > > >> > at javax.security.auth.Subject.doAs(Subject.java:422) > >> > > >> > at > >> > > >> > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1698) > >> > > >> > at > >> > > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor. > getFileSystemAsUser(AbstractHadoopProcessor.java:301) > >> > > >> > at > >> > > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor. > resetHDFSResources(AbstractHadoopProcessor.java:268) > >> > > >> > at > >> > > >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor. > abstractOnScheduled(AbstractHadoopProcessor.java:200) > >> > > >> > at > >> > org.apache.nifi.processors.hadoop.PutHDFS.onScheduled( > PutHDFS.java:191) > >> > > >> > ... 16 common frames omitted > >> > > >> > > > > > >
