Hi!
We are currently facing an issue and we have not yet find a potential
solution.
We are running 1.2 and we have are facing an interaction between the Kafka
10.2 and PutHDFS processors.
We are connecting to an external Kafka server that require SASL/PLAIN
authentication. To enable this we are using the new Kafka processor with a
JAAS configuration in the processor to authenticate, and this is working
fine.
However on an unrelated flow the PutHDFS processor stopped to work as the
processor is now using the JAAS configuration and ignoring the Kerberos
configuration of the processor. (the setup are completely unrelated)
We tried a lot of configuration but we have not yet find a way to allow
Nifi to authenticate to both Kafka 10.2 and HDFS at the same time, it's
either one or the other.
Are we missing something? Is there a way to ensure that both processor keep
their own configuration?
Thanks!
For reference PutHDFS failure log:
2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8]
o.apache.nifi.processors.hadoop.PutHDFS
PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239]
PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to invoke
@OnScheduled method due to java.lang.RuntimeException: Failed while
executing one of processor's OnScheduled task.; processor will not be
scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while
executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's
OnScheduled task.
at org.apache.nifi.controller.StandardProcessorNode.
invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
at org.apache.nifi.controller.StandardProcessorNode.access$
000(StandardProcessorNode.java:100)
at org.apache.nifi.controller.StandardProcessorNode$1.run(
StandardProcessorNode.java:1301)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$
ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$
ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.
InvocationTargetException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.nifi.controller.StandardProcessorNode.
invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.
invokeMethodsWithAnnotations(ReflectionUtils.java:137)
at org.apache.nifi.util.ReflectionUtils.
invokeMethodsWithAnnotations(ReflectionUtils.java:125)
at org.apache.nifi.util.ReflectionUtils.
invokeMethodsWithAnnotations(ReflectionUtils.java:70)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(
ReflectionUtils.java:47)
at org.apache.nifi.controller.StandardProcessorNode$1$1.
call(StandardProcessorNode.java:1305)
at org.apache.nifi.controller.StandardProcessorNode$1$1.
call(StandardProcessorNode.java:1301)
... 6 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.kafka.common.security.plain.PlainSaslServer$
PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:162)
at org.apache.hadoop.security.SaslRpcServer$
FastSaslServerFactory.<init>(SaslRpcServer.java:380)
at org.apache.hadoop.security.SaslRpcServer.init(
SaslRpcServer.java:184)
at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:577)
at org.apache.hadoop.hdfs.NameNodeProxies.
createNNProxyWithClientProtocol(NameNodeProxies.java:418)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(
NameNodeProxies.java:314)
at org.apache.hadoop.hdfs.server.namenode.ha.
ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvide
r.java:124)
at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(
RetryInvocationHandler.java:73)
at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(
RetryInvocationHandler.java:64)
at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(
NameNodeProxies.java:181)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(
DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(
FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(
AbstractHadoopProcessor.java:304)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(
AbstractHadoopProcessor.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1698)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.
getFileSystemAsUser(AbstractHadoopProcessor.java:301)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.
resetHDFSResources(AbstractHadoopProcessor.java:268)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.
abstractOnScheduled(AbstractHadoopProcessor.java:200)
at org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(
PutHDFS.java:191)
... 16 common frames omitted
