mail was sent before completion:
Security Protocol: SASL_SSL
Kerberos Service Name: myuser (please note that its mandatory to put
something here or the process fail)
SSL Context Service: KafkaSSLcontext
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainSaslServer
require username="kauser" password="XXXXX";
sasl.mechanism: PLAIN
This configuration is working fine we can consume the topics without any
issue.
4) Here is the configuration of the PutHDFS :
Hadoop Configuration Ressources:
/etc/hadoop/conf/hdfs-site.xml;/etc/hadoop/conf/core-site.xml
Kerberos Principal: [email protected]
Kerberos Keytab: /etc/security/keytabs/user1.keytab
Additional classpath is empty.
Please note that the Kafka is an external instance (and using SASL/PLAIN)
and not using the same Kerberos configuration than HDFS. I suppose that if
we were using the same Kerberos domain everything will be working fine.
A.
On Thu, May 18, 2017 at 9:22 AM, Arnaud G <[email protected]> wrote:
> Hi again,
>
> Here are more details:
>
> 1) We are redoing the test on a test cluster which is plain 1.2 vanilla
> (upgraded from 1.1 but without any additional JAR, bootstrap is the package
> version.
> 2) The org.apache.kafka.common.security.plain.PlainSaslServer is coming
> from our JAAS configuration and form understanding is part of the standard
> 0.10 client library
> 3) Here is the configuration of our ConsumeKafka_0_10:
>
>
>
>
> On Wed, May 17, 2017 at 6:50 PM, Bryan Rosander <[email protected]>
> wrote:
>
>> Hey guys,
>>
>> I also used a flow that has PublishKafka_0_10 and ConsumeKafka_0_10 as
>> well as PutHDFS and ListHDFS all talking to a secured cluster. I've tried
>> various configurations of running vs stopped processors including
>> restarting nifi in each configuration and can't seem to reproduce.
>>
>> It's strange that you're getting org.apache.kafka.common.securi
>> ty.plain.PlainSaslServer$PlainSaslServerFactory in the list of Sasl
>> service factories.
>>
>> When I put a breakpoint at org.apache.hadoop.security.Sas
>> lRpcServer$FastSaslServerFactory.<init>(SaslRpcServer.java:380) I see
>> several factory instances but the closest to that kafka class
>> is org.apache.hadoop.security.SaslPlainServer$SaslPlainServerFactory.
>>
>> Thanks,
>> Bryan Rosander
>>
>> On Wed, May 17, 2017 at 12:28 PM, Bryan Bende <[email protected]> wrote:
>>
>>> Hi Arnaud,
>>>
>>> I created a flow that had PublishKafka_0_10 running with dynamic JAAS
>>> properties, and also PutHDFS and ListHDFS talking to a kerberized
>>> HDFS, and I wasn't able to reproduce the above error. I also stopped
>>> and started all of the processors in different orders, but they
>>> continued to work.
>>>
>>> Is there any pattern for you that causes the problem? For example,
>>> does it always happen if you start the Kafka processors first and then
>>> the HDFS processors, and not the other way around?
>>>
>>> Also, can you confirm that your PutHDFS processor is not using the
>>> "Additional Classpath Resources" property, and that there were no
>>> additional JARs added to NiFi's lib directory?
>>>
>>> Just want to ensure nothing else is on the classpath of the PutHDFS
>>> processor besides what would be expected.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Wed, May 17, 2017 at 10:24 AM, Arnaud G <[email protected]>
>>> wrote:
>>> > Hi,
>>> >
>>> > We didn't fully tested it in 1.1 as we were waiting for the dynamic
>>> load of
>>> > JAAS in the new Kafka consumer/producer processor. We expected that
>>> loading
>>> > a JAAS when starting Nifi may had impact on all Kerberos processes.
>>> >
>>> > Thanks,
>>> >
>>> > Arnaud
>>> >
>>> > On Wed, May 17, 2017 at 2:31 PM, Bryan Bende <[email protected]> wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> Thanks for reporting this, we definitely want to figure out what is
>>> >> going on here.
>>> >>
>>> >> Was this flow working fine before using Apache NiFi 1.1.x (or some
>>> >> earlier version) and then stopped working when upgrading to 1.2.0?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Bryan
>>> >>
>>> >>
>>> >> On Wed, May 17, 2017 at 4:49 AM, Arnaud G <[email protected]>
>>> wrote:
>>> >> > Hi!
>>> >> >
>>> >> > We are currently facing an issue and we have not yet find a
>>> potential
>>> >> > solution.
>>> >> >
>>> >> > We are running 1.2 and we have are facing an interaction between the
>>> >> > Kafka
>>> >> > 10.2 and PutHDFS processors.
>>> >> >
>>> >> > We are connecting to an external Kafka server that require
>>> SASL/PLAIN
>>> >> > authentication. To enable this we are using the new Kafka processor
>>> with
>>> >> > a
>>> >> > JAAS configuration in the processor to authenticate, and this is
>>> working
>>> >> > fine.
>>> >> >
>>> >> > However on an unrelated flow the PutHDFS processor stopped to work
>>> as
>>> >> > the
>>> >> > processor is now using the JAAS configuration and ignoring the
>>> Kerberos
>>> >> > configuration of the processor. (the setup are completely unrelated)
>>> >> >
>>> >> > We tried a lot of configuration but we have not yet find a way to
>>> allow
>>> >> > Nifi
>>> >> > to authenticate to both Kafka 10.2 and HDFS at the same time, it's
>>> >> > either
>>> >> > one or the other.
>>> >> >
>>> >> > Are we missing something? Is there a way to ensure that both
>>> processor
>>> >> > keep
>>> >> > their own configuration?
>>> >> >
>>> >> > Thanks!
>>> >> >
>>> >> >
>>> >> > For reference PutHDFS failure log:
>>> >> >
>>> >> > 2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8]
>>> >> > o.apache.nifi.processors.hadoop.PutHDFS
>>> >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239]
>>> >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to invoke
>>> >> > @OnScheduled method due to java.lang.RuntimeException: Failed while
>>> >> > executing one of processor's OnScheduled task.; processor will not
>>> be
>>> >> > scheduled to run for 30 seconds: java.lang.RuntimeException: Failed
>>> >> > while
>>> >> > executing one of processor's OnScheduled task.
>>> >> >
>>> >> > java.lang.RuntimeException: Failed while executing one of
>>> processor's
>>> >> > OnScheduled task.
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.invokeTaskA
>>> sCancelableFuture(StandardProcessorNode.java:1480)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.access$000(
>>> StandardProcessorNode.java:100)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode$1.run(Stand
>>> ardProcessorNode.java:1301)
>>> >> >
>>> >> > at
>>> >> > java.util.concurrent.Executors$RunnableAdapter.call(Executor
>>> s.java:511)
>>> >> >
>>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1142)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:617)
>>> >> >
>>> >> > at java.lang.Thread.run(Thread.java:745)
>>> >> >
>>> >> > Caused by: java.util.concurrent.ExecutionException:
>>> >> > java.lang.reflect.InvocationTargetException
>>> >> >
>>> >> > at java.util.concurrent.FutureTas
>>> k.report(FutureTask.java:122)
>>> >> >
>>> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:206)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.invokeTaskA
>>> sCancelableFuture(StandardProcessorNode.java:1463)
>>> >> >
>>> >> > ... 9 common frames omitted
>>> >> >
>>> >> > Caused by: java.lang.reflect.InvocationTargetException: null
>>> >> >
>>> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> >> >
>>> >> > at java.lang.reflect.Method.invoke(Method.java:498)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnota
>>> tions(ReflectionUtils.java:137)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnota
>>> tions(ReflectionUtils.java:125)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnota
>>> tions(ReflectionUtils.java:70)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnota
>>> tion(ReflectionUtils.java:47)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode$1$1.call(St
>>> andardProcessorNode.java:1305)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode$1$1.call(St
>>> andardProcessorNode.java:1301)
>>> >> >
>>> >> > ... 6 common frames omitted
>>> >> >
>>> >> > Caused by: java.lang.NullPointerException: null
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.kafka.common.security.plain.PlainSaslServer$Plain
>>> SaslServerFactory.getMechanismNames(PlainSaslServer.java:162)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFacto
>>> ry.<init>(SaslRpcServer.java:380)
>>> >> >
>>> >> > at
>>> >> > org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.
>>> java:184)
>>> >> >
>>> >> > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:577)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClie
>>> ntProtocol(NameNodeProxies.java:418)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(Name
>>> NodeProxies.java:314)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailover
>>> ProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:124)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(Ret
>>> ryInvocationHandler.java:73)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(Ret
>>> ryInvocationHandler.java:64)
>>> >> >
>>> >> > at
>>> >> > org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeP
>>> roxies.java:181)
>>> >> >
>>> >> > at org.apache.hadoop.hdfs.DFSClie
>>> nt.<init>(DFSClient.java:678)
>>> >> >
>>> >> > at org.apache.hadoop.hdfs.DFSClie
>>> nt.<init>(DFSClient.java:619)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Dist
>>> ributedFileSystem.java:149)
>>> >> >
>>> >> > at
>>> >> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.
>>> java:2669)
>>> >> >
>>> >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>> >> >
>>> >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.
>>> run(AbstractHadoopProcessor.java:304)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.
>>> run(AbstractHadoopProcessor.java:301)
>>> >> >
>>> >> > at java.security.AccessController.doPrivileged(Native
>>> Method)
>>> >> >
>>> >> > at javax.security.auth.Subject.doAs(Subject.java:422)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>> upInformation.java:1698)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.ge
>>> tFileSystemAsUser(AbstractHadoopProcessor.java:301)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.re
>>> setHDFSResources(AbstractHadoopProcessor.java:268)
>>> >> >
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.ab
>>> stractOnScheduled(AbstractHadoopProcessor.java:200)
>>> >> >
>>> >> > at
>>> >> > org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDF
>>> S.java:191)
>>> >> >
>>> >> > ... 16 common frames omitted
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>
>>
>
