I will share what would interest me. The HDFS processor today runs with
authority matching the userid that NiFi is running as. Interactions with HDFS
are via that userid, which limits what it can access. Now granted there are two
options with the current PutHDFS processor (I believe). If you have a
Kerberized cluster, you can use those credentials. However if you don't have
Kerberos on your cluster then you can grant the user running NiFi to be a HDFS
superuser and use the properties to set permissions on the files after the fact.
Providing a processor for WebHDFS or Knox would offer several things that I can
tell: - Not needing the core-site.xml and hdfs-site.xml files would be one
advantage to some sites. Coordinating those between all of your Hadoop
clusters and NiFi clusters could become cumbersome.
- For target clusters that might have firewalls, being able to funnel through
Knox Gateway offers some advantage (although possibly at the cost of
performance or scalability). - For me, the thing I'd like in a Knox Gateway
processor is the ability to specify the id/pw in the definition. I have my Knox
linked with Active Directory for HDFS REST API calls so passing credentials
from the Put processor would be useful since each NFM could use whatever
application credentials made sense for a particular flow.
Thanks,Tom
From: larry mccay <[email protected]>
To: [email protected]
Sent: Thursday, April 21, 2016 6:34 PM
Subject: Re: Apache NiFi - WebHDFS
Any WebHDFS processor should make the URL and credentials configurable so that
it could go direct to WebHDFS or through the Knox Gateway.
On Thu, Apr 21, 2016 at 6:11 PM, Tom Stewart <[email protected]> wrote:
What about Knox Gateway?
> On Apr 21, 2016, at 3:21 PM, Kumiko Yada <[email protected]> wrote:
>
> Will do.
>
> Thanks
> Kumiko
>
> -----Original Message-----
> From: Joe Witt [mailto:[email protected]]
> Sent: Thursday, April 21, 2016 12:45 PM
> To: [email protected]
> Subject: Re: Apache NiFi - WebHDFS
>
> Kumiko,
>
> Not that I am aware of. If you do end up doing so and are interested in
> contributing please let us know.
>
> Thanks
> Joe
>
>> On Thu, Apr 21, 2016 at 3:43 PM, Kumiko Yada <[email protected]> wrote:
>> Hello,
>>
>>
>>
>> Has anyone written the custom process for WebHDFS?
>>
>>
>>
>> Thanks
>>
>> Kumiko
