You could certainly do it with InvokeHTTP but your right it would be extremely overcomplicated and a mess of processors that would most likely be error prone for the enduser trying to configure them for their needs.
I would like to see something like 4 processors to incorporate all of the WebHDFS commands. Since the commands are already broken in GET, POST, PUT, and DELETE "buckets" maybe have a processor for each one of those "buckets" of operations. I think if you try to cram all of the possible operations into a single processor the configuration becomes unnecessarily difficult for the end user. On Thu, Apr 21, 2016 at 8:48 PM, Simon Ball <sb...@hortonworks.com> wrote: > For webhdfs it could be difficult to build out with InvokeHttp, because of > the slightly unusual way Hadoop uses http redirects. The principal is that > you send a request without payload, get the redirect then send the request > dictated by the redirect, but with the payload (redirect should be to a > datanodes, original request to namenode). This is not strictly speaking > correct http behaviour, so might be harder to implement with InvokeHttp. On > that basis I would probably vote specific processor. > > Simon > > > On 22 Apr 2016, at 01:37, Jeremy Dyer <jdy...@gmail.com> wrote: > > Yep all of those reasons make perfect sense to me. Now the question > becomes is this something where we create new processors or just build out > templates using existing processors like InvokeHTTP that we make publicly > available? My vote would probably be for just making the processors but I > would love to hear arguments for one or the other. > > On Thu, Apr 21, 2016 at 8:15 PM, larry mccay <lmc...@apache.org> wrote: > >> All valid points. >> Of course storing credentials in clear text in the definition is less >> than ideal but we could figure something out there as well. >> >> On Thu, Apr 21, 2016 at 7:50 PM, Tom Stewart <stewartthom...@yahoo.com> >> wrote: >> >>> I will share what would interest me. The HDFS processor today runs with >>> authority matching the userid that NiFi is running as. Interactions with >>> HDFS are via that userid, which limits what it can access. Now granted >>> there are two options with the current PutHDFS processor (I believe). If >>> you have a Kerberized cluster, you can use those credentials. However if >>> you don't have Kerberos on your cluster then you can grant the user running >>> NiFi to be a HDFS superuser and use the properties to set permissions on >>> the files after the fact. >>> >>> Providing a processor for WebHDFS or Knox would offer several things >>> that I can tell: >>> - Not needing the core-site.xml and hdfs-site.xml files would be one >>> advantage to some sites. Coordinating those between all of your Hadoop >>> clusters and NiFi clusters could become cumbersome. >>> - For target clusters that might have firewalls, being able to funnel >>> through Knox Gateway offers some advantage (although possibly at the cost >>> of performance or scalability). >>> - For me, the thing I'd like in a Knox Gateway processor is the >>> ability to specify the id/pw in the definition. I have my Knox linked with >>> Active Directory for HDFS REST API calls so passing credentials from the >>> Put processor would be useful since each NFM could use whatever application >>> credentials made sense for a particular flow. >>> >>> Thanks, >>> Tom >>> >>> ------------------------------ >>> *From:* larry mccay <lmc...@apache.org> >>> *To:* users@nifi.apache.org >>> *Sent:* Thursday, April 21, 2016 6:34 PM >>> *Subject:* Re: Apache NiFi - WebHDFS >>> >>> Any WebHDFS processor should make the URL and credentials configurable >>> so that it could go direct to WebHDFS or through the Knox Gateway. >>> >>> On Thu, Apr 21, 2016 at 6:11 PM, Tom Stewart <stewartthom...@yahoo.com> >>> wrote: >>> >>> What about Knox Gateway? >>> >>> > On Apr 21, 2016, at 3:21 PM, Kumiko Yada <kumiko.y...@ds-iq.com> >>> wrote: >>> > >>> > Will do. >>> > >>> > Thanks >>> > Kumiko >>> > >>> > -----Original Message----- >>> > From: Joe Witt [mailto:joe.w...@gmail.com] >>> > Sent: Thursday, April 21, 2016 12:45 PM >>> > To: users@nifi.apache.org >>> > Subject: Re: Apache NiFi - WebHDFS >>> > >>> > Kumiko, >>> > >>> > Not that I am aware of. If you do end up doing so and are interested >>> in contributing please let us know. >>> > >>> > Thanks >>> > Joe >>> > >>> >> On Thu, Apr 21, 2016 at 3:43 PM, Kumiko Yada <kumiko.y...@ds-iq.com> >>> wrote: >>> >> Hello, >>> >> >>> >> >>> >> >>> >> Has anyone written the custom process for WebHDFS? >>> >> >>> >> >>> >> >>> >> Thanks >>> >> >>> >> Kumiko >>> >>> >>> >>> >>> >> >
