Thank you Matt and Joe for your help. On Wed, Feb 17, 2016 at 4:22 PM, Matt Burgess <[email protected]> wrote:
> Here's a Gist template that uses Joe's approach of RouteOnAttribute then > UpdateAttribute to generate URLs with the use case you described: > https://gist.github.com/mattyb149/8fd87efa13388888a70c > > On Tue, Feb 16, 2016 at 9:51 PM, Joe Witt <[email protected]> wrote: > >> Jeff, >> >> For each of the input files could it be that you would pull data from >> multiple URLs? >> >> Have you had a chance to learn about the NiFi Expression language? >> That will come in quite handy for constructing the URL used in >> InvokeHTTP. >> >> The general pattern I think makes sense here is: >> - Gather Data >> - Extract Features from data to construct URL >> - Fetch document/response from URL >> >> During 'Gather Data' you acquire the files. >> >> During 'Extract features' you pull out elements of the content of the >> file into flow file attributes. You can use RouteOnAttribute to send >> to an UpdateAttribute processor which constructs a new attribute of >> URL pattern A or URL pattern B respectively. You can also collapse >> that into a single UpdateAttribute possibly using the advanced UI and >> set specific URLs based on patterns of attributes. Lots of ways to >> slice that. >> >> During Fetch document you should be able to just have a single >> InvokeHTTP potentially which looks at some attribute you've defined >> say 'the-url' and specify in InvokeHTTP the remote URL value to be >> "${the-url}" >> >> We should publish a template for this pattern/approach if we've not >> already but let's see how you progress and decide what would be most >> useful for others. >> >> Thanks >> Joe >> >> On Tue, Feb 16, 2016 at 9:36 PM, Jeff - Data Bean Australia >> <[email protected]> wrote: >> > Hi, >> > >> > I got a use case like this: >> > >> > There are two files, say fileA and fileB, both of them contains multiple >> > lines of items and used for generate URLs. However, the algorithm for >> > generating URLs are different. If items come from fileA, the URL >> template >> > looks like this: >> > >> > foo-<item>-foo >> > >> > If items come from fileB, the template looks like this: >> > >> > bar-<item>-foo-<item>-whatever >> > >> > I am going to create a NiFi template to for the Data Flow from reading >> the >> > list file up to downloading data using InvokeHTTP, and place a >> > UpdateAttribute processor in front of the template to feed in different >> file >> > names (I have only two files). >> > >> > The problem I have so far is how to generate the URLs based on different >> > input, so that I can make a general NiFi template for reusability. >> > >> > Thanks, >> > Jeff >> > >> > >> > >> > -- >> > Data Bean - A Big Data Solution Provider in Australia. >> > > -- Data Bean - A Big Data Solution Provider in Australia.
