I've gotten closer with grabbing two files and processing them. I still have
something wrong in the paradigm though. Here's what I've got it narrowed down
to (This is in an ExecuteGroovyScript, BTW. I hope to translate it to an
InvokeScriptedProcessor later on, so I can define the transfer end points)
// get two files, always two. Read file1, and write certain fields to file2
flowFileList = session.get(2)
if (flowFileList.size() != 2) return
flowFile1 = flowFileList.get(0)
flowFile2 = flowFileList.get(1)
if(!flowFile1) return
if(!flowFile2) return
flowFile1 = session.read(flowFile1, {inputStream ->
def slurper = new groovy.json.JsonSlurper()
json1 = slurper.parse(inputStream1)
} as OutputStreamCallback)
flowFile2 = session.write(flowFile2, {outputStream ->
outputStream.write("foo plus some data from json1".bytes)
} as OutputStreamCallback)
// I really don't want to TRANSFER flow file 1, I want it to go AWAY, ,but
// I have to do this
session.transfer(flowFile1, REL_SUCCESS) // << isn't needed
session.transfer(flowFile2, REL_SUCCESS)
Both files do get read correctly, and output to success, but Nifi always
throws the error that "transfer relationship not specified", which I gather,
means that the call to transfer failed because one file (probably flowFile1) is
not up to date....
Any thoughts? How do you grab two files at once and then transfer them?
I really only want to transfer just the ONE out, since the data was merged in,
but I can manage with two files if I have to make # inputs = # outputs
Thanks
-----Original Message-----
From: Greene (US), Geoffrey N
Sent: Tuesday, November 17, 2020 8:30 PM
To: [email protected]
Subject: RE: [EXTERNAL] Re: horizontal merge
The data is actually coming from a rest call, which provides json. That is
pretty smooth at this point.
The challenge seems to be one of associating two different numbers records, and
combing them back into one single file (like how do I know when I've processed
all the records in BOTH files to give one single output file. I like your
suggestion of rolling back the sesion and returning, though, I will look into
that, (though it might mean the file has to be processed as one single file,
rather than handling them as splits/merges
I've also been playing with MergeRecords and MergeContent too, and I might be
making some progress. My struggle now is trying to figure out when I know all
records are processed, since I don't have a constant number of results to watch
for. I may end up writing a file appender, and just appending "as I go", so I
don't have to do a count.
-----Original Message-----
From: Matt Burgess [mailto:[email protected]]
Sent: Tuesday, November 17, 2020 4:22 PM
To: [email protected]
Subject: [EXTERNAL] Re: horizontal merge
Geoffrey,
Where are the two flowfiles coming from? This use case is often handled in NiFi
using LookupRecord with one of the LookupService implementations (REST, RDBMS,
CSV, etc.). We don't currently have a mechanism (besides scripting) to do
enrichment/lookups from flowfiles.
For your script, you can do session.get(2) and then check the size of the
returned List. If it is less than 2, you can rollback the session and return
(possibly yielding first if you don't want to check again rapidly).
Regards,
Matt
On Tue, Nov 17, 2020 at 4:13 PM Greene (US), Geoffrey N
<[email protected]> wrote:
>
> I’m trying to glue two flow files together HORIZONTALLY. That is,
>
> Flowfile1
>
> ID,STARTINGLETTER
>
> 1,A
>
> 2,B
>
>
>
> And flowfile2:
>
> ID, WORD
>
> 1,Apple
>
> 2, Ball
>
> 3, Cat
>
>
>
> I want it to become:
>
> ID, STARTINGLETTER, WORD
>
> 1,A,Apple
>
> 2,B,Ball
>
> 3,,Cat
>
>
>
> The only way I’ve been able to figure out how to do this is to write a custom
> “InvokeGroovyProcessor” that takes two flowfiles reads them both, and then
> concatenates them.
>
>
>
> I’m having trouble figuring out how to pop TWO flow files off the
> queue, (The order doesn’t really matter for now), and write one. I’ve
> tried
>
>
>
> @Override
>
> void onTrigger(ProcessContext context, ProcessSessionFactory
> sessionFactory) throws ProcessException {
>
> try {
>
> def session = sessionFactory.createSession()
>
> while (session.getQueueSize() != 2)
>
> {
>
> Thread.sleep(1000)
>
> log.debug("sleeping")
>
> }
>
> // we never get here
>
> log.debug(“found two flow files")
>
> // get BOTH flowfiles
>
> def flowFileList = session.get(2)
>
> def flowFile1 = flowFileList.get(0)
>
> def flowFile2 = flowFileList.get(1)
>
> // now do the glue
>
>
>
> Or Is there a better way?
>
> Thanks!
>
>
>
>
>
>
