The data is actually coming from a rest call, which provides json. That is pretty smooth at this point.
The challenge seems to be one of associating two different numbers records, and combing them back into one single file (like how do I know when I've processed all the records in BOTH files to give one single output file. I like your suggestion of rolling back the sesion and returning, though, I will look into that, (though it might mean the file has to be processed as one single file, rather than handling them as splits/merges I've also been playing with MergeRecords and MergeContent too, and I might be making some progress. My struggle now is trying to figure out when I know all records are processed, since I don't have a constant number of results to watch for. I may end up writing a file appender, and just appending "as I go", so I don't have to do a count. -----Original Message----- From: Matt Burgess [mailto:[email protected]] Sent: Tuesday, November 17, 2020 4:22 PM To: [email protected] Subject: [EXTERNAL] Re: horizontal merge Geoffrey, Where are the two flowfiles coming from? This use case is often handled in NiFi using LookupRecord with one of the LookupService implementations (REST, RDBMS, CSV, etc.). We don't currently have a mechanism (besides scripting) to do enrichment/lookups from flowfiles. For your script, you can do session.get(2) and then check the size of the returned List. If it is less than 2, you can rollback the session and return (possibly yielding first if you don't want to check again rapidly). Regards, Matt On Tue, Nov 17, 2020 at 4:13 PM Greene (US), Geoffrey N <[email protected]> wrote: > > I’m trying to glue two flow files together HORIZONTALLY. That is, > > Flowfile1 > > ID,STARTINGLETTER > > 1,A > > 2,B > > > > And flowfile2: > > ID, WORD > > 1,Apple > > 2, Ball > > 3, Cat > > > > I want it to become: > > ID, STARTINGLETTER, WORD > > 1,A,Apple > > 2,B,Ball > > 3,,Cat > > > > The only way I’ve been able to figure out how to do this is to write a custom > “InvokeGroovyProcessor” that takes two flowfiles reads them both, and then > concatenates them. > > > > I’m having trouble figuring out how to pop TWO flow files off the > queue, (The order doesn’t really matter for now), and write one. I’ve > tried > > > > @Override > > void onTrigger(ProcessContext context, ProcessSessionFactory > sessionFactory) throws ProcessException { > > try { > > def session = sessionFactory.createSession() > > while (session.getQueueSize() != 2) > > { > > Thread.sleep(1000) > > log.debug("sleeping") > > } > > // we never get here > > log.debug(“found two flow files") > > // get BOTH flowfiles > > def flowFileList = session.get(2) > > def flowFile1 = flowFileList.get(0) > > def flowFile2 = flowFileList.get(1) > > // now do the glue > > > > Or Is there a better way? > > Thanks! > > > > > > > > Geoffrey Greene > > Senior Software Ninjaneer > > (703) 414 2421 > > The Boeing Company > >
