Glad the tool is capable of what you are looking for, Kavinaesh. Good luck.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 2, 2016, at 2:34 PM, Kavinaesh Dheenadayalan <[email protected]> > wrote: > > Hi All, > > Thanks everyone for your suggestions. It was informative also. I will also > try the other options. > James's suggestion worked. > > Attribute Strategy: Keep All Unique Attributes > Correlation Attribute Name: filename > Minimum Number of Entries: 3 > > My intention was to show this point to my colleagues. "New processors can be > added to the flow without disturbing the existing flow in Production which is > more illustrative in parallel flow" > Thanks Lee and Andy. Your points added to my knowledge. > > > On Wed, Jun 1, 2016 at 11:53 PM, Lee Laim <[email protected] > <mailto:[email protected]>> wrote: > Kavinaesh, > > You can also increase the concurrency of the the three sequential ESC > processors in their respective scheduling configurations. This will run > multiple flowfiles through the 3 sequential verification steps, concurrently, > increasing overall throughput. > > > Lee Laim > > > > On Jun 2, 2016, at 12:20 AM, Andy LoPresto <[email protected] > <mailto:[email protected]>> wrote: > >> James mentioned a good point. With a very specific merge strategy, you might >> be able to achieve this. However, as MergeContent [1] requires a single >> incoming connection in order to successfully merge the flowfiles, you would >> likely need to join the multiple incoming connections into a Funnel [2] >> first. I am not guaranteeing this will work, but I would encourage you to >> explore it. >> >> [1] >> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html >> >> <http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html> >> [2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html >> <https://nifi.apache.org/docs/nifi-docs/html/user-guide.html> >> >> Andy LoPresto >> [email protected] <mailto:[email protected]> >> [email protected] <mailto:[email protected]> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Jun 1, 2016, at 23:12, Andy LoPresto <[email protected] >> <mailto:[email protected]>> wrote: >> >>> The "parallel" flow you have written isn't actually parallel, it's just >>> "independent". Each of the three processors will perform its intended >>> function and pass a flowfile containing *the JSON element it is responsible >>> for* to the destination. Unfortunately, the two components that might seem >>> helpful -- a funnel or MergeContent processor -- do not achieve what you >>> are looking for. >>> >>> If parallel performance of the three ExecuteStreamCommand processors is >>> really that important, I would recommend using the ExecuteScript processor >>> with a Groovy script that uses an individual thread for each action (either >>> via standard thread management or something like GPars for easy parallel >>> execution). >>> >>> In this case, barring an absolute requirement for "parallel" activity, I >>> would recommend using linear connections between the three ESC processors, >>> and getting the final result of the additive operation into the destination >>> processor. >>> >>> Andy LoPresto >>> [email protected] <mailto:[email protected]> >>> [email protected] <mailto:[email protected]> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>> On Jun 1, 2016, at 22:25, Kavinaesh Dheenadayalan <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>>> Hi Team, >>>> >>>> I am new to nifi and working on a demo to my colleagues. >>>> >>>> I am creating a json by using this flow. The outputs of >>>> ExecuteStreamCommand processor which are json attributes are combined to >>>> create a json. >>>> When these three ExecuteStreamCommand processors are connected >>>> sequentially to an AttributesToJSON processor, the flow works fine with >>>> all the three attributes getting populated in the final json. >>>> But when I connect these three ExecuteStreamCommand processors in >>>> parallel, only one random attribute gets populated in the final json. >>>> Please let me know if there is a way to connect these processors in >>>> parallel and get all the attributes in the final json. scheduling or >>>> merging or any other option? >>>> >>>> I have attached the templates, screen shots of the processors for your >>>> reference. >>>> >>>> Wrong output when the flow is parallel: >>>> >>>> { "checks": [ { "Check_Type": "File_Delimiter_Validation", "value": >>>> null }, { "Check_Type": "File_Header_Validation", "value": "0" }, { >>>> "Check_Type": "File_Trailer_Validation", "value": null } ], "fileName": >>>> "input.txt" } >>>> >>>> >>>> Correct output when the flow is sequential: >>>> >>>> { "checks": [ { "Check_Type": "File_Delimiter_Validation", "value": >>>> "50" }, { "Check_Type": "File_Header_Validation", "value": "0" }, { >>>> "Check_Type": "File_Trailer_Validation", "value": "100" } ], "fileName": >>>> "input.txt" } >>>> >>>> <parallel not working.jpg> >>>> <sequential working.jpg> >>>> <attributesJson.JPG> >>>> <ExecuteStreamCommand.JPG> >>>> <sequential_flow.xml> >>>> <parallel_flow.xml> >
signature.asc
Description: Message signed with OpenPGP using GPGMail
