Hi All, Thanks everyone for your suggestions. It was informative also. I will also try the other options. James's suggestion worked.
- *Attribute Strategy*: Keep All Unique Attributes - *Correlation Attribute Name*: filename - *Minimum Number of Entries*: 3 My intention was to show this point to my colleagues. "New processors can be added to the flow without disturbing the existing flow in Production which is more illustrative in parallel flow" Thanks Lee and Andy. Your points added to my knowledge. On Wed, Jun 1, 2016 at 11:53 PM, Lee Laim <[email protected]> wrote: > Kavinaesh, > > You can also increase the concurrency of the the three sequential ESC > processors in their respective scheduling configurations. This will run > multiple flowfiles through the 3 sequential verification steps, > concurrently, increasing overall throughput. > > > Lee Laim > > > > On Jun 2, 2016, at 12:20 AM, Andy LoPresto <[email protected]> > wrote: > > James mentioned a good point. With a very specific merge strategy, you > might be able to achieve this. However, as MergeContent [1] requires a > single incoming connection in order to successfully merge the flowfiles, > you would likely need to join the multiple incoming connections into a > Funnel [2] first. I am not guaranteeing this will work, but I would > encourage you to explore it. > > [1] > http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html > [2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html > > Andy LoPresto > [email protected] > [email protected] > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 1, 2016, at 23:12, Andy LoPresto <[email protected]> > wrote: > > The "parallel" flow you have written isn't actually parallel, it's just > "independent". Each of the three processors will perform its intended > function and pass a flowfile containing *the JSON element it is responsible > for* to the destination. Unfortunately, the two components that might seem > helpful -- a funnel or MergeContent processor -- do not achieve what you > are looking for. > > If parallel performance of the three ExecuteStreamCommand processors is > really that important, I would recommend using the ExecuteScript processor > with a Groovy script that uses an individual thread for each action (either > via standard thread management or something like GPars for easy parallel > execution). > > In this case, barring an absolute requirement for "parallel" activity, I > would recommend using linear connections between the three ESC processors, > and getting the final result of the additive operation into the destination > processor. > > Andy LoPresto > [email protected] > [email protected] > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 1, 2016, at 22:25, Kavinaesh Dheenadayalan <[email protected]> > wrote: > > Hi Team, > > I am new to nifi and working on a demo to my colleagues. > > I am creating a json by using this flow. The outputs of > ExecuteStreamCommand processor which are json attributes are combined to > create a json. > When these three ExecuteStreamCommand processors are connected > sequentially to an AttributesToJSON processor, the flow works fine with all > the three attributes getting populated in the final json. > But when I connect these three ExecuteStreamCommand processors in > parallel, only one random attribute gets populated in the final json. > Please let me know if there is a way to connect these processors in > parallel and get all the attributes in the final json. scheduling or > merging or any other option? > > I have attached the templates, screen shots of the processors for your > reference. > > *Wrong output when the flow is parallel:* > > { "checks": [ { "Check_Type": "File_Delimiter_Validation", *"value": > null* }, { "Check_Type": "File_Header_Validation", "value": "0" }, { > "Check_Type": "File_Trailer_Validation", *"value": null *} ], > "fileName": "input.txt" } > > > *Correct output when the flow is sequential:* > > { "checks": [ { "Check_Type": "File_Delimiter_Validation", *"value": > "50"* }, { "Check_Type": "File_Header_Validation", "value": "0" }, { > "Check_Type": "File_Trailer_Validation", *"value": "100" *} ], > "fileName": "input.txt" } > > <parallel not working.jpg> > > <sequential working.jpg> > > <attributesJson.JPG> > > <ExecuteStreamCommand.JPG> > > <sequential_flow.xml> > > <parallel_flow.xml> > >
