Hi Vikram, I replied to your question on Stack Overflow. I’m pasting the same answer here for the list.
> Your concern about having too many attributes in memory should not be an > issue here; having 30 attributes per flowfile is higher than usual, but if > these are all strings between 0 - ~100-200 characters, there should be > minimal impact. If you start trying to extract KB worth of data from the > flowfile content to the attributes on each flowfile, you will see increased > heap usage, but the framework should still be able to handle this until you > reach very high throughput (1000's of flowfiles per second on commodity > hardware like a modern laptop). > > You may want to investigate ReplaceTextWithMapping [1], as that processor can > load from a definition file and handle many replace operations using a single > processor. > > It is usually a flow design "smell" to have multiple copies of the same flow > process with different configuration values (with the occasional exception of > database interaction). Rather, see if there is a way you can genericize the > process and populate the relevant values for each flowfile using variable > population [2] (from the incoming flowfile attributes, the variable registry > [3], environment variables, etc.). [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ReplaceTextWithMapping/ [2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Using_Custom_Properties [3] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 26, 2019, at 7:51 AM, vikramas <[email protected]> wrote: > > I'm trying to integrate nifi REST API's with my application. So by mapping > input and output from my application, I am trying to call nifi REST api for > flow creation. So, in my use case most of the times I will extract the JSON > values and will apply expression languages. > > So, for simplifying all the use-cases I am using evaluate JSONpath processor > for fetching all attributes using jsonpath and apply expression language > function on that in extract processor. Below is the flow diagram regarding > that. > > <http://apache-nifi-developer-list.39713.n7.nabble.com/file/t1186/jsonpath_attributes.png> > > > Is it the right approach because for JSON to JSON manipulation having 30 > keys this is the simplest way, and as I am trying to integrate nifi REST > API's with my application I cannot generate JOLT transformation logic > dynamically based on the user mapping. > > So, in this case, does the usage of evaluating JSONpath processor creates > any performance issues for about 50 use case with different transformation > logic because as I saw in documentation attribute usage creates > performance(regarding memory) issues. > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
