Re: Nifi flow hanging . Groovy JsonSlurper Problem?

Timothy, I haven't seen anything that can cause this to hang, in the Groovy source code it might seem to "hang" [1] if there's a crazy large input; how big are your flow files going into the ExecuteScript processor? If size is not the issue, then perhaps there's an assumption about character

Re: Cluster Crashing When Nodes Unresponsive?

Hi what version of NIFI are you on? In my experience it’s usually resource issue and most of the time it’s disk I/O. On Tue, May 1, 2018 at 4:43 PM Jon Logan wrote: > We are running into issues where if a node is unresponsive for whatever > reason, the entire cluster seems

Re: Am I doing this right? with regarding to records

Oops. It is. I am not as up to speed on *Record* processors. My bad. On May 1, 2018 at 17:09:40, Bryan Bende (bbe...@gmail.com) wrote: Isn't that what PartitionRecord + RouteOnAttribute already does? On Tue, May 1, 2018 at 4:58 PM, Otto Fowler wrote: > Maybe a

Re: Am I doing this right? with regarding to records

Isn't that what PartitionRecord + RouteOnAttribute already does? On Tue, May 1, 2018 at 4:58 PM, Otto Fowler wrote: > Maybe a group and dispatch processor would help, > > JsonPath -> key > Group by key > key -> route > > > On May 1, 2018 at 16:30:28, Bryan Bende

Re: Am I doing this right? with regarding to records

Maybe a group and dispatch processor would help, JsonPath -> key Group by key key -> route On May 1, 2018 at 16:30:28, Bryan Bende (bbe...@gmail.com) wrote: I see, so the partition is helping if you want to route based on the partition and is also giving you the attribute. Right now it is the

Re: Hadoop Processors Leak Classes

This problem has been captured by the issue: https://issues.apache.org/jira/browse/NIFI-5136 Thanks for the help and the quick work finding the problems. -Dann On Fri, Apr 27, 2018 at 3:51 PM Bryan Bende wrote: > Hey Dann, > > I've spent most of the afternoon debugging this

Am I doing this right? with regarding to records

Hello all, I have one file on local disk with thousands of lines each representing valid JSON object. My flow is like this: GetFile > SplitText > PartitionRecord ( based on a key ) > MergeRecord > PutElasticSearchRecord This works well, however, I seem to bottleneck at PartitionRecord So I

Re: Efficiently caching API results in a NiFi controller service

I second caffeine as well, I have used it very effectively in controller services. > On May 1, 2018, at 9:12 AM, Otto Fowler wrote: > > We used guava in Apache Metron, but have switched to > https://github.com/ben-manes/caffeine

Re: Efficiently caching API results in a NiFi controller service

We used guava in Apache Metron, but have switched to https://github.com/ben-manes/caffeine. I would recommend taking a look at that too. On May 1, 2018 at 10:09:00, Charlie Meyer (charlie.me...@civitaslearning.com) wrote: We do something very similar in a custom controller service and utilize

Re: Efficiently caching API results in a NiFi controller service

Thanks Bryan - Should I be assuming that my service’s local map needs to be thread-safe, or would all service calls likely to be executed from within a single thread? I assume the former but want to be sure. Assuming that thread-safety is needed, it seems like I should be using something like

Re: Efficiently caching API results in a NiFi controller service

Thanks Otto - Unfortunately, the service being called doesn’t currently support full HTTP cache semantics at this time. I could add full support, and it is probably the right thing to do in the long run. But for now I was hoping for a solution that didn’t require significant enhancement to the

Re: Efficiently caching API results in a NiFi controller service

Tim, The reason the DMC works the way it does is because the cached data needs to be shared across a cluster. For example, a processor like DetectDuplicate needs to detect duplicates across all NiFi nodes and not just the local node, or the same thing with Wait/Notify. In your case I don't think

ApacheCon North America 2018 schedule is now live.

Dear Apache Enthusiast, We are pleased to announce our schedule for ApacheCon North America 2018. ApacheCon will be held September 23-27 at the Montreal Marriott Chateau Champlain in Montreal, Canada. Registration is open! The early bird rate of $575 lasts until July 21, at which time it

Re: Efficiently caching API results in a NiFi controller service

https://hc.apache.org/httpcomponents-client-ga/tutorial/html/caching.html ? On May 1, 2018 at 00:01:58, Tim Dean (tim.d...@gmail.com) wrote: Hello, I have a custom NiFi controller service that retrieves data from an external web service via HTTP requests. The results from these HTTP requests

Re: Unit testing a custom NiFi controller service?

Tim, That doesn’t mean there isn’t room for improvement. If you have an ideas of how you would like it to work, or how it would work well, please create a Jira issue. On April 30, 2018 at 23:40:36, Tim Dean (tim.d...@gmail.com) wrote: Thank you for the information, Otto. If this approach is