Re: Controlling number of threads in distributed mode
Ok. So looks like the information needed to make a request comes from a Kafka topic. Then requests are made against some HTTP endpoint for which it wants a max number of requests at once. Given how small that number is I'd not worry at all about nifi being overloaded but perhaps there is more depth to this. I'd just turn on load balancing on the connection between ConsumeKafka and InvokeHttpRequest and let the messages be consumed and requests sent naturally. Thanks On Wed, Feb 13, 2019 at 9:52 PM Milan Das wrote: > Hi Joe, > Use case is I have to query HTTP request. The HTTP request from NIFI > can only run 3 queries at a time. > > The queries to be executed are coming from Kafka topic and we have 6 node > cluster. > Ideally would like to run the 3 http queries from 3 different nodes > otherwise one of the node (primary node) will be overloaded with data. > Thanks, > Milan Das > > > > > On Feb 13, 2019, at 4:38 PM, Joe Witt wrote: > > > > Milan > > > > The only mechanism to limit whether a given component executes on a given > > nifi node or not is whether it runs on all or whether it runs on primary > > node. > > > > Can you describe the use case that would lead to wanting to run a > component > > on a specified number of nodes as opposed to all nodes? It would defy > the > > clustering logic/data distribution mechanisms of site to site and load > > balanced connections for example. > > > > Thanks > > > > On Wed, Feb 13, 2019 at 4:34 PM Milan Das wrote: > > > >> Hello, > >> Is there a way to control number of thread less than cluster member ? > >> I have a 5 node NiFI cluster and I want to run 3 instances of the > >> processor in distributed mode. > >> > >> Thanks, > >> Milan Das > >> > >> > >
Re: Controlling number of threads in distributed mode
Hi Joe, Use case is I have to query HTTP request. The HTTP request from NIFI can only run 3 queries at a time. The queries to be executed are coming from Kafka topic and we have 6 node cluster. Ideally would like to run the 3 http queries from 3 different nodes otherwise one of the node (primary node) will be overloaded with data. Thanks, Milan Das > On Feb 13, 2019, at 4:38 PM, Joe Witt wrote: > > Milan > > The only mechanism to limit whether a given component executes on a given > nifi node or not is whether it runs on all or whether it runs on primary > node. > > Can you describe the use case that would lead to wanting to run a component > on a specified number of nodes as opposed to all nodes? It would defy the > clustering logic/data distribution mechanisms of site to site and load > balanced connections for example. > > Thanks > > On Wed, Feb 13, 2019 at 4:34 PM Milan Das wrote: > >> Hello, >> Is there a way to control number of thread less than cluster member ? >> I have a 5 node NiFI cluster and I want to run 3 instances of the >> processor in distributed mode. >> >> Thanks, >> Milan Das >> >>
Re: Controlling number of threads in distributed mode
Milan The only mechanism to limit whether a given component executes on a given nifi node or not is whether it runs on all or whether it runs on primary node. Can you describe the use case that would lead to wanting to run a component on a specified number of nodes as opposed to all nodes? It would defy the clustering logic/data distribution mechanisms of site to site and load balanced connections for example. Thanks On Wed, Feb 13, 2019 at 4:34 PM Milan Das wrote: > Hello, > Is there a way to control number of thread less than cluster member ? > I have a 5 node NiFI cluster and I want to run 3 instances of the > processor in distributed mode. > > Thanks, > Milan Das > >
Controlling number of threads in distributed mode
Hello, Is there a way to control number of thread less than cluster member ? I have a 5 node NiFI cluster and I want to run 3 instances of the processor in distributed mode. Thanks, Milan Das
