Supporting to Tijo’s thought, have one scenario.
we are trying to use Nifi for a data pipeline solution. The scenario is to
coordinate between various services and provide a solution for big data analysis
In our scenario many of the activities are kind of "run on
primary" mode processors. These are being implemented on top of various
components like Yarn, Hbase, Spark, DB etc.
One issue we are seeing is all these processors to be run on
primary node [like spark execution, yarn/mr job execution etc.. ] and it is
We are thinking of having multiple primary nodes and assign the
activities using some distribution algorithm.
The idea is to handle the coordination and failover mechanism
Any thoughts on this ?
From: Jeff [mailto:jtsw...@gmail.com]
Sent: Monday, September 19, 2016 11:17 PM
To: Tijo Thomas; us...@nifi.apache.org
Subject: Re: enforce run only in promary node $ multiple primary node
To give you some information on your second question, you can design your flow
to redistribute the flowfiles coming out of your processors to other nodes in
the cluster for processing. There are several examples on how this on various
blogs/email lists/etc, and I just grabbed one for reference, written by Apache
NiFi's own Bryan Bende:
Please review that thread and let us know if you have further questions!
On Mon, Sep 19, 2016 at 1:19 PM Tijo Thomas
1. While writing a processor is it possible to enforce to run only in primary
node. I saw a Jira for this but appears to unresolved.
[NIFI-543] Provide extensions a way to indicate that they can run only on
primary node, if clustered - ASF
[NIFI-543] Provide extensions a way to indicate that they can run only on p...
2. Currently my Primary node is heavily loaded as i have many processor which
will run only in Primary node. Is it possible to define multiple primary nodes
. or is it possible to configure processors not to run in primary node.