Hi all

                Supporting to Tijo’s thought, have one scenario.

we are trying to use Nifi for a data pipeline solution. The scenario is to 
coordinate between various services and provide a solution for big data analysis
                In our scenario many of the activities are kind of "run on 
primary" mode processors. These are being implemented on top of various 
components like Yarn, Hbase, Spark, DB etc.

                One issue we are seeing is all these processors to be run on 
primary node  [like spark execution, yarn/mr job execution etc.. ] and it is 
only one.
                We are thinking of having multiple primary nodes and assign the 
activities using some distribution algorithm.
                The idea is to handle the coordination and failover mechanism 
using zookeeper.

                Any thoughts on this ?

Regards
Nijel

From: Jeff [mailto:jtsw...@gmail.com]
Sent: Monday, September 19, 2016 11:17 PM
To: Tijo Thomas; us...@nifi.apache.org
Subject: Re: enforce run only in promary node $ multiple primary node

Tijo,

To give you some information on your second question, you can design your flow 
to redistribute the flowfiles coming out of your processors to other nodes in 
the cluster for processing.  There are several examples on how this on various 
blogs/email lists/etc, and I just grabbed one for reference, written by Apache 
NiFi's own Bryan Bende: 
http://apache-nifi.1125220.n5.nabble.com/How-to-configure-site-to-site-communication-between-nodes-in-one-cluster-td8528.html

Please review that thread and let us know if you have further questions!

On Mon, Sep 19, 2016 at 1:19 PM Tijo Thomas 
<tijopara...@yahoo.in<mailto:tijopara...@yahoo.in>> wrote:

Hi ,

1. While writing a processor is it possible to enforce to run only in primary 
node. I saw a Jira for this but appears to unresolved.

[NIFI-543] Provide extensions a way to indicate that they can run only on 
primary node, if clustered - ASF 
JIRA<https://issues.apache.org/jira/browse/NIFI-543>





[NIFI-543] Provide extensions a way to indicate that they can run only on p...




2. Currently my Primary node is heavily loaded  as i have many  processor which 
will run only in Primary node.  Is it possible to define multiple primary nodes 
. or is it possible to configure processors not to run in primary node.

Tijo

Reply via email to