Re: Reg. Nifi Clustering Load Balancing

[Kevin Doran]

Hi Nishant,

A HTTP load-balancer as Bryan suggests is a good option for server-side load 
balancing so that your client only needs to keep track of a single URL. In 
addition to the options Bryan mentioned, which are great, most cloud providers 
offer managed load balancing services, which many be convenient if you happen 
to be using a commercial cloud for your NiFi cluster.

Another option is client-side load balancing, in which clients of the NiFi 
cluster discover all available nodes in the cluster and cycle through them. 
There are many ways of implementing client-side load balancing, most of which 
involve periodically refreshing a list of server nodes/endpoints. If you want 
to explore this path, the NiFi API [1] makes it easy to discover all nodes 
known to the cluster and their status using this endpoint:

GET /controller/cluster

This can be called on any node in the cluster. It will return a JSON object 
that includes an array of nodes, each of which has an address and port you 
could use to build a URL as well as some other status information that might be 
useful for implementing dynamic load-balancing logic in the client. This could 
be consumed directly by the client or by an agent of a service discovery 
framework that would maintain a directory/registry of endpoints. Examples of 
frameworks that can be used as a service discovery registry include Apache 
Zookeeper, Consul, Netlfix Eureka, and etcd.

The easiest way to get started for most is definitely the server-side load 
balancing approach using an HTTP load balancer. There are pros and cons to both 
server-side and client-side load balancing that one has to evaluate based on a 
particular use case. So I agree with Bryan's advice, and am only pointing out 
another option that may be preferable to some on this list that have a similar 
need.

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/ 

Cheers,
Kevin

On 8/29/17, 12:15, "Bryan Bende"  wrote:

Hello,

You can run a standard HTTP load-balancer in front of ListenHTTP and have
your producers use the URL of the load-balancer.

Nginx or apache httpd can be used.

Thanks,

Bryan


On Tue, Aug 29, 2017 at 11:40 AM, mayank rathi 
wrote:

> Does this help?
>
> [image: Inline image 1]
>
> On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta <
> nishantgupta1...@gmail.com> wrote:
>
>> Hello Sir/Madam,
>>
>> I need to know how we can implement load balancing and single point 
access
>> to all data producer
>>
>> I have suppose 100 machines that produce data.(flowfiles)
>> That data is sent over HTTP from 100 machines to nifi and that we can
>> access using listenHTTP processor in Nifi.(Currently I am able to do that
>> for single node Nifi)
>> Is there any way to provide 100 machines with a single URL (for all 4-5
>> nodes in Nifi Cluster)
>>
>> Can you please let me know how to achieve it.??
>>
>> Thanks and Regards,
>> Nishant Gupta
>>
>
>
>
> --
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.
>

Re: Reg. Nifi Clustering Load Balancing

[Bryan Bende]

Hello,

You can run a standard HTTP load-balancer in front of ListenHTTP and have
your producers use the URL of the load-balancer.

Nginx or apache httpd can be used.

Thanks,

Bryan


On Tue, Aug 29, 2017 at 11:40 AM, mayank rathi 
wrote:

> Does this help?
>
> [image: Inline image 1]
>
> On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta <
> nishantgupta1...@gmail.com> wrote:
>
>> Hello Sir/Madam,
>>
>> I need to know how we can implement load balancing and single point access
>> to all data producer
>>
>> I have suppose 100 machines that produce data.(flowfiles)
>> That data is sent over HTTP from 100 machines to nifi and that we can
>> access using listenHTTP processor in Nifi.(Currently I am able to do that
>> for single node Nifi)
>> Is there any way to provide 100 machines with a single URL (for all 4-5
>> nodes in Nifi Cluster)
>>
>> Can you please let me know how to achieve it.??
>>
>> Thanks and Regards,
>> Nishant Gupta
>>
>
>
>
> --
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.
>

Re: Reg. Nifi Clustering Load Balancing

[mayank rathi]

Does this help?

[image: Inline image 1]

On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta 
wrote:

> Hello Sir/Madam,
>
> I need to know how we can implement load balancing and single point access
> to all data producer
>
> I have suppose 100 machines that produce data.(flowfiles)
> That data is sent over HTTP from 100 machines to nifi and that we can
> access using listenHTTP processor in Nifi.(Currently I am able to do that
> for single node Nifi)
> Is there any way to provide 100 machines with a single URL (for all 4-5
> nodes in Nifi Cluster)
>
> Can you please let me know how to achieve it.??
>
> Thanks and Regards,
> Nishant Gupta
>



-- 
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message.