Most often, for HDFS, you'd need multiple sinks to drain the same
channel than one with a load balancer. Load balancing is mostly meant to
send data from one set of Flume agents to another set (tier 1 to 2), so
that even if one agent goes down, the data still goes to the others.
Adding more sinks or sink groups adds multithreading - so your drain
rates improve. Each sink group gets a thread - the thread chooses one of
the sinks to process the data from the channel. So for multithreading +
load balancing, you'd need several (possibly, identically configured,
but with different sinks) sink groups. Think of sinks outside of sink
groups as a sink group with just 1 sink.
[email protected] wrote:
Hello,
No they are not as far as I know. That makes it very important to know
when designing Load Balanced Flume Architecture: you must count on
multiple agents, rather than multiple sinks
Best regards
________________________________________________
Jean-François Guilmard
Accenture Digital
Email: [email protected]
Phone: +33 6 32 27 70 22
Accenture Analytics: Connect with us to learn more.
-----Original Message-----
From: Guillermo Ortiz [mailto:[email protected]]
Sent: mardi 10 mars 2015 12:15
To: [email protected]
Subject: Load Balancer
I was checking the load balancing with a group of HDFS-Sink, are they
multithreading?? I mean, each sink of the group is an independent
thread as when I have many sinks?
When should I use loadbalancer instance of independend sinks?
________________________________
This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you
have received it in error, please notify the sender immediately and
delete the original. Any other use of the e-mail by you is prohibited.
Where allowed by local law, electronic communications with Accenture
and its affiliates, including e-mail and instant messaging (including
content), may be scanned by our systems for the purposes of
information security and assessment of internal compliance with
Accenture policy.
______________________________________________________________________________________
www.accenture.com
