Might be good idea to make a note of this correlation between sink groups and threads in the user guide.
From: Hari Shreedharan <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, March 10, 2015 9:02 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Load Balancer Most often, for HDFS, you'd need multiple sinks to drain the same channel than one with a load balancer. Load balancing is mostly meant to send data from one set of Flume agents to another set (tier 1 to 2), so that even if one agent goes down, the data still goes to the others. Adding more sinks or sink groups adds multithreading - so your drain rates improve. Each sink group gets a thread - the thread chooses one of the sinks to process the data from the channel. So for multithreading + load balancing, you'd need several (possibly, identically configured, but with different sinks) sink groups. Think of sinks outside of sink groups as a sink group with just 1 sink. [email protected]<mailto:[email protected]> wrote: Hello, No they are not as far as I know. That makes it very important to know when designing Load Balanced Flume Architecture: you must count on multiple agents, rather than multiple sinks Best regards ________________________________________________ Jean-François Guilmard Accenture Digital Email: [email protected]<mailto:[email protected]> Phone: +33 6 32 27 70 22 Accenture Analytics: Connect with us to learn more. -----Original Message----- From: Guillermo Ortiz [mailto:[email protected]] Sent: mardi 10 mars 2015 12:15 To: [email protected]<mailto:[email protected]> Subject: Load Balancer I was checking the load balancing with a group of HDFS-Sink, are they multithreading?? I mean, each sink of the group is an independent thread as when I have many sinks? When should I use loadbalancer instance of independend sinks? ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com
