You can connect multiple sinks to same memory channel. Each sink (and source) will get a thread within the same flume agent process. Each sink should differ in the destination file it writes to. Configure each sink separately in the config file and set its ...channel = Property to the same channel.
-roshan From: IT CTO <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, October 8, 2015 10:45 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Flume source channel sink tunning Each if the sinks will run in a different proccess? Any way I can share the same sink configuration for both sinks? Eran בתאריך יום ה׳, 8 באוק׳ 2015, 19:50 מאת Roshan Naik <[email protected]<mailto:[email protected]>>: 1 – Bump up the –Xmx in flume-env.sh as the default is quite small 2 – increase the capacity on the channel. Looks like your source is running much faster than the sink can keep up. You can try adding more sinks to improve drain rate. From: IT CTO <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, October 8, 2015 9:46 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Flume source channel sink tunning Hi, I am using SpoolDir with memory channel to write to hdfs sink. When I use a single spoolDir I get single threaded performance so based on some mails I read I splinted the source to 5 spoolDir all writing to the same memory channel which writes to the hdfs. Now I am getting different errors 1) GC error => not engough memory for the channel => increase Xmx for the agent 2) Channel is full => the sink is not kipping up with the channel so I find myself playing with the different parameters. any best practice here or path to follow to get it tuned? I feel that even if it works it will easily break given other events Eran -- Eran | "You don't need eyes to see, you need vision" (Faithless) -- Eran | "You don't need eyes to see, you need vision" (Faithless)
