Each if the sinks will run in a different proccess? Any way I can share the same sink configuration for both sinks? Eran
בתאריך יום ה׳, 8 באוק׳ 2015, 19:50 מאת Roshan Naik <[email protected]>: > 1 – Bump up the –Xmx in flume-env.sh as the default is quite small > 2 – increase the capacity on the channel. Looks like your source is > running much faster than the sink can keep up. You can try adding more > sinks to improve drain rate. > > From: IT CTO <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Thursday, October 8, 2015 9:46 AM > To: "[email protected]" <[email protected]> > Subject: Flume source channel sink tunning > > Hi, > > I am using SpoolDir with memory channel to write to hdfs sink. > When I use a single spoolDir I get single threaded performance so based on > some mails I read I splinted the source to 5 spoolDir all writing to the > same memory channel which writes to the hdfs. > > Now I am getting different errors > 1) GC error => not engough memory for the channel => increase Xmx for the > agent > 2) Channel is full => the sink is not kipping up with the channel > > so I find myself playing with the different parameters. > *any best practice here or path to follow to get it tuned?* > I feel that even if it works it will easily break given other events > > Eran > -- > Eran | "You don't need eyes to see, you need vision" (Faithless) > -- Eran | "You don't need eyes to see, you need vision" (Faithless)
