Venkatesh, Thanks for the reply. My other question was the latter one as you mentioned in your reply. Our each sinks are tied to the file channel in our configuration i.e. each sink will have its own file channel. The number sinks will be more the number of threads (/cores) available & I think the maximum number of channel/sink we will hit will be 99. I hope flume-ng handles it nicely.
How does flume-ng handles incoming events when it has reached the full memory allocated to it? Does it exit or does it rejects & return error code to the downstream agents? Thanks, Deepesh On Thu, Sep 5, 2013 at 11:45 PM, Venkatesh S R <[email protected]> wrote: > Hi Deepesh: I think you can ignore the exception stack, its benign. Flume > tries to re-register the monitoring MBean whenever the configurations are > dynamically refreshed and it re-initializes the components. And because the > MBean was already registered, you see this exception trace. I remember > there was a Jira ticket to suppress this and Mike/Hari fixed the same in > trunk. > > On the other question I assume you are either asking if there a minimum of > the number sinks required or if there is limit to the total number of sinks > in an instance, for both questions the answer is NO. > > I would think of the heap size proportional to the Channel's (Memory) size > you are setting up, to make sure there is enough space to hold all the > events when required. The number of sinks that are setup, would relate to > the number of threads (/cores) available for you to use. > > Other folks can chime in and add as required. > > Thanks > Venkatesh > > > > On Wed, Sep 4, 2013 at 11:54 PM, Deepesh Malviya <[email protected]>wrote: > >> Hi, >> >> We have configured flume-ng in our environment & we dynamically update >> the configuration. In our case, we will be having 20+ sinks & will increase >> as we update the conf. The problem we are seeing in it is the following >> exception is coming repeatedly for most of the configured sinks >> >> >> 05 Sep 2013 05:00:38,770 ERROR [lifecycleSupervisor-1-3] >> (org.apache.flume.instrumentation.MonitoredCounterGroup.register:113) - >> Failed to register monitored counter group for type: SINK, name: serviceSink >> javax.management.InstanceAlreadyExistsException: >> org.apache.flume.sink:type=serviceSink >> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330) >> at >> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517) >> at >> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:108) >> at >> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:88) >> at org.apache.flume.sink.hdfs.HDFSEventSink.start(HDFSEventSink.java:484) >> at >> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) >> at org.apache.flume.SinkRunner.start(SinkRunner.java:79) >> at >> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:679) >> >> >> What is the cause of this error? >> Also, is there any limited number of sinks that should be configured with >> one flume-ng instance. The reason for asking this particular question is >> because of the heap size we configure in flume-ng. >> >> >> -- >> _Deepesh >> > > -- _Deepesh
