My hunch would be that you're anchoring your tuples and without acker tasks these tuple trees are never destroyed. So all the tuples from each batch are kept in memory until you run out of memory, i.e., a very, very fast memory leak.
If you disable ackers, you should probably not anchor your tuples either. -Cody On Mon, Jun 30, 2014 at 9:56 PM, [email protected] < [email protected]> wrote: > Hi all, > > I use storm 0.9.2 now, and do real-time processing by basic storm. It's > normal to use in reliable way. But the speed is extremely slow about 6000 > tuples/s. So I try to disable acker feature by conf.setNumAckers(0). > However, this leads to following error: > > java.lang.OutOfMemoryError: GC overhead limit exceeded at > java.lang.reflect.Method.copy(Method.java:151) at > java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136) at > sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300) at > java.lang.Class.copyMethods(Class.java:2891) at > java.lang.Class.getMethods(Class.java:1467) at > clojure.lang.Reflector.getMethods(Reflector.java:357) at > clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:27) at > backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128) > at > backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__5483.invoke(executor.clj:256) > at > backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > at > backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94) > at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at > clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745) > > > Can anyone tell me why this happens? > Thank you very much. > > Best regards, > James Fu > > -- Cody A. Ray, LEED AP [email protected] 215.501.7891
