Thanks Derek for the tips. quite useful. best, /Shahab
On Mon, Mar 10, 2014 at 8:49 PM, Derek Dagit <[email protected]> wrote: > Yes, set 'nimbus.childopts: "-Xmx?????"' in your storm.yaml, and restart > nimbus. If unset, I believe the default is "-Xmx1024m", for a max of 1024 > MB heap. > > You can set it to "-Xmx2048m", for example, to have a max heap size of > 2048 MB. > > Set this on the node that runs nimbus, not in your topology conf. > > -- > Derek > > > On 3/10/14, 14:19, shahab wrote: > >> Hi, >> >> I am facing "OutOfMemoryError: Java heap space" exception in Nimbus while >> running in cluster mode. I just wonder what are the possible JVM or Storm >> options that I can set to overcome this problem? >> >> I am running a storm topology in Cluster mode where all servers >> (zookeeper, nimbus, supervisor and worker) are in one machine. The toplogy >> that I use is as follows: >> >> >> conf.setMaxSpoutPending(2000); // maximum number of pending messages at >> spout >> conf.setNumWorkers(4); >> conf.put(Config.STORM_ZOOKEEPER_CONNECTION_TIMEOUT, "120000"); >> conf.setMaxTaskParallelism(2); >> >> >> but I get the following Exception in Nimbus log file: >> java.lang.OutOfMemoryError: Java heap space >> at org.apache.commons.io.output.ByteArrayOutputStream. >> toByteArray(ByteArrayOutputStream.java:271) >> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219) >> at org.apache.commons.io.FileUtils.readFileToByteArray( >> FileUtils.java:1136) >> at backtype.storm.daemon.nimbus$read_storm_topology.invoke( >> nimbus.clj:305) >> at backtype.storm.daemon.nimbus$compute_executors.invoke( >> nimbus.clj:407) >> at backtype.storm.daemon.nimbus$compute_executor__GT_ >> component.invoke(nimbus.clj:420) >> at backtype.storm.daemon.nimbus$read_topology_details.invoke( >> nimbus.clj:315) >> at backtype.storm.daemon.nimbus$mk_assignments$iter__3416__ >> 3420$fn__3421.invoke(nimbus.clj:636) >> at clojure.lang.LazySeq.sval(LazySeq.java:42) >> at clojure.lang.LazySeq.seq(LazySeq.java:60) >> at clojure.lang.RT.seq(RT.java:473) >> at clojure.core$seq.invoke(core.clj:133) >> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) >> at clojure.core.protocols$fn__5875.invoke(protocols.clj:54) >> at clojure.core.protocols$fn__5828$G__5823__5841.invoke( >> protocols.clj:13) >> at clojure.core$reduce.invoke(core.clj:6030) >> at clojure.core$into.invoke(core.clj:6077) >> at backtype.storm.daemon.nimbus$mk_assignments.doInvoke( >> nimbus.clj:635) >> at clojure.lang.RestFn.invoke(RestFn.java:410) >> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__ >> __3593$fn__3598$fn__3599.invoke(nimbus.clj:872) >> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__ >> __3593$fn__3598.invoke(nimbus.clj:871) >> at backtype.storm.timer$schedule_recurring$this__1776.invoke( >> timer.clj:69) >> at backtype.storm.timer$mk_timer$fn__1759$fn__1760.invoke( >> timer.clj:33) >> at backtype.storm.timer$mk_timer$fn__1759.invoke(timer.clj:26) >> at clojure.lang.AFn.run(AFn.java:24) >> at java.lang.Thread.run(Thread.java:744) >> 2014-03-10 20:10:02 NIOServerCnxn [ERROR] Thread >> Thread[pool-4-thread-40,5,main] died >> java.lang.OutOfMemoryError: Java heap space >> at org.apache.commons.io.output.ByteArrayOutputStream. >> toByteArray(ByteArrayOutputStream.java:271) >> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219) >> at org.apache.commons.io.FileUtils.readFileToByteArray( >> FileUtils.java:1136) >> at backtype.storm.daemon.nimbus$read_storm_topology.invoke( >> nimbus.clj:305) >> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__ >> $reify__3605.getTopologyInfo(nimbus.clj:1066) >> at backtype.storm.generated.Nimbus$Processor$ >> getTopologyInfo.getResult(Nimbus.java:1481) >> at backtype.storm.generated.Nimbus$Processor$ >> getTopologyInfo.getResult(Nimbus.java:1469) >> at org.apache.thrift7.ProcessFunction.process( >> ProcessFunction.java:32) >> at org.apache.thrift7.TBaseProcessor.process( >> TBaseProcessor.java:34) >> at org.apache.thrift7.server.TNonblockingServer$ >> FrameBuffer.invoke(TNonblockingServer.java:632) >> at org.apache.thrift7.server.THsHaServer$Invocation.run( >> THsHaServer.java:201) >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> 2014-03-10 20:10:02 util [INFO] Halting process: ("Error when processing >> an event") >> >> >> best , >> /Shahab >> >>
