Hi Dilip, you need to have proper set of configs to set this. you can not have too much data from kafka at one shot where u cant keep in jvm. So you should figure out the magic expression that should work as per your cluster/hardware available.
if you increase worker jvm then you cannot have too many workers for that supervisor. which means no parallel processing. So, please keep low and process it faster with multi worker instances. relatively, low throughout and low latency. On Thu, Mar 24, 2016 at 10:40 AM, Tech Id <[email protected]> wrote: > You need to increase worker's Heap Size by adding the following in each > worker's storm.yaml file: > > worker.childopts: "-Xmx16g" > > Then restart each worker's supervisor. > > > > On Wed, Mar 23, 2016 at 4:07 AM, Sai Dilip Reddy Kiralam < > [email protected]> wrote: > >> Hi, >> >> >> >> When I Increase the kafkaconfig.fetchSizeBytes value to 150*1024*1024,I >> get desired output from kafka-spout but after sometime kafka spout is >> throwing the exception and my spout is failing.What should I do to handle >> this.Do I need to add any lines in storm.yaml to handle this exception. >> >> java.lang.OutOfMemoryError: Java heap space >> at java.util.Arrays.copyOf(Arrays.java:3332) >> at >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) >> at >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) >> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622) >> at java.lang.StringBuilder.append(StringBuilder.java:202) >> at java.util.AbstractCollection.toString(AbstractCollection.java:464) >> at clojure.core$str.invoke(core.clj:513) >> at clojure.core$str$fn__3896.invoke(core.clj:517) >> at clojure.core$str.doInvoke(core.clj:519) >> at clojure.lang.RestFn.invoke(RestFn.java:516) >> at backtype.storm.daemon.task$mk_tasks_fn$fn__6307.invoke(task.clj:152) >> at >> backtype.storm.daemon.executor$fn__6579$fn__6594$send_spout_msg__6612.invoke(executor.clj:482) >> at >> backtype.storm.daemon.executor$fn__6579$fn$reify__6621.emit(executor.clj:526) >> at >> backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:49) >> at >> backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:63) >> at storm.kafka.PartitionManager.next(PartitionManager.java:141) >> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:141) >> at >> backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke(executor.clj:565) >> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463) >> at clojure.lang.AFn.run(AFn.java:24) >> at java.lang.Thread.run(Thread.java:745) >> >> Need suggestions >> >> >> >> >> *Best regards,* >> *K.Sai Dilip Reddy.* >> > > -- Thanks & Regards Rajasekhar
