Try putting those static initializations into the prepare. On Mar 8, 2014 12:20 PM, "Zheng Xue" <[email protected]> wrote:
> Thanks for you reply. > > I excluded the log4j dependency in the pom file. The topology can run in > Storm-0.9.0-wip21 now. > > But another error appears in the last bolt, which is responsible for > writing statistics into HBase. Here is the exception information: > > > java.lang.RuntimeException: java.lang.NullPointerException > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) > at > backtype.storm.daemon.executor$fn__3483$fn__3495$fn__3542.invoke(executor.clj:715) > at backtype.storm.util$async_loop$fn__441.invoke(util.clj:396) > at clojure.lang.AFn.run(AFn.java:24) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.NullPointerException > at > com.mycompany.app.MobileNetLogThresholdBolt.execute(MobileNetLogThresholdBolt.java:81) > at > backtype.storm.daemon.executor$fn__3483$tuple_action_fn__3485.invoke(executor.clj:610) > at > backtype.storm.daemon.executor$mk_task_receiver$fn__3406.invoke(executor.clj:381) > at > backtype.storm.disruptor$clojure_handler$reify__2948.onEvent(disruptor.clj:43) > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:79) > > Line 81 in MobileNetLogThresholdBolt.java is "table.put(put)" (see the > java code of this bolt below). Referred to > https://github.com/nathanmarz/storm/wiki/Troubleshooting, multiple > threads using the OutputCollector leads to this exception. But how can I > fix it? Thanks. > > public class MobileNetLogThresholdBolt implements IRichBolt { > private OutputCollector outputCollector; > public static Configuration configuration; > public static String tablename = "t_mobilenet_threshold"; > public static HTable table; > static { > configuration = HBaseConfiguration.create(); > configuration.set("hbase.zookeeper.property.clientPort","2181"); > configuration.set("hbase.zookeeper.quorum", "xx.xx.xx.xx"); > configuration.set("hbase.master", "xx.xx.xx.xx:60000"); > } > > > private Log log = LogFactory.getLog(MobileNetLogThresholdBolt.class); > > @Override > public void prepare(Map map, TopologyContext topologyContext, > OutputCollector outputCollector) { > this.outputCollector = outputCollector; > try { > table = new HTable(configuration, tablename); > } catch (IOException e) { > // TODO Auto-generated catch block > e.printStackTrace(); > } > > } > > > @Override > public void execute(Tuple tuple) { > > log.info("deal data " + tuple.getString(0) + "=" + > tuple.getInteger(1)); > if (tuple.getInteger(1) > 2) { > > Put put = new Put(Bytes.toBytes(tuple.getString(0))); > put.add(Bytes.toBytes("STAT_INFO"), Bytes.toBytes("COUNT"), > Bytes.toBytes(String.valueOf(tuple.getInteger(1)))); > try { > table.put(put); > } catch (IOException e) { > e.printStackTrace(); > } > } > this.outputCollector.emit(tuple, tuple.getValues()); > this.outputCollector.ack(tuple); > } > > @Override > public void cleanup() { > > } > > @Override > public void declareOutputFields(OutputFieldsDeclarer > outputFieldsDeclarer) { > } > > @Override > public Map<String, Object> getComponentConfiguration() { > return null; > } > } > > > 2014-03-08 0:53 GMT+08:00 bijoy deb <[email protected]>: > >> Hi Zheng, >> >> Did you look at the logs for what exactly the error message is? >> In case you see any error that says "multiple default.yaml in path >> ...",try the below: >> --In the with-dependencies jar,can you check if you have a default.yaml >> or storm.yaml file.If yes,please delete it and try submitting the resulting >> jar. >> >> Thanks >> Bijoy >> >> >> On Fri, Mar 7, 2014 at 10:51 AM, Zheng Xue <[email protected]> wrote: >> >>> Hi, all: >>> >>> I was trying to build a Storm topology with Hadoop and Hbase >>> dependencies. And I want to run this topology with Storm-on-Yarn. The >>> version of Storm in it is 0.9.0-wip21. I created the Jar file with Maven, >>> and the pom.xml file is attached. >>> >>> I submitted the topology (with-dependencies), and there was no >>> exceptions. But it did't run at all. I checked the supervisor logs (see >>> bottom), which shows that the workers failed to start. >>> >>> To trace the reason of this issue, I add a test topology >>> (WordCountTopology) in the jar file. The problems remains when trying to >>> submit the WordCountTopology with dependencies, but it works well when >>> trying to submit the topology without dependencies. >>> >>> To make it clear that where is this issue from, I ran the topology >>> without Storm-on-Yarn. I tried Storm-0.8.1 and Storm-0.9.0-wip21. The >>> results show that: >>> 1) Storm-0.8.1 is okay for the jar topology, no matter with or >>> without dependencies. >>> 2) Storm-0.9.0-wip21 is okay for the jar topology without >>> dependencies, but the same issue (workers failed to start) appears for the >>> "with-dependencies" jar file. >>> >>> >>> >>> I am a newer for Storm, and I don't know the differences between >>> these two Storm version. What will result in the worker failure when trying >>> to submit the "with-dependencies" topology: Is the "with-dependencies" jar >>> file too big? or something conflicting? >>> >>> Any idea? >>> Will Storm-0.8.1 work with Storm-on-Yarn? >>> Thanks~ >>> >>> >>> -- >>> ------------------ >>> Best Regards! >>> >>> Zheng Xue >>> *http://netlab.sysu.edu.cn/~zhxue <http://netlab.sysu.edu.cn/~zhxue>* >>> Computer Science and Technology college,Sun Yat-sen University >>> E-mail: [email protected], [email protected] >>> >> >> > > > -- > ------------------ > Best Regards! > > Zheng Xue > *http://netlab.sysu.edu.cn/~zhxue <http://netlab.sysu.edu.cn/~zhxue>* > Computer Science and Technology college,Sun Yat-sen University > E-mail: [email protected], [email protected] >
