Thanks for you reply.
I excluded the log4j dependency in the pom file. The topology can run in
Storm-0.9.0-wip21 now.
But another error appears in the last bolt, which is responsible for
writing statistics into HBase. Here is the exception information:
java.lang.RuntimeException: java.lang.NullPointerException
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at
backtype.storm.daemon.executor$fn__3483$fn__3495$fn__3542.invoke(executor.clj:715)
at backtype.storm.util$async_loop$fn__441.invoke(util.clj:396)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at
com.mycompany.app.MobileNetLogThresholdBolt.execute(MobileNetLogThresholdBolt.java:81)
at
backtype.storm.daemon.executor$fn__3483$tuple_action_fn__3485.invoke(executor.clj:610)
at
backtype.storm.daemon.executor$mk_task_receiver$fn__3406.invoke(executor.clj:381)
at
backtype.storm.disruptor$clojure_handler$reify__2948.onEvent(disruptor.clj:43)
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:79)
Line 81 in MobileNetLogThresholdBolt.java is "table.put(put)" (see the java
code of this bolt below). Referred to
https://github.com/nathanmarz/storm/wiki/Troubleshooting, multiple
threads using
the OutputCollector leads to this exception. But how can I fix it? Thanks.
public class MobileNetLogThresholdBolt implements IRichBolt {
private OutputCollector outputCollector;
public static Configuration configuration;
public static String tablename = "t_mobilenet_threshold";
public static HTable table;
static {
configuration = HBaseConfiguration.create();
configuration.set("hbase.zookeeper.property.clientPort","2181");
configuration.set("hbase.zookeeper.quorum", "xx.xx.xx.xx");
configuration.set("hbase.master", "xx.xx.xx.xx:60000");
}
private Log log = LogFactory.getLog(MobileNetLogThresholdBolt.class);
@Override
public void prepare(Map map, TopologyContext topologyContext,
OutputCollector outputCollector) {
this.outputCollector = outputCollector;
try {
table = new HTable(configuration, tablename);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
@Override
public void execute(Tuple tuple) {
log.info("deal data " + tuple.getString(0) + "=" +
tuple.getInteger(1));
if (tuple.getInteger(1) > 2) {
Put put = new Put(Bytes.toBytes(tuple.getString(0)));
put.add(Bytes.toBytes("STAT_INFO"), Bytes.toBytes("COUNT"),
Bytes.toBytes(String.valueOf(tuple.getInteger(1))));
try {
table.put(put);
} catch (IOException e) {
e.printStackTrace();
}
}
this.outputCollector.emit(tuple, tuple.getValues());
this.outputCollector.ack(tuple);
}
@Override
public void cleanup() {
}
@Override
public void declareOutputFields(OutputFieldsDeclarer
outputFieldsDeclarer) {
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
2014-03-08 0:53 GMT+08:00 bijoy deb <[email protected]>:
> Hi Zheng,
>
> Did you look at the logs for what exactly the error message is?
> In case you see any error that says "multiple default.yaml in path
> ...",try the below:
> --In the with-dependencies jar,can you check if you have a default.yaml
> or storm.yaml file.If yes,please delete it and try submitting the resulting
> jar.
>
> Thanks
> Bijoy
>
>
> On Fri, Mar 7, 2014 at 10:51 AM, Zheng Xue <[email protected]> wrote:
>
>> Hi, all:
>>
>> I was trying to build a Storm topology with Hadoop and Hbase
>> dependencies. And I want to run this topology with Storm-on-Yarn. The
>> version of Storm in it is 0.9.0-wip21. I created the Jar file with Maven,
>> and the pom.xml file is attached.
>>
>> I submitted the topology (with-dependencies), and there was no
>> exceptions. But it did't run at all. I checked the supervisor logs (see
>> bottom), which shows that the workers failed to start.
>>
>> To trace the reason of this issue, I add a test topology
>> (WordCountTopology) in the jar file. The problems remains when trying to
>> submit the WordCountTopology with dependencies, but it works well when
>> trying to submit the topology without dependencies.
>>
>> To make it clear that where is this issue from, I ran the topology
>> without Storm-on-Yarn. I tried Storm-0.8.1 and Storm-0.9.0-wip21. The
>> results show that:
>> 1) Storm-0.8.1 is okay for the jar topology, no matter with or
>> without dependencies.
>> 2) Storm-0.9.0-wip21 is okay for the jar topology without
>> dependencies, but the same issue (workers failed to start) appears for the
>> "with-dependencies" jar file.
>>
>>
>>
>> I am a newer for Storm, and I don't know the differences between
>> these two Storm version. What will result in the worker failure when trying
>> to submit the "with-dependencies" topology: Is the "with-dependencies" jar
>> file too big? or something conflicting?
>>
>> Any idea?
>> Will Storm-0.8.1 work with Storm-on-Yarn?
>> Thanks~
>>
>>
>> --
>> ------------------
>> Best Regards!
>>
>> Zheng Xue
>> *http://netlab.sysu.edu.cn/~zhxue <http://netlab.sysu.edu.cn/~zhxue>*
>> Computer Science and Technology college,Sun Yat-sen University
>> E-mail: [email protected], [email protected]
>>
>
>
--
------------------
Best Regards!
Zheng Xue
*http://netlab.sysu.edu.cn/~zhxue <http://netlab.sysu.edu.cn/~zhxue>*
Computer Science and Technology college,Sun Yat-sen University
E-mail: [email protected], [email protected]
