Did you try deleting the ZooKeeper state like I recommended? - Erik
On Friday, May 8, 2015, aarthi ranganathan <[email protected]> wrote: > We did a restart by deleting the entire storm local directory folder. The > inbox folder is empty. > On Fri, May 8, 2015 at 1:48 PM, bhargav sarvepalli <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> Can you check the contents of inbox folder? and probably clean it >> >> On Fri, May 8, 2015 at 1:11 PM, aarthi ranganathan < >> [email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> I tried clearing the local state and also restarted storm. Still >>> getting Illegal Argument Exception. Since jar is not getting uploaded due >>> to illegal argument exception, the topology does not exist in the zookeeper. >>> The jar is getting uploaded from tomcat to nimbus inbox. Its not getting >>> copied to storm dist folder. Im posting the recent stack trace where look >>> into highlighted parts in bold. The jar uploaded starts with <stormjar-e7> >>> while the jar that is trying to get transferred is <stormjar-f8>. Is storm >>> caching old jars and trying to run? Our team also saw this storm issue >>> https://issues.apache.org/jira/browse/STORM-467 which is still >>> unresolved where they mention trying to execute non-existent jars as the >>> issue. >>> >>> >>> 2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from >>> client to >>> /home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar >>> >>> 2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file >>> from client: >>> */home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar* >>> >>> 2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology >>> submission for topology-1431056498295 with conf {"storm.id" >>> "topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host" >>> "10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host" >>> "10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers" >>> ["10.237.247.3"], "topology.acker.executors" nil, >>> "topology.kryo.decorators" (), "topology.name" >>> "topology-1431056498295", "topology.debug" true, "topology.kryo.register" >>> nil, "storm.elasticsearch.hosts" "10.237.247.13:9300", " >>> storm.elasticsearch.cluster.name" "HAP_Development_1.0", >>> "topology.workers" 2, "nimbus.thrift.port" 6627, "storm.thrift.transport" >>> "backtype.storm.security.auth.SimpleTransportPlugin", >>> "topology.max.task.parallelism" nil} >>> >>> 2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission >>> exception. (topology name='topology-1431056498295') >>> #<IllegalArgumentException java.lang.IllegalArgumentException: >>> */home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar* >>> to copy to >>> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does >>> not exist!> >>> >>> 2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR] >>> Unexpected exception while invoking! >>> >>> java.lang.IllegalArgumentException: >>> /home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar >>> to copy to >>> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does >>> not exist! >>> >>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na] >>> >>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) >>> ~[storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) >>> [storm-core-0.9.4.jar:0.9.4] >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> [na:1.7.0_45] >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> [na:1.7.0_45] >>> >>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] >>> >>> >>> On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>>> If that doesn't work, then you might need to remove the state in >>>> zookeeper related to this topology. >>>> i.e., >>>> >>>> - /storm/storms/<topology-id> >>>> - /storm/assignments/<topology-id> >>>> >>>> - Erik >>>> >>>> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli < >>>> [email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >>>>> Try deleting your local state folder in supervisors and storm nimbus >>>>> data folder, looks like its trying to find topology which does not exist. >>>>> >>>>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan < >>>>> [email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> While checking nimbus log, Im getting the following stack track >>>>>> trace.. The previous stack trace was while running from tomcat >>>>>> application >>>>>> , which generates an illegal argument exception in nimbus logs >>>>>> >>>>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR] >>>>>> Unexpected exception while invoking! >>>>>> java.lang.IllegalArgumentException: >>>>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar >>>>>> to copy to >>>>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 >>>>>> does >>>>>> not exist! >>>>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) >>>>>> ~[clojure-1.5.1.jar:na] >>>>>> at >>>>>> backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) >>>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) >>>>>> [storm-core-0.9.4.jar:0.9.4] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> [na:1.7.0_45] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> [na:1.7.0_45] >>>>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <[email protected] >>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>> >>>>>>> You can check if there are free workers available to run the >>>>>>> topology or not. >>>>>>> >>>>>>> Thanks, >>>>>>> Nishu Tayal >>>>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <[email protected] >>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>> >>>>>>>> Your nimbus has gone down, probably because there is no space left >>>>>>>> in your machine. >>>>>>>> >>>>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan < >>>>>>>> [email protected] >>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>> >>>>>>>>> The zookeeper exists in the same machine along with storm. So I >>>>>>>>> guess the ports are accessible. >>>>>>>>> >>>>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan < >>>>>>>>> [email protected] >>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>> >>>>>>>>>> I would assume it's an issue with zookeeper. Have you checked if >>>>>>>>>> the relevant ports are open? >>>>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" < >>>>>>>>>> [email protected] >>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi >>>>>>>>>>> >>>>>>>>>>> We are using Storm Submitter via tomcat to submit topology to >>>>>>>>>>> storm cluster. The jar does not get uploaded to storm in the same >>>>>>>>>>> machine >>>>>>>>>>> as well as a different machine. We are using apache-storm >>>>>>>>>>> -0.9.4.This error >>>>>>>>>>> occurs intermittently. Following is the stack trace >>>>>>>>>>> >>>>>>>>>>> java.lang.RuntimeException: >>>>>>>>>>> org.apache.thrift7.transport.TTransportException >>>>>>>>>>> at >>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141) >>>>>>>>>>> at >>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70) >>>>>>>>>>> at >>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148) >>>>>>>>>>> at >>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48) >>>>>>>>>>> at >>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111) >>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>>> ... 3 more >>>>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) >>>>>>>>>>> at >>>>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) >>>>>>>>>>> at >>>>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162) >>>>>>>>>>> at >>>>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146) >>>>>>>>>>> at >>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127) >>>>>>>>>>> ... 8 more >>>>>>>>>>> >>>>>>>>>>> We are not able to reproduce the issue for a particular >>>>>>>>>>> scenario. It occurs randomly. Can u throw some light on the issue? >>>>>>>>>>> We have >>>>>>>>>>> checked the firewall settings and there is nothing blocked. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Aarthi R >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Aarthi R >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Aarthi R >>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Aarthi R >>> >> >> > > > -- > Aarthi R >
