I tried clearing the local state and also restarted storm. Still getting
Illegal Argument Exception. Since jar is not getting uploaded due to
illegal argument exception, the topology does not exist in the zookeeper.
The jar is getting uploaded from tomcat to nimbus inbox. Its not getting
copied to storm dist folder. Im posting the recent stack trace where look
into highlighted parts in bold. The jar uploaded starts with <stormjar-e7>
while the jar that is trying to get transferred is <stormjar-f8>. Is storm
caching old jars and trying to run? Our team also saw this storm issue
https://issues.apache.org/jira/browse/STORM-467 which is still unresolved
where they mention trying to execute non-existent jars as the issue.
2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from client
to
/home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar
2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file
from client:
*/home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar*
2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology
submission for topology-1431056498295 with conf {"storm.id"
"topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host"
"10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host"
"10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers"
["10.237.247.3"], "topology.acker.executors" nil,
"topology.kryo.decorators" (), "topology.name" "topology-1431056498295",
"topology.debug" true, "topology.kryo.register" nil,
"storm.elasticsearch.hosts" "10.237.247.13:9300", "
storm.elasticsearch.cluster.name" "HAP_Development_1.0", "topology.workers"
2, "nimbus.thrift.port" 6627, "storm.thrift.transport"
"backtype.storm.security.auth.SimpleTransportPlugin",
"topology.max.task.parallelism" nil}
2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission
exception. (topology name='topology-1431056498295')
#<IllegalArgumentException java.lang.IllegalArgumentException:
*/home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar*
to copy to
/home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
not exist!>
2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR] Unexpected
exception while invoking!
java.lang.IllegalArgumentException:
/home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar
to copy to
/home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
not exist!
at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
~[storm-core-0.9.4.jar:0.9.4]
at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
~[storm-core-0.9.4.jar:0.9.4]
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
~[storm-core-0.9.4.jar:0.9.4]
at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
~[storm-core-0.9.4.jar:0.9.4]
at
org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
[storm-core-0.9.4.jar:0.9.4]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <[email protected]>
wrote:
> If that doesn't work, then you might need to remove the state in zookeeper
> related to this topology.
> i.e.,
>
> - /storm/storms/<topology-id>
> - /storm/assignments/<topology-id>
>
> - Erik
>
> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <[email protected]>
> wrote:
>
>> Try deleting your local state folder in supervisors and storm nimbus data
>> folder, looks like its trying to find topology which does not exist.
>>
>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <
>> [email protected]> wrote:
>>
>>> Hi
>>>
>>> While checking nimbus log, Im getting the following stack track trace..
>>> The previous stack trace was while running from tomcat application , which
>>> generates an illegal argument exception in nimbus logs
>>>
>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR]
>>> Unexpected exception while invoking!
>>> java.lang.IllegalArgumentException:
>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
>>> to copy to
>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
>>> not exist!
>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>>> [storm-core-0.9.4.jar:0.9.4]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> [na:1.7.0_45]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> [na:1.7.0_45]
>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>
>>>
>>>
>>>
>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <[email protected]> wrote:
>>>
>>>> You can check if there are free workers available to run the topology
>>>> or not.
>>>>
>>>> Thanks,
>>>> Nishu Tayal
>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <[email protected]> wrote:
>>>>
>>>>> Your nimbus has gone down, probably because there is no space left in
>>>>> your machine.
>>>>>
>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> The zookeeper exists in the same machine along with storm. So I guess
>>>>>> the ports are accessible.
>>>>>>
>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <[email protected]
>>>>>> > wrote:
>>>>>>
>>>>>>> I would assume it's an issue with zookeeper. Have you checked if the
>>>>>>> relevant ports are open?
>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> We are using Storm Submitter via tomcat to submit topology to storm
>>>>>>>> cluster. The jar does not get uploaded to storm in the same machine as
>>>>>>>> well
>>>>>>>> as a different machine. We are using apache-storm -0.9.4.This error
>>>>>>>> occurs
>>>>>>>> intermittently. Following is the stack trace
>>>>>>>>
>>>>>>>> java.lang.RuntimeException:
>>>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>>>> at
>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>>>> at
>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>>>> at
>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>>>> at
>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>>>> at
>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>> ... 3 more
>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>>>> at
>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>>>> at
>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>>>> at
>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>>>> ... 8 more
>>>>>>>>
>>>>>>>> We are not able to reproduce the issue for a particular scenario.
>>>>>>>> It occurs randomly. Can u throw some light on the issue? We have
>>>>>>>> checked
>>>>>>>> the firewall settings and there is nothing blocked.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Aarthi R
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Aarthi R
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Aarthi R
>>>
>>
>>
>
--
Aarthi R
