Ohh thats bad of me. Really how could I miss nohup!
Anyways Thanks Kosala! On Wed, Feb 11, 2015 at 5:55 AM, Kosala Dissanayake <[email protected]> wrote: > Running a process in the background does not mean that it will not get > terminated when you log out of the terminal. > > For that you need to prepend 'nohup' to your command. This will run the > command with hangup signals ignored. > > On Wed, Feb 11, 2015 at 12:31 AM, Vineet Mishra <[email protected]> > wrote: > >> Hi Harsha, >> >> Thanks it worked! >> But I was wondering the reason why it was failing earlier even when I was >> initiating it as a background process something like, >> >> bin/storm nimbus & >> >> and likewise. . . >> >> Thanks! >> >> On Mon, Feb 9, 2015 at 6:22 AM, Harsha <[email protected]> wrote: >> >>> >>> Are you running nimbus, supervisors in background? looks like you are >>> sshing into machines and running ./bin/storm nimbus in foreground which >>> will get killed when you exit the ssh session. Make sure you use >>> supervisord http://supervisord.org/ to run nimbus, supervisors. >>> >>> On Sat, Feb 7, 2015, at 11:04 AM, Vineet Mishra wrote: >>> >>> >>> Including Subject! >>> >>> On Sun, Feb 8, 2015 at 12:33 AM, Vineet Mishra <[email protected]> >>> wrote: >>> >>> Hi All, >>> >>> I am running a Kafka Storm topology in distributed mode, its running >>> good for the initial run when I start the cluster(3 node cluster) deploy >>> the Storm Topology and leave it to run. >>> There are often times the whole cluster goes down(nimbus, supervisor, >>> workers) and this is most of the time happening when I submit the topology >>> to run and disconnect my session from the machine. >>> >>> I could fairly notice that on one of the worker node its throwing the >>> error : >>> >>> java.lang.RuntimeException: java.lang.RuntimeException: Client is being >>> closed, and does not take requests any more >>> >>> Config and Detailed Stack Trance is provided below. >>> >>> Node 1 - Nimbus, UI >>> Node 2 - Supervisor, Worker >>> Node 3 - Supervisor, Worker >>> >>> 2015-02-07T23:01:25.884+0530 b.s.d.worker [INFO] Shutting down worker >>> KafkaConsumerTopologyy-1-1423329275 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c >>> 6703 >>> 2015-02-07T23:01:25.884+0530 b.s.m.n.Client [INFO] Closing Netty Client >>> Netty-Client-ip-20-0-0-78/20.0.0.78:6703 >>> 2015-02-07T23:01:25.885+0530 b.s.m.n.Client [INFO] Waiting for pending >>> batchs to be sent with Netty-Client-ip-20-0-0-78/20.0.0.78:6703..., >>> timeout: 600000ms, pendings: 0 >>> 2015-02-07T23:01:25.886+0530 b.s.d.worker [INFO] Shutting down receive >>> thread >>> 2015-02-07T23:01:25.886+0530 o.a.s.c.r.ExponentialBackoffRetry [WARN] >>> maxRetries too large (300). Pinning to 29 >>> 2015-02-07T23:01:25.886+0530 b.s.u.StormBoundedExponentialBackoffRetry >>> [INFO] The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries >>> [300] >>> 2015-02-07T23:01:25.887+0530 b.s.m.n.Client [INFO] New Netty Client, >>> connect to localhost, 6703, config: , buffer_size: 5242880 >>> 2015-02-07T23:01:25.887+0530 b.s.m.n.Client [INFO] Reconnect started for >>> Netty-Client-localhost/127.0.0.1:6703... [0] >>> 2015-02-07T23:01:25.887+0530 b.s.m.loader [INFO] Shutting down >>> receiving-thread: [KafkaConsumerTopologyy-1-1423329275, 6703] >>> 2015-02-07T23:01:25.893+0530 b.s.m.n.Client [INFO] connection >>> established to a remote host Netty-Client-localhost/127.0.0.1:6703, >>> [id: 0x8f71aaa0, /127.0.0.1:59427 => localhost/127.0.0.1:6703] >>> 2015-02-07T23:01:25.893+0530 b.s.m.n.Client [INFO] Closing Netty Client >>> Netty-Client-localhost/127.0.0.1:6703 >>> 2015-02-07T23:01:25.893+0530 b.s.m.n.Client [INFO] Waiting for pending >>> batchs to be sent with Netty-Client-localhost/127.0.0.1:6703..., >>> timeout: 600000ms, pendings: 0 >>> 2015-02-07T23:01:25.894+0530 b.s.m.loader [INFO] Waiting for >>> receiving-thread:[KafkaConsumerTopologyy-1-1423329275, 6703] to die >>> 2015-02-07T23:01:25.895+0530 b.s.m.loader [INFO] Shutdown >>> receiving-thread: [KafkaConsumerTopologyy-1-1423329275, 6703] >>> 2015-02-07T23:01:25.895+0530 b.s.d.worker [INFO] Shut down receive thread >>> 2015-02-07T23:01:25.895+0530 b.s.d.worker [INFO] Terminating messaging >>> context >>> 2015-02-07T23:01:25.895+0530 b.s.d.worker [INFO] Shutting down executors >>> 2015-02-07T23:01:25.895+0530 b.s.d.executor [INFO] Shutting down >>> executor KafkaSpout:[3 3] >>> 2015-02-07T23:01:25.896+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.896+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.897+0530 b.s.util [ERROR] Async loop died! >>> java.lang.RuntimeException: java.lang.RuntimeException: Client is being >>> closed, and does not take requests any more >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.disruptor$consume_loop_STAR_$fn__1460.invoke(disruptor.clj:94) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] >>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] >>> Caused by: java.lang.RuntimeException: Client is being closed, and does >>> not take requests any more >>> at backtype.storm.messaging.netty.Client.send(Client.java:185) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__3730$fn__3731.invoke(worker.clj:330) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__3730.invoke(worker.clj:328) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.disruptor$clojure_handler$reify__1447.onEvent(disruptor.clj:58) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> ... 6 common frames omitted >>> 2015-02-07T23:01:25.900+0530 o.a.z.ZooKeeper [INFO] Session: >>> 0x14b58cc25de0115 closed >>> 2015-02-07T23:01:25.900+0530 o.a.z.ClientCnxn [INFO] EventThread shut >>> down >>> 2015-02-07T23:01:25.900+0530 b.s.d.executor [INFO] Shut down executor >>> KafkaSpout:[3 3] >>> 2015-02-07T23:01:25.901+0530 b.s.d.executor [INFO] Shutting down >>> executor KafkaSpout:[5 5] >>> 2015-02-07T23:01:25.901+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.901+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.903+0530 o.a.z.ZooKeeper [INFO] Session: >>> 0x14b58cc25de0117 closed >>> 2015-02-07T23:01:25.903+0530 o.a.z.ClientCnxn [INFO] EventThread shut >>> down >>> 2015-02-07T23:01:25.903+0530 b.s.d.executor [INFO] Shut down executor >>> KafkaSpout:[5 5] >>> 2015-02-07T23:01:25.903+0530 b.s.d.executor [INFO] Shutting down >>> executor KafkaSpout:[7 7] >>> 2015-02-07T23:01:25.904+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.904+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.905+0530 o.a.z.ZooKeeper [INFO] Session: >>> 0x14b58cc25de0114 closed >>> 2015-02-07T23:01:25.905+0530 o.a.z.ClientCnxn [INFO] EventThread shut >>> down >>> 2015-02-07T23:01:25.906+0530 b.s.d.executor [INFO] Shut down executor >>> KafkaSpout:[7 7] >>> 2015-02-07T23:01:25.906+0530 b.s.d.executor [INFO] Shutting down >>> executor KafkaSpout:[9 9] >>> 2015-02-07T23:01:25.906+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.906+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.906+0530 b.s.util [ERROR] Halting process: ("Async >>> loop died!") >>> java.lang.RuntimeException: ("Async loop died!") >>> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na] >>> at >>> backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:92) >>> [storm-core-0.9.3.jar:0.9.3] >>> at backtype.storm.util$async_loop$fn__464.invoke(util.clj:473) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] >>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] >>> 2015-02-07T23:01:25.908+0530 o.a.z.ZooKeeper [INFO] Session: >>> 0x14b58cc25de0116 closed >>> 2015-02-07T23:01:25.908+0530 o.a.z.ClientCnxn [INFO] EventThread shut >>> down >>> 2015-02-07T23:01:25.908+0530 b.s.d.executor [INFO] Shut down executor >>> KafkaSpout:[9 9] >>> 2015-02-07T23:01:25.909+0530 b.s.d.executor [INFO] Shutting down >>> executor __acker:[11 11] >>> 2015-02-07T23:01:25.909+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.909+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.909+0530 b.s.d.executor [INFO] Shut down executor >>> __acker:[11 11] >>> 2015-02-07T23:01:25.909+0530 b.s.d.executor [INFO] Shutting down >>> executor __system:[-1 -1] >>> 2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.910+0530 b.s.d.executor [INFO] Shut down executor >>> __system:[-1 -1] >>> 2015-02-07T23:01:25.910+0530 b.s.d.executor [INFO] Shutting down >>> executor FileBolt:[1 1] >>> 2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted! >>> 2015-02-07T23:01:25.911+0530 b.s.d.executor [INFO] Shut down executor >>> FileBolt:[1 1] >>> 2015-02-07T23:01:25.911+0530 b.s.d.worker [INFO] Shut down executors >>> 2015-02-07T23:01:25.916+0530 b.s.d.worker [INFO] Shutting down transfer >>> thread >>> >>> URGENT CALL! >>> >>> Thanks! >>> >>> >>> >>> >>> >> >> >
