Can you share what version you are on and what version you are trying to upgrade to?
Thanks Parth On 9/29/15, 10:55 AM, "Matthias J. Sax" <[email protected]> wrote: >As far as I know, running topologies are restarted by Nimbus if the >cluster goes online again. I have never tested it for a deactivated >topology. But I would guess, that there is no difference. > >-Matthias > >On 09/29/2015 07:46 PM, Stephen Powis wrote: >> I have no idea what happens if you bring down all of the nodes in the >> cluster while the topologies are deactivated. I'd suggest testing it >> and seeing, or maybe someone else can speak up? >> >> Also depending on the version of storm you're upgrading from, there may >> be different steps involved that may complicate things. >> >> See release notes around upgrading from 0.8.x to 0.9.0: >> >>https://storm.apache.org/2013/12/08/storm090-released.html#api-compatibil >>ity-and-upgrading >> for just an example. >> >> Additionally depending on if the storm client API changes significantly >> between versions, it may require recompiling existing topology code >> against the new API version before it can run properly on the new storm >> cluster version. Taking a wild guess... this probably really only will >> be a problem when upgrading major versions, and less of a concern for >> minor version upgrades, but again I don't really know that for sure. >> >> >> On Tue, Sep 29, 2015 at 1:36 PM, Garcia-Contractor, Joseph (CORP) >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> Stephen, ____ >> >> __ __ >> >> Thank you for the response! Helps out a lot.____ >> >> __ __ >> >> So a further question. And forgive my lack of knowledge here, I am >> not the one using Storm, only deploying and running it, so I don¹t >> understand all the reasoning behind why something is done a certain >> way in Storm.____ >> >> __ __ >> >> Let¹s say I have deactivated all the topologies. Is it necessary to >> then kill the topology? Could I not just wait a set amount of time >> to ensure the tuples have cleared, say 5 minutes, and then bring >> down the nodes?____ >> >> __ __ >> >> The reason I ask this is because it is a lot easier to activate the >> topologies after the nodes are back up with a non-interactive >> script. I would like to avoid using ³storm jar² to load the >> topology because that means I need to hard code stuff into my >> scripts or come up with a separate conf file for my script. See my >> current code below:____ >> >> __ __ >> >> function deactivate_topos {____ >> >> STORM_TOPO_STATUS=$(storm list | sed -n -e >> >>'/^-------------------------------------------------------------------/,$ >>p' >> | sed -e >> >>'/^-------------------------------------------------------------------/d' >> | awk '{print $1 ":" $2}')____ >> >> __ __ >> >> for i in $STORM_TOPO_STATUS____ >> >> do____ >> >> IFS=':' read TOPO_NAME TOPO_STATUS <<< "$i"____ >> >> echo "$TOPO_NAME $TOPO_STATUS"____ >> >> if [ $TOPO_STATUS = 'ACTIVE' ]; then____ >> >> storm deactivate ${TOPO_NAME}____ >> >> fi____ >> >> storm list | sed -n -e >> >>'/^-------------------------------------------------------------------/,$ >>p'____ >> >> done____ >> >> }____ >> >> __ __ >> >> function activate_topos {____ >> >> STORM_TOPO_STATUS=$(storm list | sed -n -e >> >>'/^-------------------------------------------------------------------/,$ >>p' >> | sed -e >> >>'/^-------------------------------------------------------------------/d' >> | awk '{print $1 ":" $2}')____ >> >> for i in $STORM_TOPO_STATUS____ >> >> do____ >> >> IFS=':' read TOPO_NAME TOPO_STATUS <<< "$i"____ >> >> echo "$TOPO_NAME $TOPO_STATUS"____ >> >> if [ $TOPO_STATUS = 'INACTIVE' ]; then____ >> >> storm activate ${TOPO_NAME}____ >> >> fi____ >> >> storm list | sed -n -e >> >>'/^-------------------------------------------------------------------/,$ >>p'____ >> >> done____ >> >> }____ >> >> __ __ >> >> *From:*Stephen Powis [mailto:[email protected] >> <mailto:[email protected]>] >> *Sent:* Tuesday, September 29, 2015 12:45 PM >> >> >> *To:* [email protected] <mailto:[email protected]> >> *Subject:* Re: Starting and stopping storm____ >> >> __ __ >> >> I would imagine the safest way would be to elect to deactivate each >> running topology, which should make your spouts stop emitting >> tuples. You'd wait for all of the currently processing tuples to >> finish processing, and then kill the topology. >> >> If tuples get processed quickly in your topologies, you can >> effectively do this by selecting kill and giving it a long enough >> wait time. IE -- Telling storm to kill your topology after 30 >> seconds means it will deactivate your spouts for 30 seconds, waiting >> for existing tuples to finish getting processed, and then kill off >> the topology. >> >> Then bring down each node, upgrade it, bring it back online and >> resubmit your topologies. ____ >> >> __ __ >> >> On Tue, Sep 29, 2015 at 10:02 AM, Garcia-Contractor, Joseph (CORP) >> <[email protected] >> <mailto:[email protected]>> wrote:____ >> >> I don't think I got my question across right or I am confused. >> >> Let me break this down in a more simple fashion. >> >> I have a Storm Cluster named "The Quiet Storm" ;) here is what it >> consists of: >> >> ****** >> Server ZK1: Running Zookeeper >> Server ZK2: Running Zookeeper >> Server ZK3: Running Zookeeper >> >> Server N1: SupervisorD running Storm Nimbus >> >> Server S1: SupervisorD running Storm Supervisor with 4 workers. >> Server S2: SupervisorD running Storm Supervisor with 4 workers. >> Server S3: SupervisorD running Storm Supervisor with 4 workers. >> ****** >> >> Now the "The Quiet Storm" can have 1-n number of topologies running >> on it. >> >> I need to shut down all the servers in the cluster for maintenance. >> What is the procedure to do this without doing harm to the currently >> running topologies? >> >> Thank you, >> >> Joe >> >> -----Original Message----- >> From: Matthias J. Sax [mailto:[email protected] >> <mailto:[email protected]>] >> Sent: Monday, September 28, 2015 12:15 PM >> To: [email protected] <mailto:[email protected]> >> Subject: Re: Starting and stopping storm >> >> Hi, >> >> as always: it depends. ;) >> >> Storm itself clear ups its own resources just fine. However, if the >> running topology needs to clean-up/release resources before it is >> shut down, Storm is not of any help. Even if there is a Spout/Bolt >> cleanup() method, Storm does not guarantee that it will be called. >> >> Thus, using "storm deactivate" is a good way to achieve proper >>cleanup. >> However, the topology must provide some code for it, too. On the >> call to Spout.deactivate(), it must emit a special "clean-up" >> message (that you have to design by yourself) that must propagate >> through the whole topology, ie, each bolt must forward this message >> to all its output streams. Furthermore, bolts must to the clean-up >> if they receive this message. >> >> Long story short: "storm deactivate" before "storm kill" makes only >> sense if the topology requires proper cleanup and if the topology >> itself can react/cleanup properly on Spout.deactivate(). >> >> Using "storm activate" in not necessary in any case. >> >> -Matthias >> >> >> On 09/28/2015 05:08 PM, Garcia-Contractor, Joseph (CORP) wrote: >> > Hi all, >> > >> > >> > >> > I am a DevOps guy and I need implement a storm >>cluster >> > with the proper start and stop init scripts on a Linux server. I >> > already went through the documentation and it seems simple >>enough. I >> > am using supervisor as my process manager. I am however having a >> > debate with one of the developers using Storm on the proper way to >> > shutdown Storm and I am hoping that you fine folks can help us out >> in this regard. >> > >> > >> > >> > The developer believes that before you tell >>supervisor >> > to kill (SIGTERM) the storm workers, supervisor, and nimbus, you >>must >> > first issue a "storm deactivate topology-name", then tell >>supervisor >> > to kill all the various processes. He believes this because he >> > doesn't know if Storm will do an orderly shutdown on SIGTERM and >>that >> > there is a chance that something will get screwed up. This also >>means >> > that when you start storm, after nimbus is up, you need to issue a >> > ""storm activate topology-name". >> > >> > >> > >> > I am of the belief that because of storms fast >>fail and >> > because it guarantees data processing, none of that is necessary >>and >> > that you can just tell supervisor to stop the process. >> > >> > >> > >> > So who is right here? >> > >> > >>---------------------------------------------------------------------- >> > -- This message and any attachments are intended only for the use >>of >> > the addressee and may contain information that is privileged and >> > confidential. If the reader of the message is not the intended >> > recipient or an authorized representative of the intended >>recipient, >> > you are hereby notified that any dissemination of this >>communication >> > is strictly prohibited. If you have received this communication in >> > error, notify the sender immediately by return email and delete >>the >> > message and any attachments from your system. >> >> >>---------------------------------------------------------------------- >> This message and any attachments are intended only for the use of >> the addressee and may contain information that is privileged and >> confidential. If the reader of the message is not the intended >> recipient or an authorized representative of the intended recipient, >> you are hereby notified that any dissemination of this communication >> is strictly prohibited. If you have received this communication in >> error, notify the sender immediately by return email and delete the >> message and any attachments from your system.____ >> >> __ __ >> >> >
