RE: Redploying a spark streaming application
Negative. You'll want to terminate the current job. I don't have a good way of doing this. I've seen examples online of writing a python script to parse the output of ps aux and then executing a kill command. I may be mistaken but I don't believe spark has the equivalent of a "spark-kill" -Original Message- From: Ashic Mahtab [as...@live.com<mailto:as...@live.com>] Sent: Thursday, November 06, 2014 05:58 PM Eastern Standard Time To: Ganelin, Ilya; user@spark.apache.org Subject: RE: Redploying a spark streaming application Thanks Ilya. Will that terminate the currently running instance automatically? From: ilya.gane...@capitalone.com To: as...@live.com; user@spark.apache.org Date: Thu, 6 Nov 2014 17:14:33 -0500 Subject: Re: Redploying a spark streaming application You’ve basically got it. Deployment step can be simply scp-ing the file to a known location on the server and then executing a run script on the server that actually runs the spark-submit. From: Ashic Mahtab mailto:as...@live.com>> Date: Thursday, November 6, 2014 at 5:01 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: Redploying a spark streaming application Hello, I'm trying to find the best way of redeploying a spark streaming application. Ideally, I was thinking of a scenario where a build server packages up a jar and a deployment step submits it to a Spark Master. On the next successful build, the next version would get deployed taking down the previous version. What would be the best way of achieving this? Thanks, Ashic. The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
RE: Redploying a spark streaming application
Thanks Ilya. Will that terminate the currently running instance automatically? From: ilya.gane...@capitalone.com To: as...@live.com; user@spark.apache.org Date: Thu, 6 Nov 2014 17:14:33 -0500 Subject: Re: Redploying a spark streaming application You’ve basically got it. Deployment step can be simply scp-ing the file to a known location on the server and then executing a run script on the server that actually runs the spark-submit. From: Ashic Mahtab Date: Thursday, November 6, 2014 at 5:01 PM To: "user@spark.apache.org" Subject: Redploying a spark streaming application Hello, I'm trying to find the best way of redeploying a spark streaming application. Ideally, I was thinking of a scenario where a build server packages up a jar and a deployment step submits it to a Spark Master. On the next successful build, the next version would get deployed taking down the previous version. What would be the best way of achieving this? Thanks,Ashic. The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Redploying a spark streaming application
You’ve basically got it. Deployment step can be simply scp-ing the file to a known location on the server and then executing a run script on the server that actually runs the spark-submit. From: Ashic Mahtab mailto:as...@live.com>> Date: Thursday, November 6, 2014 at 5:01 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: Redploying a spark streaming application Hello, I'm trying to find the best way of redeploying a spark streaming application. Ideally, I was thinking of a scenario where a build server packages up a jar and a deployment step submits it to a Spark Master. On the next successful build, the next version would get deployed taking down the previous version. What would be the best way of achieving this? Thanks, Ashic. The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Redploying a spark streaming application
Hello,I'm trying to find the best way of redeploying a spark streaming application. Ideally, I was thinking of a scenario where a build server packages up a jar and a deployment step submits it to a Spark Master. On the next successful build, the next version would get deployed taking down the previous version. What would be the best way of achieving this? Thanks,Ashic.
