Could you show your codes? Did you use `StreamingContext.awaitTermination`? If so, it will return if any exception happens.
On Wed, Jan 13, 2016 at 11:47 PM, Triones,Deng(vip.com) < triones.d...@vipshop.com> wrote: > What’s more, I am running a 7*24 hours job , so I won’t call System.exit() > by myself. So I believe somewhere of the driver kill itself > > > > *发件人:* 邓刚[技术中心] > *发送时间:* 2016年1月14日 15:45 > *收件人:* 'Yogesh Mahajan' > *抄送:* user > *主题:* 答复: 答复: spark streaming context trigger invoke stop why? > > > > Thanks for your response, ApplicationMaster is only for yarn mode. I am > using standalone mode. Could you kindly please let me know where trigger > the shutdown hook? > > > > *发件人:* Yogesh Mahajan [mailto:ymaha...@snappydata.io > <ymaha...@snappydata.io>] > *发送时间:* 2016年1月14日 12:42 > *收件人:* 邓刚[技术中心] > *抄送:* user > *主题:* Re: 答复: spark streaming context trigger invoke stop why? > > > > All the action happens in ApplicationMaster expecially in run method > > Check ApplicationMaster#startUserApplication : userThread(Driver) which > invokes ApplicationMaster#finish method. You can also try System.exit in > your program > > > > Regards, > > Yogesh Mahajan, > > SnappyData Inc, snappydata.io > > > > On Thu, Jan 14, 2016 at 9:56 AM, Yogesh Mahajan <ymaha...@snappydata.io> > wrote: > > Hi Triones, > > > > Check the org.apache.spark.util.ShutdownHookManager : It adds this > ShutDownHook when you start a StreamingContext > > > > Here is the code in StreamingContext.start() > > > > shutdownHookRef = ShutdownHookManager.addShutdownHook( > > StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown) > > > > Also looke at the following def in StreamingContext which actually stops > the context from shutdown hook : > > private def stopOnShutdown(): Unit = { > > val stopGracefully = > conf.getBoolean("spark.streaming.stopGracefullyOnShutdown", false) > > logInfo(s"Invoking stop(stopGracefully=$stopGracefully) from shutdown > hook") > > // Do not stop SparkContext, let its own shutdown hook stop it > > stop(stopSparkContext = false, stopGracefully = stopGracefully) > > } > > > > Regards, > > Yogesh Mahajan, > > SnappyData Inc, snappydata.io > > > > On Thu, Jan 14, 2016 at 8:55 AM, Triones,Deng(vip.com) < > triones.d...@vipshop.com> wrote: > > More info > > > > I am using spark version 1.5.2 > > > > > > *发件人:* Triones,Deng(vip.com) [mailto:triones.d...@vipshop.com] > *发送时间:* 2016年1月14日 11:24 > *收件人:* user > *主题:* spark streaming context trigger invoke stop why? > > > > Hi all > > As I saw the driver log, the task failed 4 times in a stage, the > stage will be dropped when the input block was deleted before make use of. > After that the StreamingContext invoke stop. Does anyone know what kind of > akka message trigger the stop or which code trigger the shutdown hook? > > > > > > Thanks > > > > > > > > > > Driver log: > > > > Job aborted due to stage failure: Task 410 in stage 215.0 failed 4 times > > [org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking > stop(stopGracefully=false) from shutdown hook > > > 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ > This communication is intended only for the addressee(s) and may contain > information that is privileged and confidential. You are hereby notified > that, if you are not an intended recipient listed above, or an authorized > employee or agent of an addressee of this communication responsible for > delivering e-mail messages to an intended recipient, any dissemination, > distribution or reproduction of this communication (including any > attachments hereto) is strictly prohibited. If you have received this > communication in error, please notify us immediately by a reply e-mail > addressed to the sender and permanently delete the original e-mail > communication and any attachments from all storage devices without making > or otherwise retaining a copy. > > > 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ > This communication is intended only for the addressee(s) and may contain > information that is privileged and confidential. You are hereby notified > that, if you are not an intended recipient listed above, or an authorized > employee or agent of an addressee of this communication responsible for > delivering e-mail messages to an intended recipient, any dissemination, > distribution or reproduction of this communication (including any > attachments hereto) is strictly prohibited. If you have received this > communication in error, please notify us immediately by a reply e-mail > addressed to the sender and permanently delete the original e-mail > communication and any attachments from all storage devices without making > or otherwise retaining a copy. > > > > > 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ > This communication is intended only for the addressee(s) and may contain > information that is privileged and confidential. You are hereby notified > that, if you are not an intended recipient listed above, or an authorized > employee or agent of an addressee of this communication responsible for > delivering e-mail messages to an intended recipient, any dissemination, > distribution or reproduction of this communication (including any > attachments hereto) is strictly prohibited. If you have received this > communication in error, please notify us immediately by a reply e-mail > addressed to the sender and permanently delete the original e-mail > communication and any attachments from all storage devices without making > or otherwise retaining a copy. >
