Sohaib Iftikhar created SPARK-12676:
---------------------------------------
Summary: There is no way to stop a spark-streaming job from a
worker in case of errors.
Key: SPARK-12676
URL: https://issues.apache.org/jira/browse/SPARK-12676
Project: Spark
Issue Type: Improvement
Components: Streaming
Affects Versions: 1.6.0
Environment: All operating systems
Reporter: Sohaib Iftikhar
Priority: Critical
Consider and application that reads data from an external source and writes it
to HDFS. If for some reason the namenode crashes the job keeps reading data
from that source but does throws errors upon writing to HDFS which eventually
leads to data loss. It would be desirable to supply some switch to the workers
which would kill the job which could trigger some alert to the admin of the
system and the job can be restarted when problem with the name node is fixed.
See related question here:
http://stackoverflow.com/questions/34195453/sparkstreaming-shut-down-job-in-case-of-error
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
