Scott Taylor created SPARK-7735:
-----------------------------------
Summary: Raise Exception on non-zero exit from pyspark pipe
commands
Key: SPARK-7735
URL: https://issues.apache.org/jira/browse/SPARK-7735
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.3.1, 1.3.0
Reporter: Scott Taylor
Priority: Minor
In pyspark errors are ignored when using the rdd.pipe function. This is
different to the scala behaviour where abnormal exit of the piped command is
raised. I have submitted a pull request on github which I believe will bring
the pyspark behaviour closer to the scala behaviour.
A simple case of where this bug may be problematic is using a network bash
utility to perform computations on an rdd. Currently, network errors will be
ignored and blank results returned when it would be more desirable to raise an
exception so that spark can retry the failed task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
