Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
I manually tested after setting `spark.python.daemon.module` to
`nonexistantmodule`. It shows the error message like this:
```python
>>> spark.range(1).rdd.map(lambda x: x).collect()
```
```
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/rdd.py", line 824, in collect
port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/.../spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py",
line 1160, in __call__
File "/.../spark/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/.../spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line
320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0
(TID 1, localhost, executor driver): org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: No module named nonexistantmodule
PYTHONPATH was:
/.../spark/python/lib/pyspark.zip:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.3.0-SNAPSHOT.jar:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/python/:
java.io.EOFException
...
Driver stacktrace:
...
Caused by: org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: No module named nonexistantmodule
PYTHONPATH was:
/.../spark/python/lib/pyspark.zip:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.3.0-SNAPSHOT.jar:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/python/:
java.io.EOFException
...
... 1 more
18/01/08 15:54:06 WARN TaskSetManager: Lost task 6.0 in stage 0.0 (TID 6,
localhost, executor driver): TaskKilled (Stage cancelled)
...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
