[
https://issues.apache.org/jira/browse/SPARK-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14622623#comment-14622623
]
Josh Rosen commented on SPARK-8976:
-----------------------------------
I think that this problem is Windows-specific. The code near line 149 of
worker.py will typically not be executed on non-Windows machines as long as
{{spark.python.worker.reuse=true}} (the default).
I think the right fix is adding a regression test which tries running simple
PySpark jobs with {{spark.python.worker.reuse=false}}, then fixing the
underlying bug by passing "rwb" instead of "a+".
If we get a regression test working on Jenkins, then we'll be able to verify
that the fix is safe for Python 2 and 3 because Jenkins tests both of those
Python versions.
Would you like to submit a pull request for this? I'd do it myself but I'm a
bit swamped with other work right now.
> Python 3 crash: ValueError: invalid mode 'a+' (only r, w, b allowed)
> --------------------------------------------------------------------
>
> Key: SPARK-8976
> URL: https://issues.apache.org/jira/browse/SPARK-8976
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.4.0
> Environment: Windows 7
> Reporter: Olivier Delalleau
>
> See Github report:
> https://github.com/apache/spark/pull/5173#issuecomment-113410652
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
