[ https://issues.apache.org/jira/browse/SPARK-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-2764. ------------------------------- Resolution: Fixed Fix Version/s: 1.1.0 > Simplify process structure of PySpark daemon / worker launching process > ----------------------------------------------------------------------- > > Key: SPARK-2764 > URL: https://issues.apache.org/jira/browse/SPARK-2764 > Project: Spark > Issue Type: Improvement > Components: PySpark > Reporter: Josh Rosen > Assignee: Josh Rosen > Fix For: 1.1.0 > > > PySpark's daemon-based worker factory has a very complicated process > structure that I've always found confusing. The per-java-worker daemon.py > process launches a numCores-sized pool of subprocesses, and those > subprocesses launching the actual worker processes that process data. > I think we can simplify this by having daemon.py launch the workers directly > without this extra layer of indirection. See my comments on the pull request > that introduced daemon.py: https://github.com/mesos/spark/pull/563 -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org