[
https://issues.apache.org/jira/browse/SPARK-32572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175376#comment-17175376
]
Hyukjin Kwon commented on SPARK-32572:
--------------------------------------
[~fokko] the problem is that PySpark runs the tests in parallel via our own
testing script https://github.com/apache/spark/blob/master/python/run-tests.py
which runs each tests in a separate subprocess.
The root cause would probably be related to how we invoke the Python tests.
Yeah, it would be nice if we can fix it with a minimal change.
> Run all the tests at once, instead of having separate entrypoints.
> ------------------------------------------------------------------
>
> Key: SPARK-32572
> URL: https://issues.apache.org/jira/browse/SPARK-32572
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.0.0
> Reporter: Fokko Driesprong
> Priority: Major
>
> Started with this comment thread:
> https://github.com/apache/spark/pull/29121/files#r456683561
> Each file is invoked separately and has a separate entry point:
> [https://github.com/apache/spark/blob/master/python/pyspark/ml/tests/test_wrapper.py#L120]
> We would replace
> [https://github.com/apache/spark/blob/master/dev/run-tests.py#L470] this
> function call to the subprocess with something that would invoke the python
> tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
