Hi all, We are currently testing SparkR on Windows[1] and it seems several problems are being identified time to time. Although It seems it is not easy to automate Spark's tests in Scala on Windows because I think we should introduce a proper change detection to run only related tests rather than running whole tests and etc., I think it is feasible to automate PySpark tests on Windows because it does not take so long time.
I believe there are pretty many data analysts using SparkR on Windows but I am not sure how many guys are using PySpark on Windows. Currently, there are some issues with SparkR testing automation (e.g. triggering some builds/tests on PRs not related with SparkR) so we might have to try this latter but I just want to know and make sure before my future PR and taking a look about this futher and then I would like to add this thread as a reference. I would appreciate if any of you shares your opinion/experience or some issues on Windows in PySpark. Thank you very much. [1] https://github.com/apache/spark/pull/14859