We are currently testing SparkR on Windows and it seems several problems
are being identified time to time. Although It seems it is not easy to
automate Spark's tests in Scala on Windows because I think we should
introduce a proper change detection to run only related tests rather than
running whole tests and etc., I think it is feasible to automate PySpark
tests on Windows because it does not take so long time.
I believe there are pretty many data analysts using SparkR on Windows but I
am not sure how many guys are using PySpark on Windows.
Currently, there are some issues with SparkR testing automation (e.g.
triggering some builds/tests on PRs not related with SparkR) so we might
have to try this latter but I just want to know and make sure before my
future PR and taking a look about this futher and then I would like to add
this thread as a reference.
I would appreciate if any of you shares your opinion/experience or some
issues on Windows in PySpark.
Thank you very much.