Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23055#discussion_r237714454
--- Diff: python/pyspark/worker.py ---
@@ -22,7 +22,12 @@
import os
import sys
import time
-import resource
+# 'resource' is a Unix specific module.
+has_resource_module = True
+try:
+ import resource
+except ImportError:
+ has_resource_module = False
--- End diff --
What is the behavior you want? What do you mean "python can't fail"?
If the memory limit is set by the user, you have two options when the
python side cannot do anything with it:
- ignore it (the current patch)
- raise an error
Both can be done from the python side and do not require any checks in the
JVM.
Checking in the JVM is the wrong thing. You're hardcoding the logic that
this feature can never work on Windows. And if someone finds out that it can,
then you'll have to change it, and that check was basically useless, since an
equivalent check exists on the python side.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
