Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r207637946
--- Diff: python/pyspark/worker.py ---
@@ -259,6 +260,26 @@ def main(infile, outfile):
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
+ # set up memory limits
+ memory_limit_mb = int(os.environ.get('PYSPARK_EXECUTOR_MEMORY_MB',
"-1"))
+ total_memory = resource.RLIMIT_AS
+ try:
+ (total_memory_limit, max_total_memory) =
resource.getrlimit(total_memory)
+ msg = "Current mem: {0} of max
{1}\n".format(total_memory_limit, max_total_memory)
+ sys.stderr.write()
+
+ if memory_limit_mb > 0 and total_memory_limit < 0:
--- End diff --
I've updated to use `resource.RLIM_INFINITY`.
I think this should only set the resource limit if it isn't already set. It
is unlikely that it's already set because this is during worker initialization,
but the intent is to not cause harm if a higher-level system (i.e. container
provider) has already set the limit.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
