Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2743#discussion_r18939839
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -53,7 +53,8 @@ private[spark] class PythonRDD(
extends RDD[Array[Byte]](parent) {
val bufferSize = conf.getInt("spark.buffer.size", 65536)
- val reuse_worker = conf.getBoolean("spark.python.worker.reuse", true)
+ val reuseWorker = conf.getBoolean("spark.python.worker.reuse", true)
+ val memoryLimit = conf.getInt("spark.executor.python.memory.limit", 0)
--- End diff --
Also - will it be confusing that we require the user to specify the memory
differently than in every other place? We should probably write a python
function for interpreting memory strings ("128m", "15g") etc. As it is now the
unit is unspecified, it's also inconsistent with other configs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
