Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21977#discussion_r209823163
  
    --- Diff: 
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
    @@ -60,14 +61,20 @@ private[spark] object PythonEvalType {
      */
     private[spark] abstract class BasePythonRunner[IN, OUT](
         funcs: Seq[ChainedPythonFunctions],
    -    bufferSize: Int,
    -    reuseWorker: Boolean,
         evalType: Int,
    -    argOffsets: Array[Array[Int]])
    +    argOffsets: Array[Array[Int]],
    +    conf: SparkConf)
       extends Logging {
     
       require(funcs.length == argOffsets.length, "argOffsets should have the 
same length as funcs")
     
    +  private val bufferSize = conf.getInt("spark.buffer.size", 65536)
    +  private val reuseWorker = conf.getBoolean("spark.python.worker.reuse", 
true)
    +  // each python worker gets an equal part of the allocation. the worker 
pool will grow to the
    +  // number of concurrent tasks, which is determined by the number of 
cores in this executor.
    +  private val memoryMb = conf.get(PYSPARK_EXECUTOR_MEMORY)
    +      .map(_ / conf.getInt("spark.executor.cores", 1))
    --- End diff --
    
    Eh, actually, I believe it uses 2 space indentation in 
general(https://spark.apache.org/contributing.html / 
https://github.com/databricks/scala-style-guide#spacing-and-indentation) and I 
am pretty sure 2 spaces are more common.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to