RE: Detect executor core count

[Ilya Matiach]

Hi Andrew,
I tried to do something similar to that in the LightGBM 
classifier/regressor/ranker in mmlspark package, I try to use the spark conf 
and if not configured I get the processors from the JVM directly:
https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/main/scala/LightGBMUtils.scala#L172

If you know of a better way, please let me know!

    val spark = dataset.sparkSession
    try {
      val confCores = spark.sparkContext.getConf
        .get("spark.executor.cores").toInt
      val confTaskCpus = spark.sparkContext.getConf
        .get("spark.task.cpus", "1").toInt
      confCores / confTaskCpus
    } catch {
      case _: NoSuchElementException =>
        // If spark.executor.cores is not defined, get the cores per JVM
        import spark.implicits._
        val numMachineCores = spark.range(0, 1)
          .map(_ => 
java.lang.Runtime.getRuntime.availableProcessors).collect.head
        numMachineCores
    }

Thank you, Ilya

From: Andrew Melo <andrew.m...@gmail.com>
Sent: Tuesday, June 18, 2019 11:32 AM
To: dev <dev@spark.apache.org>
Subject: Detect executor core count

Hello,

Is there a way to detect the number of cores allocated for an executor within a 
java-based InputPartitionReader?

Thanks!
Andrew