[ https://issues.apache.org/jira/browse/SPARK-21099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-21099: ------------------------------ Priority: Trivial (was: Major) > INFO Log Message Using Incorrect Executor Idle Timeout Value > ------------------------------------------------------------ > > Key: SPARK-21099 > URL: https://issues.apache.org/jira/browse/SPARK-21099 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.6.0, 2.1.0 > Reporter: Hazem Mahmoud > Priority: Trivial > > INFO log message is using the wrong idle timeout > (spark.dynamicAllocation.executorIdleTimeout) when printing the message that > the executor holding the RDD cache is being removed. > INFO spark.ExecutorAllocationManager: Removing executor 1 because it has been > idle for 30 seconds (new desired total will be 0) > It should be using spark.dynamicAllocation.cachedExecutorIdleTimeout when the > RDD cache timeout is reached. I was able to confirm this by doing the > following: > 1. Update spark-defaults.conf to set the following: > executorIdleTimeout=30 > cachedExecutorIdleTimeout=20 > 2. Update log4j.properties to set the following: > shell.log.level=INFO > 3. Run the following in spark-shell: > scala> val textFile = sc.textFile("/user/spark/applicationHistory/app_1234") > scala> textFile.cache().count() > 4. After 30 secs you will see 2 timeout messages, but of which are 30 secs > (whereas one *should* be for 20 secs) -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org