[ https://issues.apache.org/jira/browse/SPARK-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hans van den Bogert updated SPARK-10295: ---------------------------------------- Description: When running spark in coarse grained mode with shuffle service and dynamic allocation, the driver does not release executors if a dataset is cached. The console output OTOH shows: > 15/08/26 17:29:58 WARN SparkContext: Dynamic allocation currently does not > support cached RDDs. Cached data for RDD 9 will be lost when executors are > removed. However after the default of 1m, executors are not released. When I perform the same initial setup, loading data, etc, but without caching, the executors are released. Is this intended behaviour? If this is intended behaviour, the console warning is misleading. was: When running spark in coarse grained mode with shuffle service and dynamic reservation, the driver does not release executors if a dataset is cached. The console output OTOH shows: > 15/08/26 17:29:58 WARN SparkContext: Dynamic allocation currently does not > support cached RDDs. Cached data for RDD 9 will be lost when executors are > removed. However after the default of 1m, executors are not released. When I perform the same initial setup, loading data, etc, but without caching, the executors are released. Is this intended behaviour? If this is intended behaviour, the console warning is misleading. > Dynamic reservation in Mesos does not release when RDDs are cached > ------------------------------------------------------------------ > > Key: SPARK-10295 > URL: https://issues.apache.org/jira/browse/SPARK-10295 > Project: Spark > Issue Type: Question > Affects Versions: 1.5.0 > Environment: Spark 1.5.0 RC1 > Centos 6 > java 7 oracle > Reporter: Hans van den Bogert > Priority: Minor > > When running spark in coarse grained mode with shuffle service and dynamic > allocation, the driver does not release executors if a dataset is cached. > The console output OTOH shows: > > 15/08/26 17:29:58 WARN SparkContext: Dynamic allocation currently does not > > support cached RDDs. Cached data for RDD 9 will be lost when executors are > > removed. > However after the default of 1m, executors are not released. When I perform > the same initial setup, loading data, etc, but without caching, the executors > are released. > Is this intended behaviour? > If this is intended behaviour, the console warning is misleading. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org