[jira] [Updated] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunling Cai updated SPARK-25091: Component/s: (was: Spark Core) SQL > Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor > memory > > > Key: SPARK-25091 > URL: https://issues.apache.org/jira/browse/SPARK-25091 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.1 >Reporter: Yunling Cai >Priority: Major > > UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. > Through Spark UI, although in Storage, we see the cached table removed. In > Executor, the executors continue to hold the RDD and the memory is not > cleared. This results in huge waste in executor memory usage. As we call > CACHE TABLE, we run into issues where the cached tables are spilled to disk > instead of reclaiming the memory storage. > Steps to reproduce: > CACHE TABLE test.test_cache; > UNCACHE TABLE test.test_cache; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > CACHE TABLE test.test_cache; > CLEAR CACHE; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > Similar behavior when using pyspark df.unpersist(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunling Cai updated SPARK-25091: Priority: Critical (was: Major) > Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor > memory > > > Key: SPARK-25091 > URL: https://issues.apache.org/jira/browse/SPARK-25091 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.1 >Reporter: Yunling Cai >Priority: Critical > > UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. > Through Spark UI, although in Storage, we see the cached table removed. In > Executor, the executors continue to hold the RDD and the memory is not > cleared. This results in huge waste in executor memory usage. As we call > CACHE TABLE, we run into issues where the cached tables are spilled to disk > instead of reclaiming the memory storage. > Steps to reproduce: > CACHE TABLE test.test_cache; > UNCACHE TABLE test.test_cache; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > CACHE TABLE test.test_cache; > CLEAR CACHE; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > Similar behavior when using pyspark df.unpersist(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
Yunling Cai created SPARK-25091: --- Summary: Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory Key: SPARK-25091 URL: https://issues.apache.org/jira/browse/SPARK-25091 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.3.1 Reporter: Yunling Cai UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors continue to hold the RDD and the memory is not cleared. This results in huge waste in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled to disk instead of reclaiming the memory storage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunling Cai updated SPARK-25091: Description: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors continue to hold the RDD and the memory is not cleared. This results in huge waste in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled to disk instead of reclaiming the memory storage. Steps to reproduce: CACHE TABLE test.test_cache; UNCACHE TABLE test.test_cache; == Storage shows table is not cached; Executor shows the executor storage memory does not change == CACHE TABLE test.test_cache; CLEAR CACHE; == Storage shows table is not cached; Executor shows the executor storage memory does not change == was: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors continue to hold the RDD and the memory is not cleared. This results in huge waste in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled to disk instead of reclaiming the memory storage. > Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor > memory > > > Key: SPARK-25091 > URL: https://issues.apache.org/jira/browse/SPARK-25091 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.1 >Reporter: Yunling Cai >Priority: Major > > UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. > Through Spark UI, although in Storage, we see the cached table removed. In > Executor, the executors continue to hold the RDD and the memory is not > cleared. This results in huge waste in executor memory usage. As we call > CACHE TABLE, we run into issues where the cached tables are spilled to disk > instead of reclaiming the memory storage. > Steps to reproduce: > CACHE TABLE test.test_cache; > UNCACHE TABLE test.test_cache; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > CACHE TABLE test.test_cache; > CLEAR CACHE; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunling Cai updated SPARK-25091: Description: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors continue to hold the RDD and the memory is not cleared. This results in huge waste in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled to disk instead of reclaiming the memory storage. Steps to reproduce: CACHE TABLE test.test_cache; UNCACHE TABLE test.test_cache; == Storage shows table is not cached; Executor shows the executor storage memory does not change == CACHE TABLE test.test_cache; CLEAR CACHE; == Storage shows table is not cached; Executor shows the executor storage memory does not change == Similar behavior when using pyspark df.unpersist(). was: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors continue to hold the RDD and the memory is not cleared. This results in huge waste in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled to disk instead of reclaiming the memory storage. Steps to reproduce: CACHE TABLE test.test_cache; UNCACHE TABLE test.test_cache; == Storage shows table is not cached; Executor shows the executor storage memory does not change == CACHE TABLE test.test_cache; CLEAR CACHE; == Storage shows table is not cached; Executor shows the executor storage memory does not change == > Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor > memory > > > Key: SPARK-25091 > URL: https://issues.apache.org/jira/browse/SPARK-25091 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.1 >Reporter: Yunling Cai >Priority: Major > > UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. > Through Spark UI, although in Storage, we see the cached table removed. In > Executor, the executors continue to hold the RDD and the memory is not > cleared. This results in huge waste in executor memory usage. As we call > CACHE TABLE, we run into issues where the cached tables are spilled to disk > instead of reclaiming the memory storage. > Steps to reproduce: > CACHE TABLE test.test_cache; > UNCACHE TABLE test.test_cache; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > CACHE TABLE test.test_cache; > CLEAR CACHE; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > Similar behavior when using pyspark df.unpersist(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunling Cai updated SPARK-25091: Summary: UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory (was: Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory) > UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory > - > > Key: SPARK-25091 > URL: https://issues.apache.org/jira/browse/SPARK-25091 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.1 >Reporter: Yunling Cai >Priority: Critical > > UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. > Through Spark UI, although in Storage, we see the cached table removed. In > Executor, the executors continue to hold the RDD and the memory is not > cleared. This results in huge waste in executor memory usage. As we call > CACHE TABLE, we run into issues where the cached tables are spilled to disk > instead of reclaiming the memory storage. > Steps to reproduce: > CACHE TABLE test.test_cache; > UNCACHE TABLE test.test_cache; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > CACHE TABLE test.test_cache; > CLEAR CACHE; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > Similar behavior when using pyspark df.unpersist(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593919#comment-16593919 ] Yunling Cai commented on SPARK-25091: - Thanks [~Chao Fang] for working on this! I have changed the ticket title. Quick question: does this mean this is just a UI issue where executor information was shown incorrectly? As we saw the cache tables start falling onto disk even though we have uncache the previous copy of it. We also started seeing duplicate entries on the storage tab for same table and this is why we think the memory clean up may have actual problems. Steps to reproduce: CACHE TABLE A UNCACHE TABLE A CACHE TABLE A REFRESH TABLE has a similar behavior. Thanks! > UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory > - > > Key: SPARK-25091 > URL: https://issues.apache.org/jira/browse/SPARK-25091 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.1 >Reporter: Yunling Cai >Priority: Critical > > UNCACHE TABLE and CLEAR CACHE does not clean up executor memory. > Through Spark UI, although in Storage, we see the cached table removed. In > Executor, the executors continue to hold the RDD and the memory is not > cleared. This results in huge waste in executor memory usage. As we call > CACHE TABLE, we run into issues where the cached tables are spilled to disk > instead of reclaiming the memory storage. > Steps to reproduce: > CACHE TABLE test.test_cache; > UNCACHE TABLE test.test_cache; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > CACHE TABLE test.test_cache; > CLEAR CACHE; > == Storage shows table is not cached; Executor shows the executor storage > memory does not change == > Similar behavior when using pyspark df.unpersist(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
