Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-30 Thread via GitHub
bozhang2820 commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1584732080 ## sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala: ## @@ -161,6 +165,24 @@ object SQLExecution extends Logging { case e =>

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-30 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1584496133 ## sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala: ## @@ -161,6 +165,24 @@ object SQLExecution extends Logging { case e =>

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-30 Thread via GitHub
bozhang2820 commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1584347563 ## sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala: ## @@ -161,6 +165,24 @@ object SQLExecution extends Logging { case e =>

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-30 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1584298803 ## sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala: ## @@ -161,6 +165,24 @@ object SQLExecution extends Logging { case e =>

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-30 Thread via GitHub
ulysses-you commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1584239353 ## sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala: ## @@ -161,6 +165,24 @@ object SQLExecution extends Logging { case e =>

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-30 Thread via GitHub
ulysses-you commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1584223064 ## sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala: ## @@ -161,6 +165,24 @@ object SQLExecution extends Logging { case e =>

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-24 Thread via GitHub
cloud-fan closed pull request #45930: [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode URL: https://github.com/apache/spark/pull/45930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-24 Thread via GitHub
cloud-fan commented on PR #45930: URL: https://github.com/apache/spark/pull/45930#issuecomment-2074349569 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-23 Thread via GitHub
bozhang2820 commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1577219567 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -869,6 +874,8 @@ case class AdaptiveExecutionContext(session:

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-23 Thread via GitHub
bozhang2820 commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1575960906 ## core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala: ## @@ -187,6 +187,7 @@ private[spark] class SortShuffleManager(conf: SparkConf)

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-23 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1576289113 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -869,6 +874,8 @@ case class AdaptiveExecutionContext(session:

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-23 Thread via GitHub
bozhang2820 commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1575960906 ## core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala: ## @@ -187,6 +187,7 @@ private[spark] class SortShuffleManager(conf: SparkConf)

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-23 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1575686724 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -869,6 +874,8 @@ case class AdaptiveExecutionContext(session:

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-23 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1575684967 ## core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala: ## @@ -76,13 +78,21 @@ private[spark] class IndexShuffleBlockResolver( override

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-17 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1569819026 ## core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala: ## @@ -187,6 +187,7 @@ private[spark] class SortShuffleManager(conf: SparkConf)

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-17 Thread via GitHub
bozhang2820 commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1568946086 ## core/src/main/scala/org/apache/spark/shuffle/MigratableResolver.scala: ## @@ -35,6 +35,11 @@ trait MigratableResolver { */ def getStoredShuffles():

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-15 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1565226301 ## core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala: ## @@ -76,13 +78,19 @@ private[spark] class IndexShuffleBlockResolver( override

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-15 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1565222770 ## sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala: ## @@ -95,10 +95,26 @@ private[sql] object Dataset { new Dataset[Row](qe,

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-15 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1565220695 ## sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala: ## @@ -108,7 +108,8 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-15 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1565218933 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -2854,6 +2854,23 @@ object SQLConf { .intConf

Re: [PR] [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode [spark]

2024-04-15 Thread via GitHub
cloud-fan commented on code in PR #45930: URL: https://github.com/apache/spark/pull/45930#discussion_r1565218492 ## core/src/main/scala/org/apache/spark/shuffle/MigratableResolver.scala: ## @@ -35,6 +35,11 @@ trait MigratableResolver { */ def getStoredShuffles():