[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727: URL: https://github.com/apache/spark/pull/32727#issuecomment-853497914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727: URL: https://github.com/apache/spark/pull/32727#issuecomment-854189036 Hi, @attilapiros . I made a follow-up for your comment. - https://github.com/apache/spark/pull/32774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727: URL: https://github.com/apache/spark/pull/32727#issuecomment-853497914 I made a backport. - https://github.com/apache/spark/pull/32756 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727:
URL: https://github.com/apache/spark/pull/32727#issuecomment-852740276
Thank you, @holdenk !
Hi, @Ngone51 . It's a very common case. Try this.
```
scala> Seq((1,2)).toDF("a", "b").repartition(10).groupBy("a").count().show()
+---+-+
| a|count|
+---+-+
| 1|1|
+---+-+
$ ls -al blockmgr-b9389ef6-6328-4953-9d66-e6e2da21f65c/* | grep shuffle
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_9_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_10_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_7_0.index
-rw-r--r-- 1 dongjoon staff60 Jun 1 22:35 shuffle_1_1_0.data
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_5_0.index
-rw-r--r-- 1 dongjoon staff59 Jun 1 22:35 shuffle_0_0_0.data
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_3_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_2_0.index
-rw-r--r-- 1 dongjoon staff88 Jun 1 22:35 shuffle_0_0_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_1_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_4_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_6_0.index
-rw-r--r-- 1 dongjoon staff 1608 Jun 1 22:35 shuffle_1_8_0.index
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727: URL: https://github.com/apache/spark/pull/32727#issuecomment-852456128 For branch-3.1, I will make a new PR for backporting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727: URL: https://github.com/apache/spark/pull/32727#issuecomment-852454568 Thank you so much, @holdenk ! Merged to master for Apache Spark 3.2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32727: [SPARK-35589][CORE] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
dongjoon-hyun commented on pull request #32727: URL: https://github.com/apache/spark/pull/32727#issuecomment-851848696 cc @holdenk and @attilapiros -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
