[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-725868760 I think there is something different with how the `network-yarn` module is build. I see that there existed

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-725741508 I had added `YarnShuffleServiceSuite.java` to add unit tests related to creation of MergedShuffleFileManager Implementation. The tests are failing with ```

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-722694164 The tests failures are not related to this change. This is an automated message from the Apache Git Service. To

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-722502275 > Why would the error happen? Does it mean we calculate the wrong message length somewhere when allocating the buffer? In `RemoteBlockPushResolver` we re-use the

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-722108668 Have addressed all the review comments. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721567210 I ran an end-to-end test. One of the bugs was: ``` Caused by: java.lang.IndexOutOfBoundsException at java.nio.ByteBuffer.wrap(ByteBuffer.java:375) at

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721401673 I am resolving all the older comments which have been addressed. This is an automated message from the Apache

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-720702382 With the latest commit, all the review comments have been addressed. @Ngone51 @tgravescs @attilapiros @mridulm @Victsm @jiangxb1987 Please take another look at the changes.

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-717737324 I have updated the PR with these changes: - Added support for subdirs under merge directory. - Removed the yarn local directory path pattern. - Removed the

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-709714308 Fixed the lint errors in `RemoteBlockPushResolverSuite`. This is an automated message from the Apache Git

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-709587476 +CC @venkata91 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] otterc commented on pull request #30062: [Spark 32916][Shuffle] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-709570519 cc @Victsm @mridulm @tgravescs @jiangxb1987 @attilapiros @Ngone51 Please take a look. This is an automated