[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-11 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1067643931 ## docs/monitoring.md: ## @@ -1421,6 +1421,21 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Note:

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-11 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1067536500 ## docs/monitoring.md: ## @@ -1421,6 +1421,21 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Note:

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-11 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1067534717 ## docs/monitoring.md: ## @@ -1421,6 +1421,21 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Note:

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-10 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1066466470 ## docs/monitoring.md: ## @@ -1421,6 +1421,20 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +- Notes:

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-10 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1066466341 ## docs/monitoring.md: ## @@ -1421,6 +1421,20 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +- Notes:

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-10 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1066463478 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java: ## @@ -257,6 +274,7 @@ public void

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-10 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1066281891 ## docs/monitoring.md: ## @@ -1403,6 +1403,15 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Below

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-10 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1066281891 ## docs/monitoring.md: ## @@ -1403,6 +1403,15 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Below

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-06 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1063902183 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java: ## @@ -257,6 +274,7 @@ public void

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-03 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1060968735 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java: ## @@ -257,6 +274,7 @@ public void

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-03 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1060964263 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1904,4 +1951,52 @@ long getPos() { return pos;

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2023-01-03 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1060951175 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1220,6 +1260,7 @@ public void onData(String streamId,

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-12-22 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1055805408 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1904,4 +1941,42 @@ long getPos() { return pos;

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-12-22 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1055806229 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1197,15 +1230,15 @@ public void onData(String streamId,

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-12-22 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1055805408 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1904,4 +1941,42 @@ long getPos() { return pos;

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-12-22 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1055804525 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -593,6 +607,9 @@ public void onData(String streamId,

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-12-12 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1046602772 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1197,15 +1230,15 @@ public void onData(String streamId,

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-12-12 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1046495068 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1197,15 +1230,15 @@ public void onData(String streamId,

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-11-29 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1034531438 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java: ## @@ -1415,6 +1440,24 @@ private void pushBlockHelper(

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-10-18 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r997761687 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1904,4 +1941,42 @@ long getPos() { return pos;

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-10-18 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r997733472 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1124,9 +1143,23 @@ private boolean isDuplicateBlock() {

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-10-18 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r997751290 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -593,6 +607,9 @@ public void onData(String streamId,

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-10-17 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r997736011 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1124,9 +1143,23 @@ private boolean isDuplicateBlock() {

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-10-17 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r997733472 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1124,9 +1143,23 @@ private boolean isDuplicateBlock() {

[GitHub] [spark] rmcyang commented on a diff in pull request #37638: [SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

2022-08-31 Thread GitBox
rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r959829122 ## common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java: ## @@ -296,10 +301,15 @@ protected void serviceInit(Configuration