[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2023-01-05 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1063009392 ## core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala: ## @@ -496,7 +555,52 @@ class StageDataWrapperSerializer extends

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2023-01-05 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1062884659 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -652,6 +699,20 @@ message ShuffleReadMetricDistributions { repeated double

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2023-01-03 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1061138371 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -652,6 +699,20 @@ message ShuffleReadMetricDistributions { repeated double

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-28 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1058701274 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -100,11 +100,21 @@ message TaskDataWrapper { int64

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-28 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1058701274 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -100,11 +100,21 @@ message TaskDataWrapper { int64

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-27 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1057888476 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -100,11 +100,21 @@ message TaskDataWrapper { int64

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-09 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1044823829 ## core/src/main/scala/org/apache/spark/status/api/v1/api.scala: ## @@ -302,7 +312,9 @@ class StageData private[spark]( @JsonDeserialize(using =

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-06 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1041749200 ## core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json: ## @@ -81,7 +93,19 @@ "remoteBytesRead" : 0,

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-05 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1040453682 ## core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json: ## @@ -81,7 +93,19 @@ "remoteBytesRead" : 0,

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-03 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1038913697 ## core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala: ## @@ -1219,7 +1219,12 @@ class ShuffleBlockFetcherIteratorSuite extends

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-03 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1038913666 ## core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala: ## @@ -1780,7 +1802,19 @@ private[spark] object JsonProtocolSuite extends Assertions { |

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-12-03 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1038913313 ## core/src/main/scala/org/apache/spark/status/AppStatusStore.scala: ## @@ -319,7 +319,18 @@ private[spark] class AppStatusStore(

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-11-28 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1034238236 ## core/src/main/scala/org/apache/spark/executor/Executor.scala: ## @@ -791,6 +770,53 @@ private[spark] class Executor( } } +private def

[GitHub] [spark] mridulm commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

2022-11-28 Thread GitBox
mridulm commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1034209167 ## core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala: ## @@ -722,6 +718,63 @@ final class ShuffleBlockFetcherIterator( } } + //