[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1063026190 ## core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala: ## @@ -496,7 +555,52 @@ class StageDataWrapperSerializer extends

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1061723365 ## core/src/main/scala/org/apache/spark/status/storeTypes.scala: ## @@ -233,6 +243,38 @@ private[spark] class TaskDataWrapper( val shuffleLocalBytesRead: Long,

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1060720199 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -100,11 +100,21 @@ message TaskDataWrapper { int64

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1058019046 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -100,11 +100,21 @@ message TaskDataWrapper { int64

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1058019046 ## core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto: ## @@ -100,11 +100,21 @@ message TaskDataWrapper { int64

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1054375191 ## core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala: ## @@ -1653,7 +1663,19 @@ private[spark] object JsonProtocolSuite extends Assertions { |

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1046140712 ## core/src/test/resources/HistoryServerExpectations/failed_stage_list_json_expectation.json: ## @@ -27,13 +27,23 @@ "outputBytes" : 0, "outputRecords" : 0,

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1046139711 ## core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala: ## @@ -726,6 +736,61 @@ final class ShuffleBlockFetcherIterator( } } +

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1046138205 ## core/src/main/scala/org/apache/spark/status/api/v1/api.scala: ## @@ -302,7 +312,9 @@ class StageData private[spark]( @JsonDeserialize(using =

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1042043726 ## core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json: ## @@ -81,7 +93,19 @@ "remoteBytesRead" : 0,

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1040884126 ## core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json: ## @@ -81,7 +93,19 @@ "remoteBytesRead" : 0,

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1040478699 ## core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json: ## @@ -81,7 +93,19 @@ "remoteBytesRead" : 0,

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1039691827 ## core/src/main/scala/org/apache/spark/executor/Executor.scala: ## @@ -791,6 +770,53 @@ private[spark] class Executor( } } +private def

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1035142645 ## core/src/main/scala/org/apache/spark/InternalAccumulator.scala: ## @@ -54,6 +55,16 @@ private[spark] object InternalAccumulator { val LOCAL_BYTES_READ =

[GitHub] [spark] thejdeep commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

thejdeep commented on code in PR #36165: URL: https://github.com/apache/spark/pull/36165#discussion_r1035141286 ## core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala: ## @@ -866,6 +930,7 @@ final class ShuffleBlockFetcherIterator( }