[GitHub] [spark] xuanyuanking commented on a change in pull request #28941: [SPARK-32124][CORE][SHS] Fix taskEndReasonFromJson to handle event logs from old Spark versions
xuanyuanking commented on a change in pull request #28941:
URL: https://github.com/apache/spark/pull/28941#discussion_r448192444
##
File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
##
@@ -1078,7 +1078,10 @@ private[spark] object JsonProtocol {
val blockManagerAddress = blockManagerIdFromJson(json \ "Block Manager
Address")
val shuffleId = (json \ "Shuffle ID").extract[Int]
val mapId = (json \ "Map ID").extract[Long]
-val mapIndex = (json \ "Map Index").extract[Int]
+val mapIndex = (json \ "Map Index") match {
+ case JNothing => 0
Review comment:
Make sense. It deserves a follow-up, let me submit one soon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28941: [SPARK-32124][CORE][SHS] Fix taskEndReasonFromJson to handle event logs from old Spark versions
xuanyuanking commented on a change in pull request #28941:
URL: https://github.com/apache/spark/pull/28941#discussion_r447481083
##
File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
##
@@ -1078,7 +1078,10 @@ private[spark] object JsonProtocol {
val blockManagerAddress = blockManagerIdFromJson(json \ "Block Manager
Address")
val shuffleId = (json \ "Shuffle ID").extract[Int]
val mapId = (json \ "Map ID").extract[Long]
-val mapIndex = (json \ "Map Index").extract[Int]
+val mapIndex = (json \ "Map Index") match {
+ case JNothing => 0
Review comment:
nit: 0 is a legal value for the map index here. Since it's for backward
compatibility, an illegal value here should make more sense. Also, it's better
to comment here for emphasizing it's for backward compatibility, otherwise the
fetch failed event will be dropped when SHS loading old version logs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
