[GitHub] [spark] xuanyuanking commented on a change in pull request #28941: [SPARK-32124][CORE][SHS] Fix taskEndReasonFromJson to handle event logs from old Spark versions
xuanyuanking commented on a change in pull request #28941: URL: https://github.com/apache/spark/pull/28941#discussion_r448192444 ## File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ## @@ -1078,7 +1078,10 @@ private[spark] object JsonProtocol { val blockManagerAddress = blockManagerIdFromJson(json \ "Block Manager Address") val shuffleId = (json \ "Shuffle ID").extract[Int] val mapId = (json \ "Map ID").extract[Long] -val mapIndex = (json \ "Map Index").extract[Int] +val mapIndex = (json \ "Map Index") match { + case JNothing => 0 Review comment: Make sense. It deserves a follow-up, let me submit one soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28941: [SPARK-32124][CORE][SHS] Fix taskEndReasonFromJson to handle event logs from old Spark versions
xuanyuanking commented on a change in pull request #28941: URL: https://github.com/apache/spark/pull/28941#discussion_r447481083 ## File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ## @@ -1078,7 +1078,10 @@ private[spark] object JsonProtocol { val blockManagerAddress = blockManagerIdFromJson(json \ "Block Manager Address") val shuffleId = (json \ "Shuffle ID").extract[Int] val mapId = (json \ "Map ID").extract[Long] -val mapIndex = (json \ "Map Index").extract[Int] +val mapIndex = (json \ "Map Index") match { + case JNothing => 0 Review comment: nit: 0 is a legal value for the map index here. Since it's for backward compatibility, an illegal value here should make more sense. Also, it's better to comment here for emphasizing it's for backward compatibility, otherwise the fetch failed event will be dropped when SHS loading old version logs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org