Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/19788#discussion_r158124202
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -203,22 +203,23 @@ private ShuffleMetrics() {
this.appId = appId;
this.execId = execId;
String[] blockId0Parts = blockIds[0].split("_");
- if (blockId0Parts.length != 4 ||
!blockId0Parts[0].equals("shuffle")) {
+ if (blockId0Parts.length != 5 ||
!blockId0Parts[0].equals("shuffle")) {
--- End diff --
This format change can cause incompatibility between shuffle service and
spark application - causing a restart of the cluster and update of all spark
applications .... I wish we had a better way to encode this information which
was not so brittle.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
