henrymai opened a new pull request, #41413:
URL: https://github.com/apache/spark/pull/41413
### What changes were proposed in this pull request?
Consolidating BlockId parsing and handling helps to cut down on errors
arising from parsing the BlockId and also eliminates the need to manually
synchronize the code across different places that parse the BlockIds.
Most of the offending code was in common/network-shuffle and it looks like
the reason was because BlockId was in core/ so it wasn't usable by
common/network-shuffle. So this patch moves BlockId.scala into
common/storage-common to make it accessible everywhere.
Validated that there are no further manual BlockId parsing by running the
following commands:
```
git grep -i 'split("_")' **.scala | grep -v "/test/"
core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala:
val index =
eventLogFileName.stripPrefix(EVENT_LOG_FILE_NAME_PREFIX).split("_")(0)
git grep -i 'split("_")' **.java | grep -v "/test/"
common/network-common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java:
String[] array = streamChunkId.split("_");
```
The only hits from these commands are not related to BlockId parsing.
### Why are the changes needed?
These changes are needed to cut down on bugs that arise from manual BlockId
parsing and manual synchronization of BlockId parsing code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```
mvn clean install
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
