attilapiros commented on a change in pull request #24499: [SPARK-27677][Core]
Serve local disk persisted blocks by the external service after releasing
executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r285761100
##########
File path:
common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/TestShuffleDataContext.java
##########
@@ -97,13 +96,34 @@ public void insertSortShuffleData(int shuffleId, int
mapId, byte[][] blocks) thr
/** Creates spill file(s) within the local dirs. */
public void insertSpillData() throws IOException {
- String filename = "temp_local_" + UUID.randomUUID();
- OutputStream dataStream = null;
+ String filename = "temp_local_uuid";
Review comment:
Yes, but it is called by once for each data context.
To improve it:
1) an assert can be added or
2) a counter can be introduced which increased after each call and at the
checks this counter can considered when the expected values are constructed
I think the assert should be enough here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
