attilapiros commented on a change in pull request #24499: [SPARK-27677][Core]
Serve local disk persisted blocks by the external service after releasing
executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r285781962
##########
File path:
common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/TestShuffleDataContext.java
##########
@@ -97,13 +96,34 @@ public void insertSortShuffleData(int shuffleId, int
mapId, byte[][] blocks) thr
/** Creates spill file(s) within the local dirs. */
public void insertSpillData() throws IOException {
- String filename = "temp_local_" + UUID.randomUUID();
- OutputStream dataStream = null;
+ String filename = "temp_local_uuid";
Review comment:
In the old code the test filter was:
org.apache.spark.network.shuffle.NonShuffleFilesCleanupSuite#filter:
~~~java
private static FilenameFilter filter = new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
// Don't delete shuffle data or shuffle index files.
return !name.endsWith(".index") && !name.endsWith(".data");
}
};
~~~
And the tested code the filter in:
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver#deleteNonShuffleFiles
~~~java
private void deleteNonShuffleFiles(String[] dirs) {
FilenameFilter filter = new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
// Don't delete shuffle data or shuffle index files.
return !name.endsWith(".index") && !name.endsWith(".data");
}
};
~~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
