Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/9214#discussion_r43155130
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
---
@@ -155,10 +170,16 @@ public void write(Iterator<Product2<K, V>> records)
throws IOException {
writer.commitAndClose();
}
- partitionLengths =
- writePartitionedFile(shuffleBlockResolver.getDataFile(shuffleId,
mapId));
- shuffleBlockResolver.writeIndexFile(shuffleId, mapId,
partitionLengths);
+ final File tmpDataFile =
blockManager.diskBlockManager().createTempShuffleBlock()._2();
+
+ partitionLengths = writePartitionedFile(tmpDataFile);
--- End diff --
the type of tmp block used here doesn't matter, since
[`writePartitionedFile`](https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java#L182)
doesn't go through `blockManager.getDiskWriter`. However, I'm a bit confused
on this worked before. The [original partition files might be
compressed](https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java#L141),
and those bytes just get copied to the final data file, though data files seem
like they are [never
compressed](https://github.com/apache/spark/blob/360ed832f5213b805ac28cf1d2828be09480f2d6/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1165)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
