[
https://issues.apache.org/jira/browse/SPARK-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15652263#comment-15652263
]
Krish Dey commented on SPARK-12333:
-----------------------------------
The constructor still seems to be the same as it is. Doesn't this to be changed
to accommodate encryption of spill to disk? Moreover passing the
DummySerializerInstance it should be allowed to pass any Serializer
public UnsafeSorterSpillWriter(BlockManager blockManager, int fileBufferSize,
ShuffleWriteMetrics writeMetrics, int numRecordsToWrite) throws IOException{
final Tuple2<TempLocalBlockId, File> spilledFileInfo =
blockManager.diskBlockManager().createTempLocalBlock();
this.file = spilledFileInfo._2();
this.blockId = spilledFileInfo._1();
this.numRecordsToWrite = numRecordsToWrite;
// Unfortunately, we need a serializer instance in order to construct a
DiskBlockObjectWriter.
// Our write path doesn't actually use this serializer (since we end up
calling the `write()`
// OutputStream methods), but DiskBlockObjectWriter still calls some
methods on it. To work
// around this, we pass a dummy no-op serializer.
writer = blockManager.getDiskWriter(
blockId, file, DummySerializerInstance.INSTANCE, fileBufferSize,
writeMetrics);
// Write the number of records
writeIntToBuffer(numRecordsToWrite, 0);
writer.write(writeBuffer, 0, 4);
}
> Support shuffle spill encryption in Spark
> -----------------------------------------
>
> Key: SPARK-12333
> URL: https://issues.apache.org/jira/browse/SPARK-12333
> Project: Spark
> Issue Type: New Feature
> Components: Shuffle
> Reporter: Ferdinand Xu
>
> Like shuffle file encryption in SPARK-5682, spills data should also be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
