Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/18317#discussion_r138189593
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -72,10 +72,15 @@ public UnsafeSorterSpillReader(
bufferSizeBytes = DEFAULT_BUFFER_SIZE_BYTES;
}
+ final Double readAheadFraction =
+ SparkEnv.get() == null ? 0.5 :
+
SparkEnv.get().conf().getDouble("spark.unsafe.sorter.spill.read.ahead.fraction",
0.5);
+
final InputStream bs =
new NioBufferedFileInputStream(file, (int) bufferSizeBytes);
try {
- this.in = serializerManager.wrapStream(blockId, bs);
+ this.in = new
ReadAheadInputStream(serializerManager.wrapStream(blockId, bs),
--- End diff --
Could you add an internal conf to disable it? It will allow the user to
disable it when the new feature causes a regression.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
