RexXiong commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1639955635
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition
$partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
Why we need a creation window for this?
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -187,7 +189,8 @@ private static final class CelebornInputStreamImpl extends
CelebornInputStream {
int shuffleId,
int partitionId,
ExceptionMaker exceptionMaker,
- MetricsCallback metricsCallback)
+ MetricsCallback metricsCallback,
+ boolean chunkPrefetchEnabled)
Review Comment:
We can directly use conf.clientChunkPrefetchEnabled to get the value,
instead of passing it from elsewhere and changing ShuffleClient/Impl method
signature.
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4864,6 +4866,23 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val CLIENT_CHUNK_PREFETCH_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.client.chunk.prefetch.enabled")
Review Comment:
Only for spark? celeborn.client -> celeborn.client.spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
