mridulm commented on code in PR #2349:
URL:
https://github.com/apache/incubator-celeborn/pull/2349#discussion_r1508875682
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -89,6 +92,23 @@ class CelebornShuffleReader[K, C](
}
}
+ val exceptionMaker = new ExceptionMaker() {
+ override def makeException(
+ appShuffleId: Int,
+ shuffleId: Int,
+ partitionId: Int,
+ e: Exception): Exception = {
+ new FetchFailedException(
+ null,
+ handle.shuffleId,
+ -1,
+ -1,
+ partitionId,
+ SparkUtils.FETCH_FAILURE_ERROR_MSG + shuffleId,
Review Comment:
Include both appShuffleId and shuffleId in message (if they are different)
to help debugging ?
##########
client/src/main/java/org/apache/celeborn/client/ShuffleClient.java:
##########
@@ -210,13 +211,34 @@ public abstract void mapPartitionMapperEnd(
* @return
* @throws IOException
*/
- public abstract CelebornInputStream readPartition(
+ public CelebornInputStream readPartition(
int shuffleId,
int partitionId,
int attemptNumber,
int startMapIndex,
int endMapIndex,
MetricsCallback metricsCallback)
+ throws IOException {
+ return readPartition(
+ shuffleId,
+ partitionId,
+ attemptNumber,
+ startMapIndex,
+ endMapIndex,
+ shuffleId,
+ metricsCallback,
+ null);
Review Comment:
Incorrect parameter order ?
```suggestion
return readPartition(
shuffleId,
shuffleId,
partitionId,
attemptNumber,
startMapIndex,
endMapIndex,
metricsCallback,
null);
```
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -89,6 +92,23 @@ class CelebornShuffleReader[K, C](
}
}
+ val exceptionMaker = new ExceptionMaker() {
+ override def makeException(
+ appShuffleId: Int,
+ shuffleId: Int,
+ partitionId: Int,
+ e: Exception): Exception = {
+ new FetchFailedException(
+ null,
+ handle.shuffleId,
Review Comment:
This should be spark's shuffle id ?
```suggestion
appShuffleId,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
