littlexyw commented on code in PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#discussion_r2224262416
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -526,6 +523,9 @@ private ByteBuf getNextChunk() throws IOException {
throw new CelebornIOException(
"Fetch data from excluded worker! " +
currentReader.getLocation());
}
+ if (!currentReader.hasNext()) {
Review Comment:
> I think it is not a good idea to just do this fix. Which version are you
using? If you are using 0.5, this situation may be caused by inconsistent data
between the primary and replica partition. It is likely that hard splits has
occurred when pushing data. The fix submitted here can really solve the hang
task problem, but it may cause missing data, which is more dangerous. You can
try to turn off either the push replicate or
`celeborn.client.shuffle.batchHandleCommitPartition.enabled` first, which may
solve your problem. Further solutions may require code modifications.
Internally, we are trying to do some optimizations such as checking commit
result and so on.
Under which case will it cause missing data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
