tillrohrmann commented on issue #8242: [FLINK-6227][network] Introduce the DataConsumptionException for downstream task failure URL: https://github.com/apache/flink/pull/8242#issuecomment-489994956 > I mean the disk/network hardware problems on producer side which could not be restored in short time. So it is better to restart the producer in another machine. We ever encountered this corner case in production. I guess that this problem won't be solely solved by restarting the producer. Imagine that the restarted producer will be placed on the same machine again. Then it will run into the same problem eventually. Additionally, one would need to blacklist this machine so that these resources are no longer used. Since this is a bigger issue, I would like to keep it out of the scope of this PR and address it with a follow up. > d. BlockingResultSubpartitionView#getNextBuffer thrown IOException: the reason is the same as above c. PartitionNotFound might also be used here. I'm not 100% sure that this should throw an `PartitionNotFound` exception. The reason for the `IOException` could maybe be something transient as well (e.g. if you read from a NFS). I would propose that this should be a `DataTransferException` or something similar. If the file were removed, then the next lookup (due to the task restart) of the subpartition should fail with a `PartitionNotFound` exception which would then trigger the restart of the producer. In general I would like to distinguish `DataTransferException`, which signals a problem after a connection has been established and after the partition has been found, from `PartitionNotFoundException` which signals that a partition could not be found. I think we are missing another case: * f. The TM is not reachable because it failed. Hence it is not possible to open a connection to it. After the retry time, this should also fail with a `PartitionNotFoundException` or a `ConnectionTimeoutException` which would be treated like a `PartitionNotFoundException`. I would propose to start with covering the cases `{b, c, f}` and address the other cases as a follow up because that way we keep the scope narrow without preventing us from covering the other cases later.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
