[ https://issues.apache.org/jira/browse/HADOOP-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597298#comment-17597298 ]
Prabhu Joseph commented on HADOOP-17458: ---------------------------------------- [~ste...@apache.org] A flink job which reads data from S3 and intermittently few tasks fails with below exception. Does this patch will fix the below issue. Thanks. {code} Data read has a different length than the expected: dataLength=53427; expectedLength=65536; includeSkipped=true; in.getClass()=class com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0 at com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151) at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:93) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:84) at com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:99) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:84) at org.apache.hadoop.fs.s3a.S3AInputStream.closeStream(S3AInputStream.java:529) at org.apache.hadoop.fs.s3a.S3AInputStream.close(S3AInputStream.java:490) at java.io.FilterInputStream.close(FilterInputStream.java:181) at org.apache.flink.fs.s3hadoop.common.HadoopDataInputStream.close(HadoopDataInputStream.java:91) at org.apache.flink.api.common.io.FileInputFormat.close(FileInputFormat.java:913) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:219) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.lang.Thread.run(Thread.java:750) {code} > S3A to treat "SdkClientException: Data read has a different length than the > expected" as EOFException > ----------------------------------------------------------------------------------------------------- > > Key: HADOOP-17458 > URL: https://issues.apache.org/jira/browse/HADOOP-17458 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Bogdan Stolojan > Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > A test run with network problems caught exceptions > "com.amazonaws.SdkClientException: Data read has a different length than the > expected:", which then escalated to failure. > these should be recoverable if they are recognised as such. > translateException could do this. Yes, it would have to look @ the text, but > as {{signifiesConnectionBroken()}} already does that for "Failed to sanitize > XML document destined for handler class", it'd just be adding a new text > string to look for. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org