[
https://issues.apache.org/jira/browse/HADOOP-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597298#comment-17597298
]
Prabhu Joseph commented on HADOOP-17458:
----------------------------------------
[[email protected]] A flink job which reads data from S3 and intermittently
few tasks fails with below exception. Does this patch will fix the below issue.
Thanks.
{code}
Data read has a different length than the expected: dataLength=53427;
expectedLength=65536; includeSkipped=true; in.getClass()=class
com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0;
resetSinceLastMarked=false; markCount=0; resetCount=0
at
com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151)
at
com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:93)
at
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:84)
at
com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:99)
at
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:84)
at
org.apache.hadoop.fs.s3a.S3AInputStream.closeStream(S3AInputStream.java:529)
at
org.apache.hadoop.fs.s3a.S3AInputStream.close(S3AInputStream.java:490)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at
org.apache.flink.fs.s3hadoop.common.HadoopDataInputStream.close(HadoopDataInputStream.java:91)
at
org.apache.flink.api.common.io.FileInputFormat.close(FileInputFormat.java:913)
at
org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:219)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:750)
{code}
> S3A to treat "SdkClientException: Data read has a different length than the
> expected" as EOFException
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17458
> URL: https://issues.apache.org/jira/browse/HADOOP-17458
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Bogdan Stolojan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> A test run with network problems caught exceptions
> "com.amazonaws.SdkClientException: Data read has a different length than the
> expected:", which then escalated to failure.
> these should be recoverable if they are recognised as such.
> translateException could do this. Yes, it would have to look @ the text, but
> as {{signifiesConnectionBroken()}} already does that for "Failed to sanitize
> XML document destined for handler class", it'd just be adding a new text
> string to look for.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
