[
https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073363#comment-15073363
]
Matthew Paduano commented on HADOOP-10542:
------------------------------------------
This change seems to break some things. In particular, have a closer look at:
S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException
but instead IOException)
FileSystem.exists() (which no longer returns false but
instead raises IOException)
S3FileSystem.create() (which no longer succeeds but instead raises
IOException)
While the javadoc suggests that the API permits one to raise IOException, most
of the code I have
encountered while debugging a command like "hadoop distcp
hdfs://localhost:9000/test s3://xxx:[email protected]/"
seems to expect (1) retrieveINode() to return null and (2)
FileNotFoundException to be raised when a file is not
found (i.e. when get() fails!).
2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1449826461866_0005_m_000006_0 - exited : java.io.IOException: /test
doesn't exist
at
org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170)
at
org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source)
at
org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
changing the "raise IOE..." to "return null" fixes all of the above code sites
and
allows distcp to succeed.
> Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
> ---------------------------------------------------------------------------
>
> Key: HADOOP-10542
> URL: https://issues.apache.org/jira/browse/HADOOP-10542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Minor
> Fix For: 2.7.0
>
> Attachments: hadoop-10542-001.patch
>
>
> {code}
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> {code}
> get() may return null.
> The while loop dereferences in without null check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
