Shen, Neal created HDFS-13412: --------------------------------- Summary: filesystem will throw out-of-bounds access for webhdfs using DirectoryListing Key: HDFS-13412 URL: https://issues.apache.org/jira/browse/HDFS-13412 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 3.0.0 Reporter: Shen, Neal Fix For: 3.0.0
In a big directory, there are a lot of entries; so we use LISTSTATUS_BATCH over webhdfs; and if the first batch is returned and preparing to query with startAfter but the left entries are deleted from the directory. for this scenario,the hadoop client will throw out-of-bounds access at line 'return (T)entries.getEntries()[i++];' @Override @SuppressWarnings("unchecked") public T next() throws IOException { Preconditions.checkState(hasNext(), "No more items in iterator"); if (i == entries.getEntries().length) { fetchMore(); } return (T)entries.getEntries()[i++]; } here is the exception stack: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.fs.FileSystem$DirListingIterator.next(FileSystem.java:2120) at org.apache.hadoop.fs.FileSystem$DirListingIterator.next(FileSystem.java:2084) at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch$1.run(TestWebHdfsListBatch.java:149) at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch$1.run(TestWebHdfsListBatch.java:135) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch.testLargeDirectory(TestWebHdfsListBatch.java:135) at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch.main(TestWebHdfsListBatch.java:102) 2018-03-30 22:53:00,731 DEBUG util.ShutdownHookManager: ShutdownHookManger complete shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org