Improvements for NativeS3FileSystem
-----------------------------------
Key: HADOOP-3912
URL: https://issues.apache.org/jira/browse/HADOOP-3912
Project: Hadoop Core
Issue Type: Improvement
Components: fs/s3
Reporter: Albert Chern
In the process of porting NativeS3FileSystem for use with Hadoop 12, I made the
following changes and improvements which might be helpful (apologies if I
should have opened separate issues, but I was lazy):
1. The single byte read() method of NativeS3InputStream incorrectly treats the
return value of InputStream.read() as the number of bytes read, which is
actually always 1.
2. It allows people to write files ending with the folder suffix. I prevented
this by doing a check in the create() method.
3. Similarly, it allows people to open directories for reading. I prevented
this by doing a check in the open() method.
4. If you write a file to a nonexistent directory tree, say /a/b/c/d/file, and
then you delete/rename that file or one of its parent directories, the whole
directory tree vanishes. I fixed this by always creating the parent of a
deleted/renamed file.
5. Recursive delete(), rename(), and getContentLength() can be sped up
tremendously by working directly with S3 listings rather than working at the
FileSystem level. All sub-files/sub-directories should begin with the parent
directory name as a prefix.
6. HADOOP-3506 is still relevant.
I don't have patches since I created a new file, but I have attached my source
if Tom wants to take a look at it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
