[
https://issues.apache.org/jira/browse/HADOOP-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602078#action_12602078
]
Tom White commented on HADOOP-930:
----------------------------------
bq. From what I can tell, s3service.listObjects returns an array of S3Object,
where each instance already has any associated meta-data in a HashMap.
I don't think all the fields in S3Object are populated - just those returned in
the list keys response. See
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/ListingKeysResponse.html
I think Jets3t does validate MD5 checksums on reads - but I'll double check.
> Add support for reading regular (non-block-based) files from S3 in
> S3FileSystem
> -------------------------------------------------------------------------------
>
> Key: HADOOP-930
> URL: https://issues.apache.org/jira/browse/HADOOP-930
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Affects Versions: 0.10.1
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.18.0
>
> Attachments: hadoop-930-v2.patch, hadoop-930-v3.patch,
> hadoop-930-v4.patch, hadoop-930.patch, jets3t-0.6.0.jar
>
>
> People often have input data on S3 that they want to use for a Map Reduce job
> and the current S3FileSystem implementation cannot read it since it assumes a
> block-based format.
> We would add the following metadata to files written by S3FileSystem: an
> indication that it is block oriented ("S3FileSystem.type=block") and a
> filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files
> would not have the type metadata so S3FileSystem would not try to interpret
> them as inodes.
> An extension to write regular files to S3 would not be covered by this change
> - we could do this as a separate piece of work (we still need to decide
> whether to introduce another scheme - e.g. rename block-based S3 to "s3fs"
> and call regular S3 "s3" - or whether to just use a configuration property to
> control block-based vs. regular writes).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
