On 9 November 2011 20:59, Daniel Dai <da...@hortonworks.com> wrote: > How about "hadoop fs -ls"? Pig rely on hadoop for s3 access. If hadoop does > not support this, the fix should be on hadoop side.
Ah, thanks. I've shut down the instance for now, will check next time it's reactivated. However from a quick look elsewhere I find https://issues.apache.org/jira/browse/HADOOP-6146 "The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets." ...and that was integrated into Hadoop back in July 2009. http://www.jets3t.org/toolkit/configuration.html has "Requester Pays Settings": httpclient.requester-pays-buckets-enabled When set to true, JetS3t will be able to access Requester Pays buckets and the library user will be liable for any subsequent S3 request and bandwidth fees. "The main configuration properties for the JetS3t toolkit and applications are stored in a file called jets3t.properties. This file specifies advanced communication properties. A properties file with default settings is included in the configs directory JetS3t release." It looks from searching around as if jets3t.properties should probably live in Hadoop's conf directory. Investigating... cheers, Dan
