See https://wiki.apache.org/hadoop/AmazonS3 and S3AFilesystem.java in the Hadoop source tree.
It looks like Alluxio also has a similar filesystem implementation. I don't know anything about Alluxio or how well it works in practce, just giving pointers in case you are interested in developing the support. On Tue, Dec 13, 2016 at 10:01 PM, 黄志 <[email protected]> wrote: > Hi, I am interested in this, I saw Impala use boto3 to implement S3Client, > but Alluxio only offer Java API,how do we use it?and where is S3AFileSystem? > > 在 2016年12月9日星期五 UTC+8上午1:34:42，Tim Armstrong写道： >> >> It looks like Alluxio provides some kind of Hadoop Filesystem API >> wrapper: http://www.alluxio.org/docs/master/en/File-System- >> API.html#hadoop-api. I don't know if it's been upstreamed into Apache >> Hadoop. >> >> We do use similar Hadoop Filesystem API adapters to provide S3 and Isilon >> support without massive code changes so it seems likely that it's possible >> to do the same for Alluxio. Our experience with S3 and Isilon was that it >> required additional work to get it to work and perform well because >> different filesystems have different quirks even if they implement the same >> interface. >> >> It would require at least some small code changes to switch it on (e.g. >> you can grep for S3AFilesystem to see how we turned on S3 support). Please >> let us know if you experiment with it and have any success (or no success). >> >> - Tim >> >> On Thu, Dec 8, 2016 at 9:07 AM, Jim Apple <[email protected]> wrote: >> >>> I don't know that Impala has been tested with Alluxio. >>> >>> This mailing list is deprecated; please use >>> [email protected] >>> >>> >>> On Wednesday, December 7, 2016 at 11:08:59 PM UTC-8, >>> [email protected] wrote: >>>> >>>> Hi >>>> How are you configure the impala on the alluxio; >>>> >>>> Alluxio: Open Source Memory Speed Virtual Distributed Storage >>>> Alluxio: http://alluxio.org/ >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Impala Dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Impala Dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. >
