one more thing, how Impala access s3? use the DFSClient in HDFS?Have you added code for accessing s3? Or you just simply add a hadoop-aws jar? Is there book about the code design of Impala? For example, what the fe nodule mean and for?And what the be module mean?
在 2016年12月15日星期四 UTC+8下午2:15:45，黄志写道： > > Hi, I see the fe module uses hadoop-aws.jar to support S3, and I see the > wrapper class in it, can I use the same way to use Alluxio? and is there > any need to add code in Impala? > > 在 2016年12月15日星期四 UTC+8下午1:07:02，黄志写道： >> >> I see it!Thanks,I want to know how Impala uses S3, I did't see the code >> in Impala about choosing s3 and invoke s3's API >> >> 在 2016年12月14日星期三 UTC+8下午11:47:33，Tim Armstrong写道： >>> >>> See https://wiki.apache.org/hadoop/AmazonS3 and S3AFilesystem.java in >>> the Hadoop source tree. >>> >>> It looks like Alluxio also has a similar filesystem implementation. I >>> don't know anything about Alluxio or how well it works in practce, just >>> giving pointers in case you are interested in developing the support. >>> >>> On Tue, Dec 13, 2016 at 10:01 PM, 黄志 <[email protected]> wrote: >>> >>>> Hi, I am interested in this, I saw Impala use boto3 to implement >>>> S3Client, but Alluxio only offer Java API,how do we use it?and where is >>>> S3AFileSystem? >>>> >>>> 在 2016年12月9日星期五 UTC+8上午1:34:42，Tim Armstrong写道： >>>>> >>>>> It looks like Alluxio provides some kind of Hadoop Filesystem API >>>>> wrapper: >>>>> http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api. >>>>> I don't know if it's been upstreamed into Apache Hadoop. >>>>> >>>>> We do use similar Hadoop Filesystem API adapters to provide S3 and >>>>> Isilon support without massive code changes so it seems likely that it's >>>>> possible to do the same for Alluxio. Our experience with S3 and Isilon >>>>> was >>>>> that it required additional work to get it to work and perform well >>>>> because >>>>> different filesystems have different quirks even if they implement the >>>>> same >>>>> interface. >>>>> >>>>> It would require at least some small code changes to switch it on >>>>> (e.g. you can grep for S3AFilesystem to see how we turned on S3 support). >>>>> Please let us know if you experiment with it and have any success (or no >>>>> success). >>>>> >>>>> - Tim >>>>> >>>>> On Thu, Dec 8, 2016 at 9:07 AM, Jim Apple <[email protected]> wrote: >>>>> >>>>>> I don't know that Impala has been tested with Alluxio. >>>>>> >>>>>> This mailing list is deprecated; please use >>>>>> [email protected] >>>>>> >>>>>> >>>>>> On Wednesday, December 7, 2016 at 11:08:59 PM UTC-8, >>>>>> [email protected] wrote: >>>>>>> >>>>>>> Hi >>>>>>> How are you configure the impala on the alluxio; >>>>>>> >>>>>>> Alluxio: Open Source Memory Speed Virtual Distributed Storage >>>>>>> Alluxio: http://alluxio.org/ >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Impala Dev" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Impala Dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> >>>
