I was able to connect to the endpoint by setting the property 'fs.s3a.endpoint' to the appropriate url 'https://storage.xxx.com:8181'
I am now able to query the data in the bucket. However, as soon as I enable the S3 plugin - the response from Drill becomes extremely slow. This is true even if I am not querying the S3 bucket. As an example, just issuing a 'use' command takes forever: with the S3 plugin disabled: 0: jdbc:drill:zk=local> use cp; +-------+---------------------------------+ | ok | summary | +-------+---------------------------------+ | true | Default schema changed to [cp] | +-------+---------------------------------+ 1 row selected (0.543 seconds) with the S3 plugin enabled: 0: jdbc:drill:zk=local> use cp; +-------+---------------------------------+ | ok | summary | +-------+---------------------------------+ | true | Default schema changed to [cp] | +-------+---------------------------------+ 1 row selected (221.293 seconds) The S3 bucket configured in the plugin has approximately 20,000 objects. My assumption is that there is some sort of metadata scan that occurs anytime a command is executed? Any suggestions on how to improve performance? Thanks On Wed, May 24, 2017 at 3:14 PM, Abhishek Girish <[email protected]> wrote: > I'm not sure if anyone has ever tried that. Connecting to S3 buckets (AWS) > works via the S3a library. You could file a enhancement request on JIRA > [1]. > > If someone has any experience with it, they can share details on the JIRA, > or work on it. You are welcome to contribute yourself. > > [1] https://issues.apache.org/jira/browse/DRILL > > On Wed, May 24, 2017 at 12:01 PM, Raz Baluchi <[email protected]> > wrote: > > > Where would I specify to use SSL since the endpoint is https? > > > > On Wed, May 24, 2017 at 1:13 PM, Gautam Parai <[email protected]> wrote: > > > > > Hi Raz, > > > > > > > > > Please see here for an example https://drill.apache.org/docs/ > > > s3-storage-plugin/ > > > > > > Gautam > > > > > > > > > ________________________________ > > > From: [email protected] <[email protected]> on behalf of Raz > > > Baluchi <[email protected]> > > > Sent: Wednesday, May 24, 2017 7:03:12 AM > > > To: [email protected] > > > Subject: S3 configuration for ceph or atmos > > > > > > Is there a guide for configuring the S3 storage plugin for non AWS S3 > > > storage? > > > > > > As and example, we have Ceph storage that is accessible via the S3 API > at > > > an endpoint like: "https://storage.xxx.com:8181"; and bucket:"xyz" > > > > > > How would I go about configuring the S3 storage plugin? > > > > > > Thanks > > > > > >
