Re: Writing to s3 using Drill

Hi Nitin, The "tmp" object exists inside the s3 bucket. Even though, it throws the same error: Error: SYSTEM ERROR: IllegalArgumentException: URI has an authority component Fragment 0:0 On Fri, May 26, 2017 at 5:14 PM, Nitin Pawar wrote: > you have

Re: Writing to s3 using Drill

you have to create a tmp object in your bucket to make it work. s3://bucket_name/tmp has to be created and then it should work On Fri, May 26, 2017 at 5:02 PM, Shuporno Choudhury < shuporno.choudh...@manthan.com> wrote: > Hi Nitin, > > Thanks for the config settings. > > Now, after entering

Re: Writing to s3 using Drill

Hi Nitin, Thanks for the config settings. Now, after entering those config settings 1. s3.tmp does appear in the "show schemas" result 2. Also, it doesn't disappear when I add a custom folder in the location attribute But when I try to run a CTAS statement, I get the following error:

Re: Writing to s3 using Drill

Can you try with following s3 config { "type": "file", "enabled": true, "connection": "s3a://bucket_name", "config": { "fs.s3a.connection.maximum": "1", "fs.s3a.access.key": "access_key", "fs.s3a.secret.key": "secret_key", "fs.s3a.buffer.dir": "/tmp",

Re: Writing to s3 using Drill

May 25, 2017 12:11:40 AM To: user@drill.apache.org Subject: Re: Writing to s3 using Drill My s3 plugin info is as follows: { "type": "file", "enabled": true, "connection": "s3a://abcd", "config": { "fs.s3a.access.key&quo

Re: Writing to s3 using Drill

Hi, Can someone at Drill help me with issue please? On Thu, May 25, 2017 at 1:33 PM, Shuporno Choudhury < shuporno.choudh...@manthan.com> wrote: > HI, > > I corrected the "show schemas" output by putting only "/" in the > "location" . Now it shows s3.tmp in the output. > > But, it has a weird

Re: Writing to s3 using Drill

My s3 plugin info is as follows: { "type": "file", "enabled": true, "connection": "s3a://abcd", "config": { "fs.s3a.access.key": "abcd", "fs.s3a.secret.key": "abcd" }, "workspaces": { "root": { "location": "/", "writable": false, "defaultInputFormat":

Re: Writing to s3 using Drill

HI, I corrected the "show schemas" output by putting only "/" in the "location" . Now it shows s3.tmp in the output. But, it has a weird problem. The moment I add a folder to the location, eg: "/myfolder", then s3.tmp vanishes from the "show schemas" output. Also, when I try to write into s3,

Re: Writing to s3 using Drill

Sorry, I was wrong - please ignore my previous message. Looks like we do support writing to S3, but there were small differences necessary to make this work: First, I had to prefix the CTAS table name with the S3 plugin name. And second, I had to either update the s3 storage plugin configuration

Re: Writing to s3 using Drill

I don't think we support writing to Object stores such as S3. We do support reading from S3 buckets via the S3a library. However, we have limited support with the plugin. You could file a enhancement request on JIRA [1]. If someone has any experience with it, they can share details on the JIRA,

Re: Writing to s3 using Drill

Hi Shuporno, Can you please share your S3 plugin configuration ? Looks like in your configuration you might be missing something like below: "tmp": { "location": "drill-tmp", "writable": true, "defaultInputFormat": null } Thanks, Sorabh

Re: Writing to s3 using Drill

Hi Shuporno, Could you please specify the configuration from S3 storage plugin here and the output of `show schemas` as it pertains to s3? Is `writable` set to true? Gautam From: Shuporno Choudhury Sent: Monday, May 22, 2017