Hi Nitin,
The "tmp" object exists inside the s3 bucket. Even though, it throws the
same error:
Error: SYSTEM ERROR: IllegalArgumentException: URI has an authority
component
Fragment 0:0
On Fri, May 26, 2017 at 5:14 PM, Nitin Pawar <[email protected]>
wrote:
> you have to create a tmp object in your bucket to make it work.
> s3://bucket_name/tmp has to be created and then it should work
>
> On Fri, May 26, 2017 at 5:02 PM, Shuporno Choudhury <
> [email protected]> wrote:
>
> > Hi Nitin,
> >
> > Thanks for the config settings.
> >
> > Now, after entering those config settings
> > 1. s3.tmp does appear in the "show schemas" result
> > 2. Also, it doesn't disappear when I add a custom folder in the
> > location attribute
> >
> > But when I try to run a CTAS statement, I get the following error:
> >
> > *Error: SYSTEM ERROR: IllegalArgumentException: URI has an authority
> > component*
> > *Fragment 0:0*
> >
> > Query that I am trying to run:
> > *create table s3.tmp.`abcd` as select 1 from (values(1));*
> >
> > However, this query runs when I use dfs.tmp instead of s3.tmp
> >
> > On Fri, May 26, 2017 at 12:44 PM, Nitin Pawar <[email protected]>
> > wrote:
> >
> > > Can you try with following s3 config
> > >
> > > {
> > > "type": "file",
> > > "enabled": true,
> > > "connection": "s3a://bucket_name",
> > > "config": {
> > >
> > > "fs.s3a.connection.maximum": "10000",
> > > "fs.s3a.access.key": "access_key",
> > > "fs.s3a.secret.key": "secret_key",
> > > "fs.s3a.buffer.dir": "/tmp",
> > > "fs.s3a.multipart.size": "10485760",
> > > "fs.s3a.multipart.threshold": "104857600"
> > > },
> > > "workspaces": {
> > > "root": {
> > > "location": "/",
> > > "writable": false,
> > > "defaultInputFormat": null
> > > },
> > > "tmp": {
> > > "location": "/tmp",
> > > "writable": true,
> > > "defaultInputFormat": null
> > > }
> > > },
> > > "formats": {
> > > "psv": {
> > > "type": "text",
> > > "extensions": [
> > > "tbl"
> > > ],
> > > "delimiter": "|"
> > > },
> > > "csv": {
> > > "type": "text",
> > > "extensions": [
> > > "csv"
> > > ],
> > > "extractHeader": true,
> > > "delimiter": ","
> > > },
> > > "tsv": {
> > > "type": "text",
> > > "extensions": [
> > > "tsv"
> > > ],
> > > "delimiter": "\t"
> > > },
> > > "parquet": {
> > > "type": "parquet"
> > > },
> > > "json": {
> > > "type": "json",
> > > "extensions": [
> > > "json"
> > > ]
> > > },
> > > "avro": {
> > > "type": "avro"
> > > },
> > > "sequencefile": {
> > > "type": "sequencefile",
> > > "extensions": [
> > > "seq"
> > > ]
> > > },
> > > "csvh": {
> > > "type": "text",
> > > "extensions": [
> > > "csvh"
> > > ],
> > > "extractHeader": true,
> > > "delimiter": ","
> > > }
> > > }
> > > }
> > >
> > > On Fri, May 26, 2017 at 10:29 AM, Shuporno Choudhury <
> > > [email protected]> wrote:
> > >
> > > > Hi,
> > > > Can someone at Drill help me with issue please?
> > > >
> > > > On Thu, May 25, 2017 at 1:33 PM, Shuporno Choudhury <
> > > > [email protected]> wrote:
> > > >
> > > > > HI,
> > > > >
> > > > > I corrected the "show schemas" output by putting only "/" in the
> > > > > "location" . Now it shows s3.tmp in the output.
> > > > >
> > > > > But, it has a weird problem.
> > > > > The moment I add a folder to the location, eg: "/myfolder", then
> > s3.tmp
> > > > > vanishes from the "show schemas" output.
> > > > >
> > > > > Also, when I try to write into s3, I get the following error:
> > > > >
> > > > > Exception in thread "drill-executor-9" java.lang.
> > UnsatisfiedLinkError:
> > > > > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(
> > > > > Ljava/lang/String;I)Z
> > > > > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(
> > > Native
> > > > > Method)+--+
> > > > > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(
> > > > > NativeIO.java:609)
> > > > >
> > > > > This is only a snippet of the error associated with writing to s3
> > > > >
> > > > > On Thu, May 25, 2017 at 12:41 PM, Shuporno Choudhury <
> > > > > [email protected]> wrote:
> > > > >
> > > > >> My s3 plugin info is as follows:
> > > > >>
> > > > >> {
> > > > >> "type": "file",
> > > > >> "enabled": true,
> > > > >> "connection": "s3a://abcd",
> > > > >> "config": {
> > > > >> "fs.s3a.access.key": "abcd",
> > > > >> "fs.s3a.secret.key": "abcd"
> > > > >> },
> > > > >> "workspaces": {
> > > > >> "root": {
> > > > >> "location": "/",
> > > > >> "writable": false,
> > > > >> "defaultInputFormat": null
> > > > >> },
> > > > >> "tmp": {
> > > > >> "location": "/",
> > > > >> "writable": *true*,
> > > > >> "defaultInputFormat": "parquet"
> > > > >> }
> > > > >> }
> > > > >>
> > > > >>
> > > > >> I have removed the info about the formats to keep the mail small.
> > > > >> Also, I am using Dill on *Windows 10*
> > > > >>
> > > > >> On Mon, May 22, 2017 at 3:57 PM, Shuporno Choudhury <
> > > > >> [email protected]> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> Is it possible to write to a folder in an s3 bucket using the
> > > *s3.tmp*
> > > > >>> workspace?
> > > > >>> Whenever I try, it gives me the follwing error:
> > > > >>>
> > > > >>> *Error: VALIDATION ERROR: Schema [s3.tmp] is not valid with
> respect
> > > to
> > > > >>> either root schema or current default schema.*
> > > > >>> *Current default schema: s3.root*
> > > > >>>
> > > > >>> Also, s3.tmp doesn't appear while using the command "*show
> > schemas*"
> > > > >>> though the tmp workspace exists in the web console
> > > > >>>
> > > > >>> I am using Drill Version 1.10; embedded mode on my local system.
> > > > >>>
> > > > >>> However, I have no problem reading from an s3 bucket, the problem
> > is
> > > > >>> only writing to a s3 bucket.
> > > > >>> --
> > > > >>> Regards,
> > > > >>> Shuporno Choudhury
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >> Shuporno Choudhury
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Shuporno Choudhury
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Shuporno Choudhury
> > > >
> > >
> > >
> > >
> > > --
> > > Nitin Pawar
> > >
> >
> >
> >
> > --
> > Regards,
> > Shuporno Choudhury
> >
>
>
>
> --
> Nitin Pawar
>
--
Regards,
Shuporno Choudhury
