Hi Nitin,
Thanks for the config settings.
Now, after entering those config settings
1. s3.tmp does appear in the "show schemas" result
2. Also, it doesn't disappear when I add a custom folder in the
location attribute
But when I try to run a CTAS statement, I get the following error:
*Error: SYSTEM ERROR: IllegalArgumentException: URI has an authority
component*
*Fragment 0:0*
Query that I am trying to run:
*create table s3.tmp.`abcd` as select 1 from (values(1));*
However, this query runs when I use dfs.tmp instead of s3.tmp
On Fri, May 26, 2017 at 12:44 PM, Nitin Pawar <[email protected]>
wrote:
> Can you try with following s3 config
>
> {
> "type": "file",
> "enabled": true,
> "connection": "s3a://bucket_name",
> "config": {
>
> "fs.s3a.connection.maximum": "10000",
> "fs.s3a.access.key": "access_key",
> "fs.s3a.secret.key": "secret_key",
> "fs.s3a.buffer.dir": "/tmp",
> "fs.s3a.multipart.size": "10485760",
> "fs.s3a.multipart.threshold": "104857600"
> },
> "workspaces": {
> "root": {
> "location": "/",
> "writable": false,
> "defaultInputFormat": null
> },
> "tmp": {
> "location": "/tmp",
> "writable": true,
> "defaultInputFormat": null
> }
> },
> "formats": {
> "psv": {
> "type": "text",
> "extensions": [
> "tbl"
> ],
> "delimiter": "|"
> },
> "csv": {
> "type": "text",
> "extensions": [
> "csv"
> ],
> "extractHeader": true,
> "delimiter": ","
> },
> "tsv": {
> "type": "text",
> "extensions": [
> "tsv"
> ],
> "delimiter": "\t"
> },
> "parquet": {
> "type": "parquet"
> },
> "json": {
> "type": "json",
> "extensions": [
> "json"
> ]
> },
> "avro": {
> "type": "avro"
> },
> "sequencefile": {
> "type": "sequencefile",
> "extensions": [
> "seq"
> ]
> },
> "csvh": {
> "type": "text",
> "extensions": [
> "csvh"
> ],
> "extractHeader": true,
> "delimiter": ","
> }
> }
> }
>
> On Fri, May 26, 2017 at 10:29 AM, Shuporno Choudhury <
> [email protected]> wrote:
>
> > Hi,
> > Can someone at Drill help me with issue please?
> >
> > On Thu, May 25, 2017 at 1:33 PM, Shuporno Choudhury <
> > [email protected]> wrote:
> >
> > > HI,
> > >
> > > I corrected the "show schemas" output by putting only "/" in the
> > > "location" . Now it shows s3.tmp in the output.
> > >
> > > But, it has a weird problem.
> > > The moment I add a folder to the location, eg: "/myfolder", then s3.tmp
> > > vanishes from the "show schemas" output.
> > >
> > > Also, when I try to write into s3, I get the following error:
> > >
> > > Exception in thread "drill-executor-9" java.lang.UnsatisfiedLinkError:
> > > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(
> > > Ljava/lang/String;I)Z
> > > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(
> Native
> > > Method)+--+
> > > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(
> > > NativeIO.java:609)
> > >
> > > This is only a snippet of the error associated with writing to s3
> > >
> > > On Thu, May 25, 2017 at 12:41 PM, Shuporno Choudhury <
> > > [email protected]> wrote:
> > >
> > >> My s3 plugin info is as follows:
> > >>
> > >> {
> > >> "type": "file",
> > >> "enabled": true,
> > >> "connection": "s3a://abcd",
> > >> "config": {
> > >> "fs.s3a.access.key": "abcd",
> > >> "fs.s3a.secret.key": "abcd"
> > >> },
> > >> "workspaces": {
> > >> "root": {
> > >> "location": "/",
> > >> "writable": false,
> > >> "defaultInputFormat": null
> > >> },
> > >> "tmp": {
> > >> "location": "/",
> > >> "writable": *true*,
> > >> "defaultInputFormat": "parquet"
> > >> }
> > >> }
> > >>
> > >>
> > >> I have removed the info about the formats to keep the mail small.
> > >> Also, I am using Dill on *Windows 10*
> > >>
> > >> On Mon, May 22, 2017 at 3:57 PM, Shuporno Choudhury <
> > >> [email protected]> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Is it possible to write to a folder in an s3 bucket using the
> *s3.tmp*
> > >>> workspace?
> > >>> Whenever I try, it gives me the follwing error:
> > >>>
> > >>> *Error: VALIDATION ERROR: Schema [s3.tmp] is not valid with respect
> to
> > >>> either root schema or current default schema.*
> > >>> *Current default schema: s3.root*
> > >>>
> > >>> Also, s3.tmp doesn't appear while using the command "*show schemas*"
> > >>> though the tmp workspace exists in the web console
> > >>>
> > >>> I am using Drill Version 1.10; embedded mode on my local system.
> > >>>
> > >>> However, I have no problem reading from an s3 bucket, the problem is
> > >>> only writing to a s3 bucket.
> > >>> --
> > >>> Regards,
> > >>> Shuporno Choudhury
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Shuporno Choudhury
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shuporno Choudhury
> > >
> >
> >
> >
> > --
> > Regards,
> > Shuporno Choudhury
> >
>
>
>
> --
> Nitin Pawar
>
--
Regards,
Shuporno Choudhury
