+ Trang - see Jason's comments below. Jason - I updated the core-site file to have all s3,s3n, and s3a configs.
Now....I'm getting an "IllegalArgumentException: Can not create a Path from an empty string" error now from all my queries below. My core-site file is edited with s3, s3n, and s3a configs - each with corresponding plugins on the web interface. <property> <name>fs.s3.awsAccessKeyId</name> <value> </value> </property> <property> <name>fs.s3.awsSecretAccessKey</name> <value> </value> </property> <property> <name>fs.s3n.awsAccessKeyId</name> <value> </value> </property> <property> <name>fs.s3n.awsSecretAccessKey</name> <value> </value> </property> <property> <name>fs.s3a.awsAccessKeyId</name> <value> </value> </property> <property> <name>fs.s3a.awsSecretAccessKey</name> <value> </value> </property> SELECT * FROM s3a. `/data/year=2016/month=04/day=01/part-r-00000-2e33bf90-9e25-41a9-9a71-8ab758295686.gz.parquet` LIMIT 3; SELECT * FROM s3n. `/data/year=2016/month=04/day=01/part-r-00000-2e33bf90-9e25-41a9-9a71-8ab758295686.gz.parquet` LIMIT 3; SELECT * FROM s3. `/data/year=2016/month=04/day=01/part-r-00000-2e33bf90-9e25-41a9-9a71-8ab758295686.gz.parquet` LIMIT 3; SELECT * FROM s3a. `/data/year=2016/month=04/day=01/` LIMIT 3; RESPONSE: IllegalArgumentException: Can not create a Path from an empty string. Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence | www.inrix.com | mobile +1 646-248-4105 | -----Original Message----- From: Jason Altekruse [mailto:ja...@dremio.com] Sent: Wednesday, April 20, 2016 5:43 PM To: user@drill.apache.org Cc: abhishek.gir...@gmail.com Subject: Re: Unable to connect to S3 parquet data using Drill Looking here it appears you need to set up an empty bucket to store a filesystem if you are going to use s3:// [1]. Are you trying to connect to a bucket you have populated with the normal S3 bucket APIs and not just the HDFS FileSystem API calls? Have you tried connecting instead with s3a? It looks like from this doc page that s3n and s3a are designed to connect to existing buckets filled with files, with s3a being a complete replacement for s3n. I believe the error you are seeing means it cannot find the path "/". It is probably trying to look up the root of the filesystem wherever it puts metadata in the bucket (maybe a hidden file or something?) and it isn't finding it. This makes me think that your bucket isn't set up as it is expected to be for a connection using s3://. [1] - https://wiki.apache.org/hadoop/AmazonS3 Jason Altekruse Software Engineer at Dremio Apache Drill Committer On Wed, Apr 20, 2016 at 2:34 PM, Nick Monetta <ni...@inrix.com> wrote: > Thanks for the quick responses! > > I'm using drill 1.4. I think I may have sorted out my S3 connections > issues, but I'm not sure because I'm having trouble executing a query: > > > My s3 connection (named "s3"): > { > "type": "file", > "enabled": true, > "connection": "s3://inrixprod-tapp/", > "workspaces": { > "root": { > "location": "/", > "writable": false, > "defaultInputFormat": null > } > > Query: > SELECT * FROM > s3.`data/year=2016/month=02/day=28/part-r-00000-f2b42e00-ff01-4d82-84e > 3-c75aafa007ae.gz.parquet` > LIMIT 3; > > Response: > Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > IOException: / doesn't exist [Error Id: > 9e076a2b-c4fa-4020-af2e-4d43c2e9588c on > NickM-LPT02.inrix.corpnet.local:31010] > > > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence | > www.inrix.com | mobile +1 646-248-4105 | > > > -----Original Message----- > From: Jason Altekruse [mailto:ja...@dremio.com] > Sent: Wednesday, April 20, 2016 4:45 PM > To: user@drill.apache.org > Cc: abhishek.gir...@gmail.com > Subject: Re: Unable to connect to S3 parquet data using Drill > > Which version of Drill are you running? The config block for adding > your credentials was added in a recent release, I believe 1.5. > > Jason Altekruse > Software Engineer at Dremio > Apache Drill Committer > > On Wed, Apr 20, 2016 at 1:38 PM, Nick Monetta <ni...@inrix.com> wrote: > > > Copying and pasting your JSON directly into a new configuration gets > > me “Error (invalid JSON Mapping)”. > > > > > > > > What am I doing wrong? > > > > > > > > > > > > > > > > > > > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence | > > www.inrix.com | mobile +1 646-248-4105 | > > > > > > > > > > > > -----Original Message----- > > From: Jason Altekruse [mailto:ja...@dremio.com] > > Sent: Wednesday, April 20, 2016 4:27 PM > > To: user@drill.apache.org > > Cc: abhishek.gir...@gmail.com > > Subject: Re: Unable to connect to S3 parquet data using Drill > > > > > > > > { > > > > "type": "file", > > > > "enabled": true, > > > > "connection": "s3a://PATH.TO.BUCKET/", > > > > "config": { > > > > "fs.s3a.access.key": "<YOUR ACCESS KEY HERE>", > > > > "fs.s3a.secret.key": "<YOUR SECRET KEY HERE>" > > > > }, > > > > "workspaces": { > > > > "root": { > > > > "location": "/", > > > > "writable": false, > > > > "defaultInputFormat": null > > > > }, > > > > "tmp": { > > > > "location": "/tmp", > > > > "writable": true, > > > > "defaultInputFormat": null > > > > } > > > > }, > > > > "formats": { > > > > "psv": { > > > > "type": "text", > > > > "extensions": [ > > > > "tbl" > > > > ], > > > > "delimiter": "|" > > > > }, > > > > "csv": { > > > > "type": "text", > > > > "extensions": [ > > > > "csv" > > > > ], > > > > "delimiter": "," > > > > }, > > > > "tsv": { > > > > "type": "text", > > > > "extensions": [ > > > > "tsv" > > > > ], > > > > "delimiter": "\t" > > > > }, > > > > "parquet": { > > > > "type": "parquet" > > > > }, > > > > "json": { > > > > "type": "json", > > > > "extensions": [ > > > > "json" > > > > ] > > > > }, > > > > "avro": { > > > > "type": "avro" > > > > }, > > > > "sequencefile": { > > > > "type": "sequencefile", > > > > "extensions": [ > > > > "seq" > > > > ] > > > > }, > > > > "csvh": { > > > > "type": "text", > > > > "extensions": [ > > > > "csvh" > > > > ], > > > > "extractHeader": true, > > > > "delimiter": "," > > > > } > > > > } > > > > } > > > > > > > > Jason Altekruse > > > > Software Engineer at Dremio > > > > Apache Drill Committer > > > > > > > > On Wed, Apr 20, 2016 at 1:24 PM, Nick Monetta <ni...@inrix.com> wrote: > > > > > > > > > Can you send me the full JSON for the new config example you provided? > > > > > I keep getting JSON errors. > > > > > > > > > > > > > > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence | > > > > > www.inrix.com | mobile +1 646-248-4105 | > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Abhishek Girish [mailto:abhishek.gir...@gmail.com > > <abhishek.gir...@gmail.com>] > > > > > Sent: Wednesday, April 20, 2016 12:57 PM > > > > > To: user <user@drill.apache.org> > > > > > Subject: Re: Unable to connect to S3 parquet data using Drill > > > > > > > > > > Hey Trang, > > > > > > > > > > A similar issue related to S3 config was discussed today on the > > > > > mailing list [1]. Can you see if that helps resolve the issue? > > > > > > > > > > [1] > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/drill-dev/201604.mbox/%3C > > > CA > > > N6 > > > > > ttnukzsAKgQE-RTF0RNCvBr1uWsB9SaxnS_7y-v0yBdUj%3Dw%40mail.gmail.com > > > %3 > > > E > > > > > > > > > > > > > > > -Abhishek > > > > > > > > > > On Tue, Apr 19, 2016 at 6:38 PM, Trang Nguyen > > > <trang.ngu...@inrix.com> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I am having trouble to connect to an Amazon S3 bucket containing > > > > > > parquet files. > > > > > > I followed the instructions on > > > > > > https://drill.apache.org/docs/s3-storage-plugin/ to download > > > > > > jets3_0.9.3 on my Ubuntu VM. > > > > > > My storage configs: > > > > > > { > > > > > > "type": "file", > > > > > > "enabled": true, > > > > > > "connection": "s3://inrixprod-tapp", > > > > > > "config": null, > > > > > > "workspaces": { > > > > > > "root": { > > > > > > "location": "/", > > > > > > "writable": false, > > > > > > "defaultInputFormat": null > > > > > > }, > > > > > > "tmp": { > > > > > > "location": "/tmp", > > > > > > "writable": true, > > > > > > "defaultInputFormat": null > > > > > > } > > > > > > }, > > > > > > ... > > > > > > } > > > > > > > > > > > > I've started the embedded-drill instance but get the following > > > > error > > > > > > trying to connect: > > > > > > 0: jdbc:drill:zk=local> use s3-trips.`root`; > > > > > > Error: SYSTEM ERROR: IOException: / doesn't exist > > > > > > > > > > > > > > > > > > [Error Id: 081c66e6-177d-48fa-8eca-4ee1370ae785 on > > > > > > ubuntu-VirtualBox:31010] (state=,code=0) > > > > > > > > > > > > Any advice would be appreciated! > > > > > > > > > > > > Thanks, > > > > > > Trang > > > > > > > > > > > > > >
