If you were getting similar exception given below with original AWS jars with Drill 1.11, you can set DRILLBIT_JAVA_OPTS as mentioned in the previous mail. Earlier, there were regions that support only V2 and setting this parameter may be problematic with those regions. Now I see that all the regions support V4. So setting this property should be fine.
-- Without 'com.amazonaws.services.s3.enableV4=true' set in dril-env.sh 0: jdbc:drill:schema=dfs> select * from s3.root.`0_0_0.parquet` limit 10; Error: [MapR][DrillJDBCDriver](500165) Query execution error: SYSTEM ERROR: AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 1A036C181EA7B8D4, AWS Error Code: null, AWS Error Message: Bad Request [Error Id: bbe7d74a-b191-4ab2-99d7-10c5092c31d3 on arjun-lab-72:31010] (state=HY000,code=500165) 0: jdbc:drill:schema=dfs> -- With 'com.amazonaws.services.s3.enableV4=true' set in dril-env.sh "what ever the mind of man can conceive and believe, drill can query" 0: jdbc:drill:schema=dfs> select * from s3.root.`0_0_0.parquet` limit 10; +-----+ | id | +-----+ | 1 | +-----+ 1 row selected (7.975 seconds) 0: jdbc:drill:schema=dfs> https://docs.aws.amazon.com/general/latest/gr/rande.html Thanks, Arjun ________________________________ From: Arjun kr <[email protected]> Sent: Tuesday, February 13, 2018 8:47 PM To: [email protected] Subject: Re: S3 Connection Issues Hi Anup, Please see if below steps help. 1) Add below option $DRILL_HOME/conf/drill-env.sh export DRILLBIT_JAVA_OPTS="$DRILLBIT_JAVA_OPTS -Dcom.amazonaws.services.s3.enableV4=true" 2) Restart the drillbit service and try querying S4 region. Thanks, Arjun ________________________________ From: Anup Tiwari <[email protected]> Sent: Tuesday, February 13, 2018 3:16 PM To: [email protected] Subject: Re: S3 Connection Issues Hi Padma, As you have mentioned "Last time I tried, using Hadoop 2.8.1 worked for me." so have you build drill with hadoop 2.8.1 ? If yes then can you provide steps ? Since i have downloaded tar ball of 1.11.0 and replaced hadoop-aws-2.7.1.jar with hadoop-aws-2.9.0.jar but still not able to query successfully to s3 bucket; queries are going in starting state. We are trying to query : "ap-south-1" region which supports only v4 signature. On Thu, Oct 19, 2017 9:44 AM, Padma Penumarthy [email protected] wrote: Which AWS region are you trying to connect to ? We have a problem connecting to regions which support only v4 signature since the version of hadoop we include in Drill is old. Last time I tried, using Hadoop 2.8.1 worked for me. Thanks Padma > On Oct 18, 2017, at 8:14 PM, Charles Givre <[email protected]> wrote: > > Hello all, > I’m trying to use Drill to query data in an S3 bucket and running into some issues which I can’t seem to fix. I followed the various instructions online to set up Drill with S3, and put my keys in both the conf-site.xml and in the plugin config, but every time I attempt to do anything I get the following errors: > > > jdbc:drill:zk=local> show databases; > Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 56D1999BD1E62DEB, AWS Error Code: null, AWS Error Message: Forbidden > > > [Error Id: 65d0bb52-a923-4e98-8ab1-65678169140e on charless-mbp-2.fios-router.home:31010] (state=,code=0) > 0: jdbc:drill:zk=local> show databases; > Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 4D2CBA8D42A9ECA0, AWS Error Code: null, AWS Error Message: Forbidden > > > [Error Id: 25a2d008-2f4d-4433-a809-b91ae063e61a on charless-mbp-2.fios-router.home:31010] (state=,code=0) > 0: jdbc:drill:zk=local> show files in s3.root; > Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 2C635944EDE591F0, AWS Error Code: null, AWS Error Message: Forbidden > > > [Error Id: 02e136f5-68c0-4b47-9175-a9935bda5e1c on charless-mbp-2.fios-router.home:31010] (state=,code=0) > 0: jdbc:drill:zk=local> show schemas; > Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 646EB5B2EBCF7CD2, AWS Error Code: null, AWS Error Message: Forbidden > > > [Error Id: 954aaffe-616a-4f40-9ba5-d4b7c04fe238 on charless-mbp-2.fios-router.home:31010] (state=,code=0) > > I have verified that the keys are correct but using the AWS CLI and downloaded some of the files, but I’m kind of at a loss as to how to debug. Any suggestions? > Thanks in advance, > — C Regards, Anup Tiwari Sent with Mixmax
