Hi Padma,
I’m wondering if the issue is that I only have access to a subfolder in the s3
bucket. IE:
s3://bucket/folder1/folder2/folder3 <s3://bucket/folder1/folder2/folder3>
I only have access to folder3. Might that be causing the issue?
—C
> On Oct 24, 2017, at 13:49, Padma Penumarthy <ppenumar...@mapr.com> wrote:
>
> Charles, can you try exactly what I did.
> I did not do anything else other than enable the S3 plugin and change the
> plugin
> configuration like this.
>
> {
> "type": "file",
> "enabled": true,
> "connection": "s3a://<bucket-name>",
> "config": {
> "fs.s3a.access.key": “XXXX",
> "fs.s3a.secret.key": “YYYY"
> },
>
>
> Thanks
> Padma
>
>
>> On Oct 24, 2017, at 10:06 AM, Charles Givre <cgi...@gmail.com
>> <mailto:cgi...@gmail.com>> wrote:
>>
>> Hi everyone and thank you for your help. I’m still not able to connect to
>> S3.
>>
>>
>> Here is the error I’m getting:
>>
>> 0: jdbc:drill:zk=local> use s3;
>> Error: RESOURCE ERROR: Failed to create schema tree.
>>
>>
>> [Error Id: 57c82d90-2166-4a37-94a0-1cfeb0cdc4b6 on
>> charless-mbp-2.fios-router.home:31010] (state=,code=0)
>> java.sql.SQLException: RESOURCE ERROR: Failed to create schema tree.
>>
>>
>> [Error Id: 57c82d90-2166-4a37-94a0-1cfeb0cdc4b6 on
>> charless-mbp-2.fios-router.home:31010]
>> at
>> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
>> at
>> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
>> at
>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
>> at
>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
>> at
>> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
>> at
>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
>> at
>> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
>> at
>> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
>> at
>> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
>> at
>> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:121)
>> at
>> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:101)
>> at sqlline.Commands.execute(Commands.java:841)
>> at sqlline.Commands.sql(Commands.java:751)
>> at sqlline.SqlLine.dispatch(SqlLine.java:746)
>> at sqlline.SqlLine.begin(SqlLine.java:621)
>> at sqlline.SqlLine.start(SqlLine.java:375)
>> at sqlline.SqlLine.main(SqlLine.java:268)
>> Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
>> ERROR: Failed to create schema tree.
>>
>>
>> [Error Id: 57c82d90-2166-4a37-94a0-1cfeb0cdc4b6 on
>> charless-mbp-2.fios-router.home:31010]
>> at
>> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
>> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:368)
>> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:90)
>> at
>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
>> at
>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
>> at
>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>> at
>> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>> at
>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>> at
>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>> at
>> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>> at
>> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
>> at
>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>> at java.lang.Thread.run(Thread.java:745)
>> 0: jdbc:drill:zk=local>
>>
>>
>> Here is my conf.site <http://conf.site/> <http://conf.site/
>> <http://conf.site/>>.xml file:
>>
>> <property>
>> <name>fs.s3.awsAccessKeyId</name>
>> <value>XXX</value>
>> </property>
>>
>> <property>
>> <name>fs.s3.awsSecretAccessKey</name>
>> <value> XXX </value>
>> </property>
>> <property>
>> <name>fs.s3n.awsAccessKeyId</name>
>> <value> XXX </value>
>> </property>
>>
>> <property>
>> <name>fs.s3n.awsSecretAccessKey</name>
>> <value> XXX </value>
>> </property>
>> <property>
>> <name>fs.s3a.awsAccessKeyId</name>
>> <value> XXX </value>
>> </property>
>>
>> <property>
>> <name>fs.s3a.awsSecretAccessKey</name>
>> <value> XXX </value>
>> </property>
>>
>> And my config info:
>>
>> {
>> "type": "file",
>> "enabled": true,
>> "connection": "s3://<my bucket>",
>> "config": null,
>> "workspaces": {
>> "root": {
>> "location": "/",
>> "writable": false,
>> "defaultInputFormat": null
>> }
>> },
>>
>> I did copy jets3t-0.9.4.jar to the /jars/3rdparty path. Any debugging
>> suggestions?
>> —C
>>
>>
>>> On Oct 20, 2017, at 15:55, Arjun kr <arjun...@outlook.com> wrote:
>>>
>>> Hi Charles,
>>>
>>>
>>> I'm not aware of any such settings. As Padma mentioned in previous mail, It
>>> works fine for me by following instructions in
>>> https://drill.apache.org/docs/s3-storage-plugin/ .
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Arjun
>>>
>>>
>>> ________________________________
>>> From: Charles Givre <cgi...@gmail.com>
>>> Sent: Friday, October 20, 2017 11:48 PM
>>> To: user@drill.apache.org
>>> Subject: Re: S3 Connection Issues
>>>
>>> Hi Arjun,
>>> Thanks for your help. Are there settings in S3 that would prevent Drill
>>> from connecting? I’ll try hdfs shell, but I am able to connect with the
>>> CLI tool. My hunch is that there is a permission not set correctly on S3
>>> or I’m missing some config variable in Drill.
>>> — C
>>>
>>>
>>>> On Oct 20, 2017, at 14:12, Arjun kr <arjun...@outlook.com> wrote:
>>>>
>>>> Hi Charles,
>>>>
>>>>
>>>> Any chance you can test s3 connectivity with other tools like hdfs shell
>>>> or hive in case you haven't tried already (and these tools available)?
>>>> This may help to identify if it is Drill specific issue.
>>>>
>>>>
>>>> For connecting via hdfs , you may try below command.
>>>>
>>>>
>>>> hadoop fs -Dfs.s3a.access.key="XXXX" -Dfs.s3a.secret.key="YYYYY" -ls
>>>> s3a://<bucket-name>/
>>>>
>>>>
>>>> Enable DEBUG logging if needed.
>>>>
>>>>
>>>> export HADOOP_ROOT_LOGGER=hadoop.root.logger=DEBUG,console
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Arjun
>>>>
>>>>
>>>> ________________________________
>>>> From: Padma Penumarthy <ppenumar...@mapr.com>
>>>> Sent: Friday, October 20, 2017 3:00 AM
>>>> To: user@drill.apache.org
>>>> Subject: Re: S3 Connection Issues
>>>>
>>>> Hi Charles,
>>>>
>>>> I tried us-west-2 and it worked fine for me with drill built from latest
>>>> source.
>>>> I did not do anything special.
>>>> Just enabled the S3 plugin and updated the plugin configuration like this.
>>>>
>>>> {
>>>> "type": "file",
>>>> "enabled": true,
>>>> "connection": "s3a://<bucket-name>",
>>>> "config": {
>>>> "fs.s3a.access.key": “XXXX",
>>>> "fs.s3a.secret.key": “YYYY"
>>>> },
>>>>
>>>> I am able to do show databases and also can query the parquet files I
>>>> uploaded to the bucket.
>>>>
>>>> 0: jdbc:drill:zk=local> show databases;
>>>> +---------------------+
>>>> | SCHEMA_NAME |
>>>> +---------------------+
>>>> | INFORMATION_SCHEMA |
>>>> | cp.default |
>>>> | dfs.default |
>>>> | dfs.root |
>>>> | dfs.tmp |
>>>> | s3.default |
>>>> | s3.root |
>>>> | sys |
>>>> +---------------------+
>>>> 8 rows selected (2.892 seconds)
>>>>
>>>>
>>>> Thanks
>>>> Padma
>>>>
>>>> On Oct 18, 2017, at 9:18 PM, Charles Givre
>>>> <cgi...@gmail.com<mailto:cgi...@gmail.com>> wrote:
>>>>
>>>> Hi Padma,
>>>> The bucket is is us-west-2. I also discovered that some of the variable
>>>> names in the documentation on the main Drill site are incorrect. Do I
>>>> need to specify the region in the configuration somewhere?
>>>>
>>>> As an update, after discovering that the variable names are incorrect and
>>>> that I didn’t have Jets3t installed properly, I’m now getting the
>>>> following error:
>>>>
>>>> jdbc:drill:zk=local> show databases;
>>>> Error: RESOURCE ERROR: Failed to create schema tree.
>>>>
>>>>
>>>> [Error Id: e6012aa2-c775-46b9-b3ee-0af7d0b0871d on
>>>> charless-mbp-2.fios-router.home:31010]
>>>>
>>>> (org.apache.hadoop.fs.s3.S3Exception)
>>>> org.jets3t.service.S3ServiceException: Service Error Message. --
>>>> ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml
>>>> version="1.0"
>>>> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The
>>>> request signature we calculated does not match the signature you provided.
>>>> Check your key and signing method.</Message></Error>
>>>> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get():175
>>>> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode():221
>>>>
>>>> Thanks,
>>>> — C
>>>>
>>>>
>>>> On Oct 19, 2017, at 00:14, Padma Penumarthy
>>>> <ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> wrote:
>>>>
>>>> Which AWS region are you trying to connect to ?
>>>> We have a problem connecting to regions which support only v4 signature
>>>> since the version of hadoop we include in Drill is old.
>>>> Last time I tried, using Hadoop 2.8.1 worked for me.
>>>>
>>>> Thanks
>>>> Padma
>>>>
>>>>
>>>> On Oct 18, 2017, at 8:14 PM, Charles Givre
>>>> <cgi...@gmail.com<mailto:cgi...@gmail.com>> wrote:
>>>>
>>>> Hello all,
>>>> I’m trying to use Drill to query data in an S3 bucket and running into
>>>> some issues which I can’t seem to fix. I followed the various
>>>> instructions online to set up Drill with S3, and put my keys in both the
>>>> conf-site.xml and in the plugin config, but every time I attempt to do
>>>> anything I get the following errors:
>>>>
>>>>
>>>> jdbc:drill:zk=local> show databases;
>>>> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service:
>>>> Amazon S3, AWS Request ID: 56D1999BD1E62DEB, AWS Error Code: null, AWS
>>>> Error Message: Forbidden
>>>>
>>>>
>>>> [Error Id: 65d0bb52-a923-4e98-8ab1-65678169140e on
>>>> charless-mbp-2.fios-router.home:31010] (state=,code=0)
>>>> 0: jdbc:drill:zk=local> show databases;
>>>> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service:
>>>> Amazon S3, AWS Request ID: 4D2CBA8D42A9ECA0, AWS Error Code: null, AWS
>>>> Error Message: Forbidden
>>>>
>>>>
>>>> [Error Id: 25a2d008-2f4d-4433-a809-b91ae063e61a on
>>>> charless-mbp-2.fios-router.home:31010] (state=,code=0)
>>>> 0: jdbc:drill:zk=local> show files in s3.root;
>>>> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service:
>>>> Amazon S3, AWS Request ID: 2C635944EDE591F0, AWS Error Code: null, AWS
>>>> Error Message: Forbidden
>>>>
>>>>
>>>> [Error Id: 02e136f5-68c0-4b47-9175-a9935bda5e1c on
>>>> charless-mbp-2.fios-router.home:31010] (state=,code=0)
>>>> 0: jdbc:drill:zk=local> show schemas;
>>>> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service:
>>>> Amazon S3, AWS Request ID: 646EB5B2EBCF7CD2, AWS Error Code: null, AWS
>>>> Error Message: Forbidden
>>>>
>>>>
>>>> [Error Id: 954aaffe-616a-4f40-9ba5-d4b7c04fe238 on
>>>> charless-mbp-2.fios-router.home:31010] (state=,code=0)
>>>>
>>>> I have verified that the keys are correct but using the AWS CLI and
>>>> downloaded some of the files, but I’m kind of at a loss as to how to
>>>> debug. Any suggestions?
>>>> Thanks in advance,
>>>> — C
