Thanks.
I tried to follow up on the upcoming Jets3t support for IAM roles.
Dropping in a snapshot build of Jets3t with Drill was not enough. I was
going to try patching Drill to take advantage, but I found that Drill
inherits its ability to treat S3 as a DFS from hadoop-common’s
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore, which delegates
some initialization to org.apache.hadoop.fs.s3.S3Credentials, which
explicitly throws IllegalArgumentException if the accessKey and/or
secretAccessKey is not provided (stack trace below). So we’re looking at
waiting for Jets3t making a release, having it picked up by Hadoop, patch
Hadoop to allow use of IAM role, Hadoop making a release, and having that
available in your stack (e.g. AWS offering it in EMR). That’s going to
take quite some time. One could probably do the patching oneself and slip
it into the classpath, but it makes more sense to try EMRFS.
2015-06-18 18:12:28,997 [qtp1208140271-50] ERROR
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access
Key must be specified as the username or password (respective
ly) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or
fs.s3n.awsSecretAccessKey properties (respectively).
[Error Id: 32c0357e-b8df-48f0-98ae-f200d48dd0f5 on
ip-172-24-7-103.ec2.internal:31010]
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected
exception during fragment initialization: AWS Access Key ID and Secret
Access Key must be specified as the username or password (respect
ively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or
fs.s3n.awsSecretAccessKey properties (respectively).
org.apache.drill.exec.work.foreman.Foreman.run():251
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.IllegalArgumentException) AWS Access Key ID and
Secret Access Key must be specified as the username or password
(respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId
or fs.s3n.awsSecretAccessKey properties (respectively).
org.apache.hadoop.fs.s3.S3Credentials.initialize():70
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize():73
sun.reflect.NativeMethodAccessorImpl.invoke0():-2
sun.reflect.NativeMethodAccessorImpl.invoke():57
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():606
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():190
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():103
org.apache.hadoop.fs.s3native.$Proxy65.initialize():-1
org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize():272
org.apache.hadoop.fs.FileSystem.createFileSystem():2397
org.apache.hadoop.fs.FileSystem.access$200():89
org.apache.hadoop.fs.FileSystem$Cache.getInternal():2431
org.apache.hadoop.fs.FileSystem$Cache.get():2413
org.apache.hadoop.fs.FileSystem.get():368
org.apache.hadoop.fs.FileSystem.get():167
org.apache.drill.exec.store.dfs.DrillFileSystem.<init>():85
org.apache.drill.exec.util.ImpersonationUtil$1.run():156
org.apache.drill.exec.util.ImpersonationUtil$1.run():153
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():415
org.apache.hadoop.security.UserGroupInformation.doAs():1556
org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():153
org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():145
org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():133
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.<ini
t>():125
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.createSchema():114
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<i
nit>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():6
4
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():131
org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.regist
erSchemas():330
org.apache.drill.exec.ops.QueryContext.getRootSchema():158
org.apache.drill.exec.ops.QueryContext.getRootSchema():147
org.apache.drill.exec.ops.QueryContext.getRootSchema():135
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():121
org.apache.drill.exec.planner.sql.DrillSqlWorker.<init>():90
org.apache.drill.exec.work.foreman.Foreman.runSQL():900
org.apache.drill.exec.work.foreman.Foreman.run():240
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResult
Handler.java:118) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at
org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:111
) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithC
onnection.java:47) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithC
onnection.java:32) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61)
~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233)
~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205)
~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessage
Decoder.java:89) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractCh
annelHandlerContext.java:339)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChan
nelHandlerContext.java:324)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java
:254) ~[netty-handler-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractCh
annelHandlerContext.java:339)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChan
nelHandlerContext.java:324)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessage
Decoder.java:103) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractCh
annelHandlerContext.java:339)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChan
nelHandlerContext.java:324)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecode
r.java:242) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractCh
annelHandlerContext.java:339)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChan
nelHandlerContext.java:324)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHan
dlerAdapter.java:86) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractCh
annelHandlerContext.java:339)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChan
nelHandlerContext.java:324)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipel
ine.java:847) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollIn
Ready(AbstractEpollStreamChannel.java:618)
~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:329)
~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventE
xecutor.java:111) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_71]
On 6/18/15, 11:28 AM, "Ted Dunning" <[email protected]> wrote:
>On Thu, Jun 18, 2015 at 8:24 AM, Paul Mogren <[email protected]>
>wrote:
>
>> Following up. Ted gave sound advice regarding reading S3 vs HDFS, but
>> didn¹t address EMRFS specifically. Here is what I have learned.
>>
>
>Great summary. Very useful when people help by feeding back what they
>have
>learned.
