Maybe another way to go is to copy Jets3tNativeFileSystemStore and any necessary dependent classes to another name, modify, and register it under a different URL scheme (not s3n)
On 6/18/15, 3:54 PM, "Paul Mogren" <[email protected]> wrote: >Thanks. > > >I tried to follow up on the upcoming Jets3t support for IAM roles. >Dropping in a snapshot build of Jets3t with Drill was not enough. I was >going to try patching Drill to take advantage, but I found that Drill >inherits its ability to treat S3 as a DFS from hadoop-common’s >org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore, which delegates >some initialization to org.apache.hadoop.fs.s3.S3Credentials, which >explicitly throws IllegalArgumentException if the accessKey and/or >secretAccessKey is not provided (stack trace below). So we’re looking at >waiting for Jets3t making a release, having it picked up by Hadoop, patch >Hadoop to allow use of IAM role, Hadoop making a release, and having that >available in your stack (e.g. AWS offering it in EMR). That’s going to >take quite some time. One could probably do the patching oneself and slip >it into the classpath, but it makes more sense to try EMRFS. > > >2015-06-18 18:12:28,997 [qtp1208140271-50] ERROR >o.a.d.e.server.rest.QueryResources - Query from Web UI Failed >org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access >Key must be specified as the username or password (respective >ly) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or >fs.s3n.awsSecretAccessKey properties (respectively). > > > > >[Error Id: 32c0357e-b8df-48f0-98ae-f200d48dd0f5 on >ip-172-24-7-103.ec2.internal:31010] > > > (org.apache.drill.exec.work.foreman.ForemanException) Unexpected >exception during fragment initialization: AWS Access Key ID and Secret >Access Key must be specified as the username or password (respect >ively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or >fs.s3n.awsSecretAccessKey properties (respectively). > org.apache.drill.exec.work.foreman.Foreman.run():251 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > Caused By (java.lang.IllegalArgumentException) AWS Access Key ID and >Secret Access Key must be specified as the username or password >(respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId > or fs.s3n.awsSecretAccessKey properties (respectively). > org.apache.hadoop.fs.s3.S3Credentials.initialize():70 > >org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize():73 > sun.reflect.NativeMethodAccessorImpl.invoke0():-2 > sun.reflect.NativeMethodAccessorImpl.invoke():57 > sun.reflect.DelegatingMethodAccessorImpl.invoke():43 > java.lang.reflect.Method.invoke():606 > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():190 > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():103 > org.apache.hadoop.fs.s3native.$Proxy65.initialize():-1 > org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize():272 > org.apache.hadoop.fs.FileSystem.createFileSystem():2397 > org.apache.hadoop.fs.FileSystem.access$200():89 > org.apache.hadoop.fs.FileSystem$Cache.getInternal():2431 > org.apache.hadoop.fs.FileSystem$Cache.get():2413 > org.apache.hadoop.fs.FileSystem.get():368 > org.apache.hadoop.fs.FileSystem.get():167 > org.apache.drill.exec.store.dfs.DrillFileSystem.<init>():85 > org.apache.drill.exec.util.ImpersonationUtil$1.run():156 > org.apache.drill.exec.util.ImpersonationUtil$1.run():153 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1556 > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():153 > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():145 > org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():133 > >org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.<in >i >t>():125 > >org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.createSchema():114 > >org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.< >i >nit>():77 > >org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(): >6 >4 > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():131 > >org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.regis >t >erSchemas():330 >org.apache.drill.exec.ops.QueryContext.getRootSchema():158 > org.apache.drill.exec.ops.QueryContext.getRootSchema():147 > org.apache.drill.exec.ops.QueryContext.getRootSchema():135 > org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():121 > org.apache.drill.exec.planner.sql.DrillSqlWorker.<init>():90 > org.apache.drill.exec.work.foreman.Foreman.runSQL():900 > org.apache.drill.exec.work.foreman.Foreman.run():240 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > > > at >org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResul >t >Handler.java:118) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at >org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:11 >1 >) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at >org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWith >C >onnection.java:47) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at >org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWith >C >onnection.java:32) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) >~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at >org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) >~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at >org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) >~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] > at >io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessag >e >Decoder.java:89) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC >h >annelHandlerContext.java:339) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCha >n >nelHandlerContext.java:324) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.jav >a >:254) ~[netty-handler-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC >h >annelHandlerContext.java:339) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCha >n >nelHandlerContext.java:324) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessag >e >Decoder.java:103) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC >h >annelHandlerContext.java:339) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCha >n >nelHandlerContext.java:324) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecod >e >r.java:242) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC >h >annelHandlerContext.java:339) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCha >n >nelHandlerContext.java:324) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHa >n >dlerAdapter.java:86) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC >h >annelHandlerContext.java:339) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCha >n >nelHandlerContext.java:324) >~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipe >l >ine.java:847) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > at >io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollI >n >Ready(AbstractEpollStreamChannel.java:618) >~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at >io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:329 >) > ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at >io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250) >~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at >io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEvent >E >xecutor.java:111) ~[netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_71] > > > > > > > > >On 6/18/15, 11:28 AM, "Ted Dunning" <[email protected]> wrote: > >>On Thu, Jun 18, 2015 at 8:24 AM, Paul Mogren <[email protected]> >>wrote: >> >>> Following up. Ted gave sound advice regarding reading S3 vs HDFS, but >>> didn¹t address EMRFS specifically. Here is what I have learned. >>> >> >>Great summary. Very useful when people help by feeding back what they >>have >>learned. >
