If you go down that route, it may be better to ditch Jets3t in favor of the AWS SDK, as Presto and much of the community has.
https://github.com/facebook/presto/pull/991/files https://github.com/facebook/presto/blob/master/presto-hive/src/main/java/co m/facebook/presto/hive/HdfsConfigurationUpdater.java On 6/18/15, 4:08 PM, "Paul Mogren" <[email protected]> wrote: >Maybe another way to go is to copy Jets3tNativeFileSystemStore and any >necessary dependent classes to another name, modify, and register it under >a different URL scheme (not s3n) > > > > >On 6/18/15, 3:54 PM, "Paul Mogren" <[email protected]> wrote: > >>Thanks. >> >> >>I tried to follow up on the upcoming Jets3t support for IAM roles. >>Dropping in a snapshot build of Jets3t with Drill was not enough. I was >>going to try patching Drill to take advantage, but I found that Drill >>inherits its ability to treat S3 as a DFS from hadoop-common’s >>org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore, which >>delegates >>some initialization to org.apache.hadoop.fs.s3.S3Credentials, which >>explicitly throws IllegalArgumentException if the accessKey and/or >>secretAccessKey is not provided (stack trace below). So we’re looking at >>waiting for Jets3t making a release, having it picked up by Hadoop, patch >>Hadoop to allow use of IAM role, Hadoop making a release, and having that >>available in your stack (e.g. AWS offering it in EMR). That’s going to >>take quite some time. One could probably do the patching oneself and >>slip >>it into the classpath, but it makes more sense to try EMRFS. >> >> >>2015-06-18 18:12:28,997 [qtp1208140271-50] ERROR >>o.a.d.e.server.rest.QueryResources - Query from Web UI Failed >>org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >>java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access >>Key must be specified as the username or password (respective >>ly) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or >>fs.s3n.awsSecretAccessKey properties (respectively). >> >> >> >> >>[Error Id: 32c0357e-b8df-48f0-98ae-f200d48dd0f5 on >>ip-172-24-7-103.ec2.internal:31010] >> >> >> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected >>exception during fragment initialization: AWS Access Key ID and Secret >>Access Key must be specified as the username or password (respect >>ively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or >>fs.s3n.awsSecretAccessKey properties (respectively). >> org.apache.drill.exec.work.foreman.Foreman.run():251 >> java.util.concurrent.ThreadPoolExecutor.runWorker():1145 >> java.util.concurrent.ThreadPoolExecutor$Worker.run():615 >> java.lang.Thread.run():745 >> Caused By (java.lang.IllegalArgumentException) AWS Access Key ID and >>Secret Access Key must be specified as the username or password >>(respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId >> or fs.s3n.awsSecretAccessKey properties (respectively). >> org.apache.hadoop.fs.s3.S3Credentials.initialize():70 >> >>org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize():73 >> sun.reflect.NativeMethodAccessorImpl.invoke0():-2 >> sun.reflect.NativeMethodAccessorImpl.invoke():57 >> sun.reflect.DelegatingMethodAccessorImpl.invoke():43 >> java.lang.reflect.Method.invoke():606 >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():190 >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():103 >> org.apache.hadoop.fs.s3native.$Proxy65.initialize():-1 >> org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize():272 >> org.apache.hadoop.fs.FileSystem.createFileSystem():2397 >> org.apache.hadoop.fs.FileSystem.access$200():89 >> org.apache.hadoop.fs.FileSystem$Cache.getInternal():2431 >> org.apache.hadoop.fs.FileSystem$Cache.get():2413 >> org.apache.hadoop.fs.FileSystem.get():368 >> org.apache.hadoop.fs.FileSystem.get():167 >> org.apache.drill.exec.store.dfs.DrillFileSystem.<init>():85 >> org.apache.drill.exec.util.ImpersonationUtil$1.run():156 >> org.apache.drill.exec.util.ImpersonationUtil$1.run():153 >> java.security.AccessController.doPrivileged():-2 >> javax.security.auth.Subject.doAs():415 >> org.apache.hadoop.security.UserGroupInformation.doAs():1556 >> org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():153 >> org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():145 >> org.apache.drill.exec.util.ImpersonationUtil.createFileSystem():133 >> >>org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.<i >>n >>i >>t>():125 >> >>org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.createSchema():114 >> >>org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema. >>< >>i >>nit>():77 >> >>org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas() >>: >>6 >>4 >> >>org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():131 >> >>org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.regi >>s >>t >>erSchemas():330 >>org.apache.drill.exec.ops.QueryContext.getRootSchema():158 >> org.apache.drill.exec.ops.QueryContext.getRootSchema():147 >> org.apache.drill.exec.ops.QueryContext.getRootSchema():135 >> org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():121 >> org.apache.drill.exec.planner.sql.DrillSqlWorker.<init>():90 >> org.apache.drill.exec.work.foreman.Foreman.runSQL():900 >> org.apache.drill.exec.work.foreman.Foreman.run():240 >> java.util.concurrent.ThreadPoolExecutor.runWorker():1145 >> java.util.concurrent.ThreadPoolExecutor$Worker.run():615 >> java.lang.Thread.run():745 >> >> >> at >>org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResu >>l >>t >>Handler.java:118) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at >>org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:1 >>1 >>1 >>) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at >>org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWit >>h >>C >>onnection.java:47) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at >>org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWit >>h >>C >>onnection.java:32) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) >>~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at >>org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) >>~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at >>org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) >>~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0] >> at >>io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessa >>g >>e >>Decoder.java:89) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstract >>C >>h >>annelHandlerContext.java:339) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCh >>a >>n >>nelHandlerContext.java:324) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.ja >>v >>a >>:254) ~[netty-handler-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstract >>C >>h >>annelHandlerContext.java:339) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCh >>a >>n >>nelHandlerContext.java:324) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessa >>g >>e >>Decoder.java:103) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstract >>C >>h >>annelHandlerContext.java:339) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCh >>a >>n >>nelHandlerContext.java:324) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDeco >>d >>e >>r.java:242) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstract >>C >>h >>annelHandlerContext.java:339) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCh >>a >>n >>nelHandlerContext.java:324) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundH >>a >>n >>dlerAdapter.java:86) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstract >>C >>h >>annelHandlerContext.java:339) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractCh >>a >>n >>nelHandlerContext.java:324) >>~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPip >>e >>l >>ine.java:847) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] >> at >>io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epoll >>I >>n >>Ready(AbstractEpollStreamChannel.java:618) >>~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] >> at >>io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:32 >>9 >>) >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] >> at >>io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250) >>~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] >> at >>io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEven >>t >>E >>xecutor.java:111) ~[netty-common-4.0.27.Final.jar:4.0.27.Final] >> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_71] >> >> >> >> >> >> >> >> >>On 6/18/15, 11:28 AM, "Ted Dunning" <[email protected]> wrote: >> >>>On Thu, Jun 18, 2015 at 8:24 AM, Paul Mogren <[email protected]> >>>wrote: >>> >>>> Following up. Ted gave sound advice regarding reading S3 vs HDFS, but >>>> didn¹t address EMRFS specifically. Here is what I have learned. >>>> >>> >>>Great summary. Very useful when people help by feeding back what they >>>have >>>learned. >> >
