Hey Michael, Have you copied over the core-site.xml file onto Drill's conf directory? You could also set the credentials directly in the s3 storage plugin [1].
[1] https://drill.apache.org/docs/s3-storage-plugin/ -Abhishek On Tue, Apr 18, 2017 at 4:17 PM Knapp, Michael <[email protected]> wrote: > Drill Developers, > > I have been struggling to change the aws credentials when running drill. > I am using session tokens in a local profile. I also run the app using IAM > roles, but right now I just want it to work locally. This is in my > core-site.xml: > > > <property> > <key>fs.s3a.aws.credentials.provider</key> > > <value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value> > </property> > <property> > <key>fs.s3a.security.credential.provider.path</key> > <value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value> > </property> > <property> > <key>hadoop.security.credential.provider.path</key> > <value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value> > </property> > > > but unfortunately when I run the application (with several loggers set to > trace) I still see this: > 2017-04-18 23:07:31,725 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > TRACE o.a.d.exec.util.ImpersonationUtil - Creating DrillFileSystem for > proxy user: drill (auth:SIMPLE) > 2017-04-18 23:07:31,986 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials > from BasicAWSCredentialsProvider: Access key or secret key is null > 2017-04-18 23:07:34,003 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials > from InstanceProfileCredentialsProvider: Unable to load credentials from > Amazon EC2 metadata service > 2017-04-18 23:07:34,030 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > DEBUG o.a.drill.exec.work.foreman.Foreman - > 270963cb-cb2a-aa04-3ad1-1a92384a31f5: State change requested STARTING --> > FAILED > org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception > during fragment initialization: Unable to load AWS credentials from any > provider in the chain > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:298) > [drill-java-exec-1.10.0.jar:1.10.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > Caused by: com.amazonaws.AmazonClientException: Unable to load AWS > credentials from any provider in the chain > > > The application is only checking BasicAWDCredentialsProvider and > InstanceProfileCredentialsProvider. Neither of those will work for me. I > have been searching the source code for a while now and none of the > properties I try to set seem to actually work. > > Would somebody please tell me how to configure the credentials provider > chain in drill? > > Michael Knapp > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
