After spending a day or two, I found the solution to the issue. We need to add the following snippet in the core-site.xml
<property> <name>fs.s3a.aws.credentials.provider</name> <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value> </property> This explicitly tells drill to use that mode of fetching credentials instead of relying on env variables or explicit access keys. I would recommend adding it to the official document so that other folks can benefit from this. If there is a way to do it myself, let me know (I can raise a pull request etc too) Thanks, Nitin On Mon, Sep 17, 2018 at 5:24 PM Nitin S <[email protected]> wrote: > Hi , > > I did try that before and it did not work. I was not able to find it in > the documentation > > Thanks > Nitin > > On Mon, Sep 17, 2018 at 3:37 PM Khurram Faraaz <[email protected]> wrote: > >> Hi Nitin, >> >> You can read the details from here - >> http://drill-user.incubator.apache.narkive.com/mP5GE7fq/s3-using-iam-roles >> >> A drill user hit the same issue in the past and he has documented the >> steps >> for others to benefit. Hope that helps. >> >> Regards, >> Khurram >> >> On Mon, Sep 17, 2018 at 2:20 PM, Nitin S <[email protected]> wrote: >> >> > Hello, >> > >> > I am trying to setup drill in distributed mode and access some parquet >> > files in S3. The instructions here: >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__drill. >> > apache.org_docs_s3-2Dstorage-2Dplugin_&d=DwIBaQ&c= >> > cskdkSMqhcnjZxdQVpwTXg&r=H5JEl9vb-mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI&m= >> > I3BfnYvwCJajF5EHuHVLb6Pzlb1C_56Em6a6kgxv1Ac&s= >> > hQMUN2MEoN9AGwiPvEjbZR7mnh37WG9x2XWhPPm3ky8&e= do not provide any info >> > on what to do if we are using IAM roles for S3 (instead of static keys) >> > >> > Kindly advise how to set that up. >> > >> > Thanks, >> > Nitin >> > >> >
