Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21510 )
Change subject: IMPALA-13156: Investigation: Set explicit credential provider for S3 builds ...................................................................... IMPALA-13156: Investigation: Set explicit credential provider for S3 builds Lately we have seen several failures during S3 builds that relied on AWS EC2 IAM instances roles for S3 bucket access credentials. The failure mode was a spurious failure for the s3a IAM Instance Credential Provider to actually provide the credentials. This patch is an attempt to extract more debugging information from such a failure: according to Hadoop-AWS developers, "Unable to load credentials from system settings..." is a generic error message from the Hadoop credential providers when they operate in a chained fashion. This happens when there is no explicit credetial provider specified in core-site.xml, and the credential providers are tried in sequence. The patch specifies the Hadoop s3a IAM Instance Credential Provider when all of the following conditions are true: - the default file system is S3, - the minicluster is running on an AWS EC2 VM, - AWS credentials are provided by an IAM instance role attached to the VM. The conditions are detected by following the rule matrix set up in bin/impala-config.sh; the same rule set is evaluated in core-site.xml.py, the Python script that generates the working copy of core-site.xml. The patch also XFAILs test_keys_do_not_work() in test_s3_access.py in custom cluster tests, because the test assumes the default chain of Hadoop AWS credential providers being available. This patch restricts the credential provider set to the IAM Instance Credential Provider only, which breaks the test mechanism. The test will be reinstated when a suitable workaround mechanism is found, but at this stage the extended diagnostic provided by this patch for investigating the credential provider flakiness is considered more valuable. Tests: - private build on S3 for the positive case - regression tested on a HDFS private build - in both cases the generated core-site.xml file was inspected during the test run. Change-Id: Ia8c09f8d042a69c5d3227398c720ea38e1c7e12f Reviewed-on: http://gerrit.cloudera.org:8080/21510 Reviewed-by: Michael Smith <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/custom_cluster/test_s3a_access.py 2 files changed, 14 insertions(+), 0 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia8c09f8d042a69c5d3227398c720ea38e1c7e12f Gerrit-Change-Number: 21510 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
