Carl Boettiger created ARROW-15127:
--------------------------------------
Summary: More visible documentation of
AWS_EC2_METADATA_DISABLED=TRUE
Key: ARROW-15127
URL: https://issues.apache.org/jira/browse/ARROW-15127
Project: Apache Arrow
Issue Type: Improvement
Components: R
Affects Versions: 6.0.1
Reporter: Carl Boettiger
S3Filesystem operations can fail with opaque curl errors that are hard to debug
from R, as I could see no way of accessing what URL or header information was
being sent even when we enter the R debugger. We are left with relatively
opaque errors such as:
{code:java}
Error: IOError: When resolving region for bucket 'analysis': AWS Error [code
99]: curlCode: 6, Couldn't resolve host name {code}
which could have a whole host of underlying causes, from incorrect URL
construction (e.g. incorrectly specified endpoint_override, default_region,
etc) to other network issues https certificate failure or network connectivity
problems.
In any event, a common source of this seems to be collisions with AWS
credentials in environmental variables or elsewhere interfering with the
intended credentials (which may be fine with 'default' values – e.g. like when
accessing a MINIO bucket we do not need a "region" (actually it's not needed in
AWS as it can be attained from the endpoint; or at least that is the argument
given by MINIO).
In a recent case, I'd already made sure I didn't have any AWS_DEFAULT_REGION or
AWS_S3_ENDPOINT env var set, but still kept getting this error on a MINIO
bucket on one machine , but not others.
Neal pointed out to me the existence of `AWS_EC2_METADATA_DISABLED`, and
indeed, setting that to TRUE resolved the issue. Others may also be unaware of
this option, it might be good to highlight in the S3Filesystem vignette and/or
docs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
