stevenmanton opened a new issue, #36416:
URL: https://github.com/apache/arrow/issues/36416

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   I use `pyarrow.dataset.dataset` often to load data from S3. However, it's 
always been a bit finicky regarding named AWS profiles. For example, let's say 
I have the following `~/.aws/credentials` file that defines two identical 
profiles and uses an external process to get credentials (which shouldn't 
matter, but I'm adding it for completeness):
   
   ```
   [default]
   region = us-east-1
   credential_process = /path/to/get-creds.sh
   
   [dev]
   region = us-east-1
   credential_process = /path/to/get-creds.sh
   ```
   
   These commands both work, which validates that both accounts have access to 
S3:
   ```
   aws s3 ls s3://bucket/path --profile default
   aws s3 ls s3://bucket/path --profile dev
   ```
   
   Now, as I understand it, the only way to specify a named AWS profile with 
pyarrow is to use environment variables. To test the access, I use a trivial 
script:
   
   ```python
   # script.py
   import pyarrow.dataset as ds
   
   dataset = ds.dataset("s3://bucket/path/")
   ```
   
   Now, here's the strangeness:
   ```bash
   # Check that no environment variables are set in this shell:
   env | grep AWS
   
   # This works:
   python ./script.py
   # So does this:
   AWS_PROFILE=default python ./script.py
   # But this fails:
   AWS_PROFILE=dev python ./script.py
   ```
   
   The error is:
   ```
   Traceback (most recent call last):
     File "./script.py", line 3, in <module>
       dataset = ds.dataset("s3://bucket/path/")
     File 
"/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py",
 line 763, in dataset
       return _filesystem_dataset(source, **kwargs)
     File 
"/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py",
 line 446, in _filesystem_dataset
       fs, paths_or_selector = _ensure_single_source(source, filesystem)
     File 
"/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py",
 line 413, in _ensure_single_source
       file_info = filesystem.get_file_info(path)
     File "pyarrow/_fs.pyx", line 571, in pyarrow._fs.FileSystem.get_file_info
     File "pyarrow/error.pxi", line 144, in 
pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
   OSError: When getting information for key 'path' in bucket 'bucket': AWS 
Error ACCESS_DENIED during HeadObject operation: No response body.
   ```
   
   I'm really quite at a loss for how this could happen. And since the 
underlying code isn't python, it seems tricky for me to help debug. It's 
possible that I'm doing something dumb, but I really can't imagine why two 
identical profiles would give different results like this.
   
   The pyarrow version I'm using is `12.0.1`.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to