[
https://issues.apache.org/jira/browse/HADOOP-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474791#comment-17474791
]
Steve Loughran commented on HADOOP-18078:
-----------------------------------------
While I don't deny we may have broken things in 3.3.1 (especially the aws
region bug) the fact the first two operations work but not the third imply
something is happening through different spark codepaths.
You going to have to begin debugging it there. And when I say "you", I'm afraid
I do mean *you* as it works for me. Sorry.
I'd recommend switching to scala to avoid one layer of complication... Once you
get an example which fails there, ideally in spark standalone, then turn
org.apache.hadoop.fs.s3a logging to get more info as to what is going on.
Meanwhile, a full stack would be informative.
Incidentally, when launched, spark submit will pick up the AWS_ env var values
and set s3a options from them, even if you've explicitly set them in your
config. so make sure they are unset before you submit your work.
> TemporaryAWSCredentialsProvider has no credentials
> --------------------------------------------------
>
> Key: HADOOP-18078
> URL: https://issues.apache.org/jira/browse/HADOOP-18078
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.1
> Environment: python:3.9.5
> openjdk:11.0.13
> spark:3.2.0
> hadoop:3.3.1
> Reporter: Björn Boschman
> Priority: Major
> Attachments: spark_test.py
>
>
> Not quite sure how to phrase this bugreport but I'll give it a try..
> We are using a SparkSession to access parquet files on AWS/S3
> it is ok, if there is only one s3a URL supplied
> it used to be ok if there is a bunch of s3a URLs - that's broken siince
> hadoop:3.3.1
>
>
> I've attached a sample script - yet it relys on spark+hadoop installed
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
