ahmarsuhail commented on PR #5421: URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1479763570
I have some concerns for about this region first approach for third party stores as well. The region probe is not going to work. I don't know how regions actually work for third party stores, does the value of the region matter? If no, set region to any valid region, otherwise third party store users must always set `fs.s3a.endpoint.region`. This kind of ties into a previous discussion, about detecting if the store is AWS S3 or third party. I looked into it, and it's difficult to infer if a store is AWS S3 or not reliably from the existing config values. Was thinking we could do something like `Endpoint is not set || Endpoint is set && ends in amazonaws.com || ARN is set ` but pretty sure this will miss edge cases, and will be hard to maintain with new features etc. Maybe can add in a new config, something like `fs.s3a.is.AWS` instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
