viirya commented on pull request #35994: URL: https://github.com/apache/spark/pull/35994#issuecomment-1082604486
I cannot give the exact stack trace here, but the exceptions are `AWSClientIOException`. Thanks for the hint. I re-checked the stack trace with the S3AFileSystem codebase, seems it is already retried. I don't think this would be at FS API layer as different FS implementations might have different way, some might not be able to retry. Personally I think a config at Spark is much easier to follow up from an end-user perspective. It is harder to follow up with different FS implementation cases as they might have different way to configure the behavior. Users will also look for Spark configs, instead of configs at underlying libraries. As for this case (S3A FS), seems it is already retried and so there is no urgent requirement to add this config from my side now, I'm fine to close it if you still think this should not be added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
