ramitkataria commented on PR #47035: URL: https://github.com/apache/airflow/pull/47035#issuecomment-2691912417
After discussion with @o-nikolas, I ended up removing the storage of sessions in hook instances because they would get re-instantiated again anyway. The only feasible way seems to be to serialize, encrypt and store them in the file system, which can be an improvement implemented in the future. I was going to implement retries using tenacity so that the hook would wait instead of failing in case of throttling but I did some performance testing to see the how it would handle a throttling scenario and it looks like boto waits instead of returning an error response so that shouldn't happen. Either way, if someone does manage to get errors related to throttling and wants the hook to retry instead of fail, boto has [this guide about setting up retries](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html) which seems like the best way to go about this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
