jroachgolf84 opened a new pull request, #55070: URL: https://github.com/apache/airflow/pull/55070
This PR aims to reduce the memory footprint for the `S3KeySensor`. This was done by altering the `get_file_metadata` method in the S3Hook to `yield` records in the paginated response, rather than loading them into a single list. The return type for the `get_file_metadata` method is not an `Iterator`. An assertion was added to validate this, and all appropriate tests were updated. closes: #55039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
