gpoulin opened a new issue, #23760: URL: https://github.com/apache/airflow/issues/23760
### Apache Airflow Provider(s) google ### Versions of Apache Airflow Providers apache-airflow-providers-google 7.0.0 ### Apache Airflow version 2.3.0 (latest released) ### Operating System GKE container-optimize OS ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### What happened When using `GCSObjectsWithPrefixExistenceSensor` on a bucket with a lot of object, all the blob name matching the prefix will be loaded in-memory which can lead of OOM. In majority of cases having the list of all the blob name is not necessary. ### What you think should happen instead There should be a way to limit the number of blob name loaded. ### How to reproduce _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
