kristopherkane opened a new issue, #32499: URL: https://github.com/apache/airflow/issues/32499
### Description The Python Google Cloud Dataproc API version has been updated in Airflow to support filtering on the Dataproc Batches API. The DataprocListBatchesOperator can be updated to make use of this filtering. Currently, DataprocListBatchesOperator returns, and populates xcom with every job run in the project. This almost surely will fail as the return object is large and xcom storage is low, especially with MySQL. Filtering on job prefix and create_time are much more useful capabilities. ### Use case/motivation The ability to filter lists of GCP Dataproc Batches jobs. ### Related issues None ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
