LaPetiteSouris commented on issue #25640: URL: https://github.com/apache/airflow/issues/25640#issuecomment-1212003713
@potiuk I am not sure if this is the right place to put it, but here is the deal: https://github.com/apache/airflow/pull/24554 added `num_batches` parameter to `SQSSensor` in Amazon provider. You did ask me as a contributor of the feature to test it for AWS provider [amazon: 4.1.0rc1](https://pypi.org/project/apache-airflow-providers-amazon/4.1.0rc1) in https://github.com/apache/airflow/issues/25037#event-7006630694 Unfortunately I did not have time to do so in time for the release. Taking advantage of this 5.0.0 release for Amazon Providers, I tested the feature. ### What have been tested Given `SQSSensor` without `num_batches` and the same sensor with `num_batches`, ensure that: - Both sensors can get messages normally from SQS - SQSSensor with `num_batches` actually get batches of messages instead of a single poke - Version of AWS provider  ### Result 1. The sensor ```python read_from_queue = SqsSensor( aws_conn_id="aws_sqs_test", task_id="read_from_queue", sqs_queue=sqs_queue, ) # Retrieve multiple batches of messages from SQS. # The SQS API only returns a maximum of 10 messages per poll. read_from_queue_in_batch = SqsSensor( aws_conn_id="aws_sqs_test", task_id="read_from_queue_in_batch", sqs_queue=sqs_queue, # Get maximum 10 messages each poll max_messages=3, # Combine 3 polls before returning results num_batches=3, ) ``` 2. The result of task execution (success)  3. The result of `SQSSensor` without `num_batches` enabled. Only a few messages are available in `xcom`  4. The result of `SQSSensor` with `num_batches=3` and `max_messages=3`. It does get `3 x 3 = 9` messages for each execution.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
