LaPetiteSouris commented on code in PR #24554:
URL: https://github.com/apache/airflow/pull/24554#discussion_r903448436
##########
airflow/providers/amazon/aws/sensors/sqs.py:
##########
@@ -215,3 +229,68 @@ def __init__(self, *args, **kwargs):
stacklevel=2,
)
super().__init__(*args, **kwargs)
+
+
+class SqsBatchSensor(SqsSensor):
+ """
+ Get messages from an Amazon SQS queue in batches and then delete the
retrieved messages from the queue.
+ If deletion of messages fails an AirflowException is thrown. Otherwise,
all messages
+ are pushed through XCom with the key ``messages``.
+
+ This sensor is identical to SqsSensor, except that SqsSensor performs one
and only one SQS call
+ per poke, which limits the result to a maximum of 10 messages, while
SqsBatchSensor performs multiple
+ SQS API calls per poke and combines the results into one list.
+
+ .. seealso::
+ For more information on how to use this sensor, take a look at the
guide:
+ :ref:`howto/sensor:SqsBatchSensor`
+
+ :param num_batches: The number of times the sensor will call the SQS API
to receive messages (default: 1)
Review Comment:
@eladkal from what I know and what we have tested in a Production
environment for few months( we have about 100-200 messages per second on SQS
queue ), there should be no impact.
My concern was more of a feeling than factual (aka: I hesitate to say what I
don't know).
However, from factual testing ourselves, we don't see any backward non
compatible impact.
So let's go with integrating `num_batches` in `SqsSensor` then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
