LaPetiteSouris opened a new pull request #21159:
URL: https://github.com/apache/airflow/pull/21159


   ## What
   
   Add opt-in in `SQSSensor` to disable the behavior of automatically delete 
messages on consumption. This option is opt-in. By default it is set to 
`False`, which means the original feature is preserved: 
   
   `SQS messages are deleted automatically from the queue upon consumption`
   
   ## Why
   
   In many cases, the `SQSSensor` itselft is not the part of the code base that 
actually processes the messages. It serves as a short of polling to get 
messages out for further processing.
   
   If the actually code/Aiflow task that processes the messages failed, the 
message should be put back in the queue for further processing. This is the 
very basic idea of SQS Polling provided by Amazon.
   
   Reference to SQS with AWS Lambda function, [AWS 
docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) states :
   
   ```
   When your function *successfully* processes a batch, Lambda deletes its 
messages from the queue.
   ```
   
   In many case, the `SQSSensor` polls the message and distributes the workload 
to different task, in a very similar way to AWS Lambda polling.
   
   
![SQSPolling](https://user-images.githubusercontent.com/6369285/151395485-f277165b-7ec2-42ca-880f-6cd74879c1d6.png)
   
   
   Thus, there should be option to disable to default behavior, which is to 
delete the message upon reception.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to