mancunian1792 opened a new issue #22628: URL: https://github.com/apache/airflow/issues/22628
### Description **Problem**: There can be days where this queue doesn't get any message. When that happens, my SQS Sensor continues to poll till it receives a message. Since i want it to make as real-time as possible, this prevents me from choosing a logical schedule interval for the DAG. **Question**: How would you implement for cases where there may be a mix of lots of messages in a day and no message at all? People who post messages may work in short bursts (difficult to predict when they'd do that.). What's the ideal scenario for anyone to use an SQS sensor? ### Use case/motivation **My Use case**: Pick up a message from SQS and run a set of tasks. (Lots of data transformations and integrations happen here that it makes sense to use airflow and not dump everything in a script.) **Suggestions**: If there is a max_try parameter to the sensor my whole issue would be solved. Poll 'x' (parameterized. default can be infinite till you get a message.) number of times and if you don't receive a message by then, then write an empty array to xcom for key messages. This way, one can control the run time the dag/this particular task and hence control the schedule interval to work to their needs. Although, changes the very definition of polling but for airflow's case, a max_try parameter kinda makes sense to me ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
