zoid-w commented on issue #31795:
URL: https://github.com/apache/airflow/issues/31795#issuecomment-1592964558
Hi, I have done some testing with #31803. We can see the following behavior.
I used a test setup that is expected to generate 25 events from 40 messages
(so only 25/40 messages are eligible ones).
## Logs :
```
[2023-06-15, 13:56:37 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 129)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:56:37 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 129)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:57:23 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 130)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:58:11 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 131)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:58:56 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 132)
fired: TriggerEvent<'BAR'>
[2023-06-15, 13:58:56 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 132)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:58:56 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 132)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:59:41 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 133)
fired: TriggerEvent<'FOO'>
[2023-06-15, 13:59:41 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 133)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:00:31 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 134)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:01:22 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 135)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:02:13 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 136)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:02:13 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 136)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:02:13 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 136)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:02:13 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 136)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:02:13 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 136)
fired: TriggerEvent<'FOO'>
[2023-06-15, 14:02:13 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 136)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:02:59 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 137)
fired: TriggerEvent<'BAR'>
[2023-06-15, 14:02:59 CEST] {triggerer_job_runner.py:608} INFO - Trigger
***/manual__2023-06-15T11:56:14.094686+00:00/listen_for_message/-1/1 (ID 137)
fired: TriggerEvent<'FOO'>
```
What we can see is that only 9 TriggerEvents are created (based upon ID). In
this case you event_triggered_function will therefore only be called 9 times
instead of 25. From the timestamp I suspect this issue is occuring since the
TriggerEvents are generated at exactly the same time, yielding mutiple
TriggerEvents with the same ID since the TriggerEvent ID has not yet been
increased.
From the discussion in #31703 I am unsure what the correct approach here
would be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
