dstandish commented on a change in pull request #19546:
URL: https://github.com/apache/airflow/pull/19546#discussion_r750801976
##########
File path: airflow/jobs/triggerer_job.py
##########
@@ -315,6 +339,7 @@ async def cleanup_finished_triggers(self):
details["name"],
)
self.failed_triggers.append(trigger_id)
+ self.completed_triggers.add(trigger_id)
Review comment:
how do you envision it growing? the items should only be in
`completed_triggers` for basically one or two loop cycles. it goes
`submit_event` (into completed) -> `Trigger.clean_unused` (purged from db) ->
`purge_completed_triggers_list` (purged from completed)
but maybe you're saying that in the wild the code may not behave as
intended, and triggers will accumulate in the DB, and therefore they'll
accumulate in the `completed_triggers` set? but if they accumulate in the db
they'll keep getting recreated and eventually that would be an issue in itself
because not only would they exist as ids in a set but they would be running.
using a TTL approach as you have suggested would avoid a db query though.
LMK your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
