jscheffl commented on PR #61646: URL: https://github.com/apache/airflow/pull/61646#issuecomment-3867974040
Not sure whether AI concluded this: > Specifically, the maintenance operations (_purge_jobs, _update_orphaned_jobs, _check_worker_liveness) were previously operating on all rows in these tables indiscriminately. In a multi-team setup where each team may have different configuration values, this could lead to one team's executor incorrectly purging another team's jobs or marking another team's workers as dead. This is WRONG. jobs are only purged on completion. Not while being in the queue. So should not be a problem. I would doubt that different configurations per team are needed. I'd assume that in case Edge is needed that workers are started per team. Whereas central tables are shared. Shall we make the PR "draft" until questions are clear? Regarding the questions: 1. Are queues guaranteed to be unique across teams? For example, could team_1 and team_2 both use a queue named "default"? If so, the current queue-based isolation would break down. --> Not an expert in the multi team setup ... @o-nikolas can you provide an answer? 2. If queue uniqueness across teams is intended, is there a mechanism to enforce it? The queue parameter is set by users at the DAG/task level, so there is nothing currently preventing two teams from accidentally (or intentionally) using the same queue name. --> same answer bucket. 3. Alternatively, would it be more appropriate to add a team_name column to EdgeJobModel / EdgeWorkerModel for explicit team-level filtering, rather than relying on queue-based inference? --> without knowing the answers from above... I assume so. Which would be a blocker until the DB migration tooling in integrated in Edge which is still pending from https://github.com/apache/airflow/pull/61155 - before this not merged we should not extend the DB scheme. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
