potiuk commented on issue #32666: URL: https://github.com/apache/airflow/issues/32666#issuecomment-1640459315
Not possible without completely rewriting hiw scheduler works. We heavily rely on lock mechanisms in relation databases. That does not work well with active/active setup And it is not needed IMHO. Generally speaking Airflow is not targeting 99.9% available system(8 hours a year). It is far less availability as target. And this is by design. Airflow should not be used as a backbone of system that requires it. It is a batch scheduling system. Having minutes availability for Airflow is NON-goal for us. Every single 9 there order of magnitude complexity and cost of development and requires usually much more complex maintrnance. If someone wills to invest 10x the time that has been used to develop current scheduler - maybe they should. But we never saw it as a goal. That's why we rejected things like zookeeper, non-relational databases and supporting active-active setup in the past Supporting such setup without real need is at most following a 'trend' without really considering if it is worth the cost. You have to remember that every single decision of architecture bears a cost. And in this case - the cost is huge. If someone would like to spend time and write Airflow Improvement Proposal and convince others and bear the cost of implementing it - yes sure, anyone is free to start it. But considering cost of implementation, deployment and deeply considering what it means to run Airflow in such a way, reasoning why it is necessary will be a crucial part of it. And convincing community members that it is worth it is the first step. For now I personally am not convinced. But luckily it's not me to convince. Bring it to devlist, state your reasoning, fend off people who will not like it, reach consensus, get the AIP to pass the vote and implement it. This is the way how one should approach it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
