potiuk edited a comment on pull request #19361: URL: https://github.com/apache/airflow/pull/19361#issuecomment-957490564
I will be able to take a closer look later this week. I have not yet looked at it, but it seems that this feature is something that we need a very good look at adn make sure we understand the consequences of it especially when we consider future ideas of being able to make Airflow more 'task-oriented" than "Dag-oriented" which we know is something we want to work on. Again - I have not looked at it so it is hard to comment yet and I do not know the scope and consequences of that, But I generally see three options we could treat this kind of change (since it was not discussed in an AIP/devlist before: a) if it will fit into the long term idea and is an incremental change without foreesen big impact on target architecture we "envision" (even if it is not specified yet) we could iterate on it and merge it as "tactical" solution for task scheduling b) if it will be risky - in terms of blocking some ideas and directins - we should discuss those and see if the ideas are viable/whether we can turn it into a). If we cannot and the risk is too big (and we foresee to work on more "task-based" scheduling soon then this PR might be good one to discuss stuff and eventull go to solution c) c) we could start discussing more "fully featured" solution for task-based scheduling and start discussin in the devlist and start drafting AIP - in which case this PR might serve as a good "starting point" for discussion - either as an example of things we want to avoid and implement differently or showing where we still need to get missing puzzles to fit in order to know that the direction we take is good. I have no idea whatsoever which of the options above it can fit in - it can be either, but I think it's good to agree and let @malthe know soon from the committers who would be interested, which direction is most likely here (I am at my hometown today and coming back tomorrow - so i will take a look some time later this week) - but also I'd love to hear what others who thought about the "task-based airflow" to state their opinions here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
