potiuk commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897519700
So we went very far from throwing exception in a method :). I think we are talking about two different things and we should split this diacusion off to a short term generic simple solution where you can throw an exception from 'a' method from a long term one where we can control task schedule within dags separately from the Dag schedule (which on it's own is an interesting long term feature but narrowing it down to 'declaratively specified skip condition's is really bad idea how to start thinking about it. AIP-39 already exists so whatever we come up with needs to include it. It introduces more complex scheduling patterns than Cron and whatever we do about scheduling tasks needs to be built on top, otherwise it is a regression. Also what if we want to also fail, or succeed the tasks on some different schedule ? Do we want to add more declarative conditions for those in the future? How about adding different dependencies between tasks based ona different schedule ( i think this falls in the same camp)? how does it play with not-approvwt yet Dag versioning ? . There are many more questions than answers and going in this direction without answering those questions is not really future -proof )or even future-aware). I think - since we have a simple, generic exception based approach that can be implemented quickly without breaking any of the existing properties of scheduling and execution - we should go for it now. When it comes to separating task schedule/dependencies from Dag schedule - that requires far more thinking and discussions and implementing just a very small subset of it on the way that might be incompatible with future approach we take is not a good idea (especially that we can solve the basic use case now pretty easily with configurable, synchronous pre-execute run during task execution). We should not sacrifice and ignore long term vision of the product with short term goal of serving a specific use case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
