bolkedebruin commented on PR #28333: URL: https://github.com/apache/airflow/pull/28333#issuecomment-1357491045
I do not think Iam sharing your feelings. I have already explained one use case above and this was actually based on a real world request. A DAG is typically seen a a "unit-of-work", e.g. generating a Dataset. If it is required to split this unit of work because the DSL isnt expressive enough I don't think that is the right way. Therefore, I disagree with @potiuk that what is described in https://github.com/apache/airflow/discussions/28253#discussioncomment-4376780 is actually simple. The example I gave above would mix event driven with a pull based mechanism (sensor) that is not logical neither intuitive. Next that, the approach described is inefficient. However, even if we would accept the above stance that simplicity on the user side is preferred for now and at the same time we would like to keep our options for the future open, I think your current implementation is not forward compatible. If applying the thinking of https://github.com/apache/airflow/pull/28455 all code here would need to be changed, the dag serialization schema would need to be deprecated and changed and database would need to be migrated. I do not think that is the right path. I do think that what you would like to achieve is quite easily done with the approach I put into #28455 and I would argue it is the same amount of work. In this case it would just be adding the filtering logic. If we then decide to add additional Rules in the future this can be done without changing any of the scheduler code, database and schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
