topherinternational commented on issue #54340: URL: https://github.com/apache/airflow/issues/54340#issuecomment-3218386625
> It's a "better" approach to raise AirflowSkipException instead? I have no opinion on the desired behavior here (have never used branch-return-falsy to effect a skip) but I wanted to note that the `SkipMixin` (i.e. the code containing Branch's and ShortCircuit's skipping logic) and raising an `AirflowSkipException` do _not_ behave the same way and can have big semantic differences. I'll explain `ShortCircuitOperator` and then show my example that `BranchPythonOperator` behaves the same way: - If a `ShortCircuitOperator` callable returns a falsy value, the short-circuit task is set to Success and appears green in the UI graph and grid; all downstream tasks are set to Skipped and appear as pink/rose - If a `ShortCircuitOperator` callable raises `AirflowSkipException`, the short-circuit task _itself_ is set to Skipped and appears as pink The obvious semantic distinction is "this task finished its work, and its output indicates we should stop running more tasks" vs "this task hit something that means it had to stop running, but didn't rise to the level of an error, and further tasks will also not be run" In my org we use ShortCircuit tasks to perform low-cost checks as to whether heavy data processing should be done downstream (e.g., if there's no new data, don't rerun an expensive aggregation), typical of most Airflow users' use cases I imagine. Whether the answer is yes or no, the tasks will appear as green in the UI unless an error occurs during the check itself. We have coached our data engineers to understand that green means the check was executed, and pink downstream means the check determined further work was not needed. So, anybody who wants their old BranchPythonOperator behavior can't simply switch to raising `AirflowSkipException` as the task states and UI display will be different. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
