jordanjeremy commented on PR #24401: URL: https://github.com/apache/airflow/pull/24401#issuecomment-1154461709
Hello, I wanted to try to provide some additional context on why I believe that the TaskFlow API should allow a downstream step to try to run even if an upstream didn't set an XCom value (due to things like skip/fail exceptions). In addition to passing the values between steps, when a XComArg is passed as an argument to a downstream step Airflow automatically figures out the dependencies between those steps without this having to be explicitly set when the dag is written, which makes that process easier. As described in #24338 we are getting 2 files which may or may not arrive at the same time. If either file is received we need to run another dag prior to processing the file(s) that were received. So the XComArg from both get file tasks are passed to the "run_separate_dag" task (which has the trigger_rule = none_failed_min_one_success). The trigger dag task doesn't directly use the values but passing the XComArg values establishes the dependency between these 3 tasks. If one or both of the files are received, then the other dag gets triggered. If neither was received then the trigger dag task is set to skip as well based on the set trigger_rule. After this other dag is run we have steps that would process the files which were received or skip if the associated file was not received. So a more complete realistic example is:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
