eladkal commented on a change in pull request #17421:
URL: https://github.com/apache/airflow/pull/17421#discussion_r758703752
##########
File path: airflow/operators/python.py
##########
@@ -230,17 +230,35 @@ def execute(self, context: Dict):
class ShortCircuitOperator(PythonOperator, SkipMixin):
"""
- Allows a workflow to continue only if a condition is met. Otherwise, the
- workflow "short-circuits" and downstream tasks are skipped.
+ Allows a workflow to continue only if a condition is met. Otherwise, the
workflow "short-circuits" and
+ downstream tasks are skipped. The short-circuiting can be configured to
either respect or ignore the
+ ``trigger_rule`` set for downstream tasks. If
``ignore_downstream_trigger_rules`` is set to True, the
+ default setting, all downstream tasks are skipped without considering the
``trigger_rule`` defined for
+ tasks. However, if this parameter is set to False, the direct, downstream
tasks are skipped but the
Review comment:
```suggestion
tasks. However, if this parameter is set to False, the direct downstream
tasks are skipped but the
```
##########
File path: airflow/operators/python.py
##########
@@ -230,17 +230,35 @@ def execute(self, context: Dict):
class ShortCircuitOperator(PythonOperator, SkipMixin):
"""
- Allows a workflow to continue only if a condition is met. Otherwise, the
- workflow "short-circuits" and downstream tasks are skipped.
+ Allows a workflow to continue only if a condition is met. Otherwise, the
workflow "short-circuits" and
+ downstream tasks are skipped. The short-circuiting can be configured to
either respect or ignore the
+ ``trigger_rule`` set for downstream tasks. If
``ignore_downstream_trigger_rules`` is set to True, the
+ default setting, all downstream tasks are skipped without considering the
``trigger_rule`` defined for
+ tasks. However, if this parameter is set to False, the direct, downstream
tasks are skipped but the
+ specified ``trigger_rule`` for other subsequent downstream tasks are
respected. In this mode,
+ the operator assumes the direct, downstream tasks were purposely meant to
be skipped but perhaps
+ not other subsequent tasks.
- The ShortCircuitOperator is derived from the PythonOperator. It evaluates a
- condition and short-circuits the workflow if the condition is False. Any
- downstream tasks are marked with a state of "skipped". If the condition is
- True, downstream tasks proceed as normal.
+ The ShortCircuitOperator is derived from the PythonOperator. It evaluates
a condition and short-circuits
+ the workflow if the condition is False. Any downstream tasks are marked
with a state of "skipped" based
+ on the short-circuiting mode configured. If the condition is True,
downstream tasks proceed as normal.
Review comment:
I think the docstring should start with that paragraph (as this explains
what the operator does while the prev paragraph explain about specific
parameter)
Also it seems like there is repeated explanation between this paragraph and
the previous one.
##########
File path: docs/apache-airflow/howto/operator/python.rst
##########
@@ -80,7 +80,54 @@ Otherwise you won't have access to the most context
variables of Airflow in ``op
If you want the context related to datetime objects like
``data_interval_start`` you can add ``pendulum`` and
``lazy_object_proxy``.
+
+
+.. _howto/operator:ShortCircuitOperator:
+
+ShortCircuitOperator
+========================
+
+Use the :class:`~airflow.operators.python.ShortCircuitOperator` to control
whether a pipeline continues only
+if a condition is satisfied. The evaluation of this condition is done via the
output of a ``python_callable``.
+If the ``python_callable`` returns True, the condition is considered satisfied
and the pipeline is allowed to
+continue. In the example below, the tasks that follow the "condition_is_True"
ShortCircuitOperator will
+execute while the tasks downstream of the "condition_is_False"
ShortCircuitOperator will be skipped.
+
+
+.. exampleinclude::
/../../airflow/example_dags/example_short_circuit_operator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_short_circuit]
+ :end-before: [END howto_operator_short_circuit]
+
+
+The "short-circuiting" can be configured to either respect or ignore the
``trigger_rule`` defined
+for downstream tasks. If ``ignore_downstream_trigger_rules`` is set to True,
the default configuration, all
+downstream tasks are skipped without considering the ``trigger_rule`` defined
for tasks. If this parameter is
+set to False, the direct, downstream tasks are skipped but the specified
``trigger_rule`` for other subsequent
+downstream tasks are respected. In this mode, the operator assumes the direct,
downstream task(s) were
+purposely meant to be skipped but perhaps not other subsequent tasks.
+
+In the example below, notice that the ShortCircuitOperator task is configured
to respect downstream trigger
+rules. This means while the tasks that follow the "short_circuit"
ShortCircuitOperator task will be skipped
+since the ``python_callable`` returns False, "task_7" will still execute as
its set to execute when upstream
+tasks have completed running regardless of status.
+
+.. exampleinclude::
/../../airflow/example_dags/example_short_circuit_operator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_short_circuit_trigger_rules]
+ :end-before: [END howto_operator_short_circuit_trigger_rules]
+
+
+
+Passing in arguments
+^^^^^^^^^^^^^^^^^^^^
+
+Both the ``op_args`` and ``op_kwargs`` arguments can be used in same way as
described for the PythonOperator.
+
Review comment:
I think this is redundant..?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
