[jira] [Commented] (AIRFLOW-151) trigger_rule='one_success' not allowing tasks downstream of a BranchPythonOperator to be executed
[
https://issues.apache.org/jira/browse/AIRFLOW-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296493#comment-15296493
]
William Clark commented on AIRFLOW-151:
---
[~sanand], thank you so much for the assistance! With those changes, the DAG is
now functioning as expected.
!DAG_Problem_Resolved.PNG!
> trigger_rule='one_success' not allowing tasks downstream of a
> BranchPythonOperator to be executed
> -
>
> Key: AIRFLOW-151
> URL: https://issues.apache.org/jira/browse/AIRFLOW-151
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Siddharth Anand
>Assignee: Siddharth Anand
> Attachments: DAG_Problem.png, DAG_Problem_Resolved.PNG,
> DAG_Solution_Example.png
>
>
> Porting from https://github.com/apache/incubator-airflow/issues/1521
> Dear Airflow Maintainers,
> *Environment*
> {panel}
> Airflow version: 1.7.0rc3
> Airflow components: webserver, scheduler, worker, postgres database,
> CeleryExecutor
> Relevant airflow.cfg settings: nothing special here; mostly defaults
> Python Version: 3.4.3
> Operating System: Centos 6.7
> Python packages: virtualenv with standard airflow install
> {panel}
> *Background*
> We are constructing a workflow to automate standard business processes around
> the creation and maintenance of virtual machines. After creation, we verify
> several information points on the VM to ensure that it is a viable machine
> and that no configuration errors occurred. If it fails verification and is
> not running, then it should be deleted. If it fails verification and is
> running, then we stop it first, then delete it.
> *What did you expect to happen?*
> After researching the BranchPythonOperator, I found that I should be using
> trigger_rule='one_success' to allow a task at a join point downstream of the
> branch(es) to be triggered, as mentioned in #1078. So, I defined the task as
> follows:
> {code}
> delete_vm = PythonOperator(
> task_id='delete_vm',
> trigger_rule=TriggerRule.ONE_SUCCESS,
> python_callable=_delete_vm,
> provide_context=True,
> dag=dag)
> delete_vm.set_upstream({poll_vm_stop, verify_vm})
> {code}
> *What happened instead?*
> Rather than executing correctly, the delete_vm task is marked as skipped and
> is not re-evaluated following poll_vm_stop. There is no stack trace
> available, as the task simply does not execute. Sidenote: the PythonSensor
> you see in the picture below is a sensor which evaluates the truthy- or
> falsey-ness of a Python callable. It has been tested extensively and works as
> intended.
> !DAG_Problem.png!
> Any help would be greatly appreciated. I've tested various ways of linking
> the dag, providing DummyOperators as buffers, using a second
> BranchPythonOperator to explicitly call the task; all of these have failed.
> Am I missing something obvious here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-145) trigger_rule='one_success' not allowing tasks downstream of a BranchPythonOperator to be executed
[
https://issues.apache.org/jira/browse/AIRFLOW-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Clark updated AIRFLOW-145:
--
Description:
h3.Background
We are constructing a workflow to automate standard business processes around
the creation and maintenance of virtual machines. After creation, we verify
several information points on the VM to ensure that it is a viable machine and
that no configuration errors occurred. If it fails verification and is not
running, then it should be deleted. If it fails verification and is running,
then we stop it first, then delete it.
h3.What did you expect to happen?
After researching the BranchPythonOperator, I found that I should be using
trigger_rule='one_success' to allow a task at a join point downstream of the
branch(es) to be triggered, as mentioned in #1078. So, I defined the task as
follows:
{code}
delete_vm = PythonOperator(
task_id='delete_vm',
trigger_rule=TriggerRule.ONE_SUCCESS,
python_callable=_delete_vm,
provide_context=True,
dag=dag)
delete_vm.set_upstream({poll_vm_stop, verify_vm})
{code}
h3.What happened instead?
Rather than executing correctly, the delete_vm task is marked as skipped and is
not re-evaluated following poll_vm_stop. There is no stack trace available, as
the task simply does not execute.
Sidenote: the PythonSensor you see in the attached picture is a sensor which
evaluates the truthy- or falsey-ness of a Python callable. It has been tested
extensively and works as intended.
reference: https://github.com/apache/incubator-airflow/issues/1521
was:
Background
We are constructing a workflow to automate standard business processes around
the creation and maintenance of virtual machines. After creation, we verify
several information points on the VM to ensure that it is a viable machine and
that no configuration errors occurred. If it fails verification and is not
running, then it should be deleted. If it fails verification and is running,
then we stop it first, then delete it.
What did you expect to happen?
After researching the BranchPythonOperator, I found that I should be using
trigger_rule='one_success' to allow a task at a join point downstream of the
branch(es) to be triggered, as mentioned in #1078. So, I defined the task as
follows:
{code}
delete_vm = PythonOperator(
task_id='delete_vm',
trigger_rule=TriggerRule.ONE_SUCCESS,
python_callable=_delete_vm,
provide_context=True,
dag=dag)
delete_vm.set_upstream({poll_vm_stop, verify_vm})
{code}
What happened instead?
Rather than executing correctly, the delete_vm task is marked as skipped and is
not re-evaluated following poll_vm_stop. There is no stack trace available, as
the task simply does not execute.
Sidenote: the PythonSensor you see in the attached picture is a sensor which
evaluates the truthy- or falsey-ness of a Python callable. It has been tested
extensively and works as intended.
reference: https://github.com/apache/incubator-airflow/issues/1521
> trigger_rule='one_success' not allowing tasks downstream of a
> BranchPythonOperator to be executed
> -
>
> Key: AIRFLOW-145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-145
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: Airflow 1.7.0
> Environment: Airflow version: 1.7.0rc3
> Airflow components: webserver, scheduler, worker, postgres database,
> CeleryExecutor
> Relevant airflow.cfg settings: nothing special here; mostly defaults
> Python Version: 3.4.3
> Operating System: Centos 6.7
> Python packages: virtualenv with standard airflow install
>Reporter: William Clark
> Attachments: screenshot-1.png
>
>
> h3.Background
> We are constructing a workflow to automate standard business processes around
> the creation and maintenance of virtual machines. After creation, we verify
> several information points on the VM to ensure that it is a viable machine
> and that no configuration errors occurred. If it fails verification and is
> not running, then it should be deleted. If it fails verification and is
> running, then we stop it first, then delete it.
> h3.What did you expect to happen?
> After researching the BranchPythonOperator, I found that I should be using
> trigger_rule='one_success' to allow a task at a join point downstream of the
> branch(es) to be triggered, as mentioned in #1078. So, I defined the task as
> follows:
> {code}
> delete_vm = PythonOperator(
> task_id='delete_vm',
> trigger_rule=TriggerRule.ONE_SUCCESS,
> python_callable=_delete_vm,
> provide_context=True,
> dag=dag)
> delete_vm.set_upstream({poll_vm_stop, verify_vm})
> {code}
> h3.What happened instead?
> Rather than executing
[jira] [Updated] (AIRFLOW-145) trigger_rule='one_success' not allowing tasks downstream of a BranchPythonOperator to be executed
[
https://issues.apache.org/jira/browse/AIRFLOW-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Clark updated AIRFLOW-145:
--
Attachment: screenshot-1.png
> trigger_rule='one_success' not allowing tasks downstream of a
> BranchPythonOperator to be executed
> -
>
> Key: AIRFLOW-145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-145
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: Airflow 1.7.0
> Environment: Airflow version: 1.7.0rc3
> Airflow components: webserver, scheduler, worker, postgres database,
> CeleryExecutor
> Relevant airflow.cfg settings: nothing special here; mostly defaults
> Python Version: 3.4.3
> Operating System: Centos 6.7
> Python packages: virtualenv with standard airflow install
>Reporter: William Clark
> Attachments: screenshot-1.png
>
>
> Background
> We are constructing a workflow to automate standard business processes around
> the creation and maintenance of virtual machines. After creation, we verify
> several information points on the VM to ensure that it is a viable machine
> and that no configuration errors occurred. If it fails verification and is
> not running, then it should be deleted. If it fails verification and is
> running, then we stop it first, then delete it.
> What did you expect to happen?
> After researching the BranchPythonOperator, I found that I should be using
> trigger_rule='one_success' to allow a task at a join point downstream of the
> branch(es) to be triggered, as mentioned in #1078. So, I defined the task as
> follows:
> delete_vm = PythonOperator(
> task_id='delete_vm',
> trigger_rule=TriggerRule.ONE_SUCCESS,
> python_callable=_delete_vm,
> provide_context=True,
> dag=dag)
> delete_vm.set_upstream({poll_vm_stop, verify_vm})
> What happened instead?
> Rather than executing correctly, the delete_vm task is marked as skipped and
> is not re-evaluated following poll_vm_stop. There is no stack trace
> available, as the task simply does not execute.
> reference: https://github.com/apache/incubator-airflow/issues/1521
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-145) trigger_rule='one_success' not allowing tasks downstream of a BranchPythonOperator to be executed
William Clark created AIRFLOW-145:
-
Summary: trigger_rule='one_success' not allowing tasks downstream
of a BranchPythonOperator to be executed
Key: AIRFLOW-145
URL: https://issues.apache.org/jira/browse/AIRFLOW-145
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: Airflow 1.7.0
Environment: Airflow version: 1.7.0rc3
Airflow components: webserver, scheduler, worker, postgres database,
CeleryExecutor
Relevant airflow.cfg settings: nothing special here; mostly defaults
Python Version: 3.4.3
Operating System: Centos 6.7
Python packages: virtualenv with standard airflow install
Reporter: William Clark
Background
We are constructing a workflow to automate standard business processes around
the creation and maintenance of virtual machines. After creation, we verify
several information points on the VM to ensure that it is a viable machine and
that no configuration errors occurred. If it fails verification and is not
running, then it should be deleted. If it fails verification and is running,
then we stop it first, then delete it.
What did you expect to happen?
After researching the BranchPythonOperator, I found that I should be using
trigger_rule='one_success' to allow a task at a join point downstream of the
branch(es) to be triggered, as mentioned in #1078. So, I defined the task as
follows:
delete_vm = PythonOperator(
task_id='delete_vm',
trigger_rule=TriggerRule.ONE_SUCCESS,
python_callable=_delete_vm,
provide_context=True,
dag=dag)
delete_vm.set_upstream({poll_vm_stop, verify_vm})
What happened instead?
Rather than executing correctly, the delete_vm task is marked as skipped and is
not re-evaluated following poll_vm_stop. There is no stack trace available, as
the task simply does not execute.
reference: https://github.com/apache/incubator-airflow/issues/1521
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
