[jira] [Assigned] (AIRFLOW-2528) Airflow cli does not allow disabled stdin
[
https://issues.apache.org/jira/browse/AIRFLOW-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao-Han Tsai reassigned AIRFLOW-2528:
--
Assignee: Chao-Han Tsai
> Airflow cli does not allow disabled stdin
> -
>
> Key: AIRFLOW-2528
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2528
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.9.0
>Reporter: Brent Johnson
>Assignee: Chao-Han Tsai
>Priority: Major
>
> So basically, I am trying to automated regression testing by executing an
> airflow dag. I
>
> Using the cli I can successfully run the following command:
> {code:java}
> ./airflow run regression-testing regression-ingestion 2018-05-25{code}
> Unfortunately, I want to be triggering this against our staging instance in
> production on AWS.
> I figured an easy way to do this would be to use [AWS System
> Manager|https://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html]
> unfortunately any airflow command I call returns:
> {code:java}
> the input device is not a TTY
> {code}
> I was able to recreate this running the following command locally by piping
> stdin to anywhere:
> {code:java}
> ./airflow run regression-testing regression-ingestion 2018-05-25
> 0>/dev/null{code}
> This is of course an extreme example but it feels like a bug for a cli to
> require stdin to be open.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2613) DagBag stops reading DAGs and zipfiles if it does not encounter 'DAG' or 'airflow' in content
Trevor Edwards created AIRFLOW-2613: --- Summary: DagBag stops reading DAGs and zipfiles if it does not encounter 'DAG' or 'airflow' in content Key: AIRFLOW-2613 URL: https://issues.apache.org/jira/browse/AIRFLOW-2613 Project: Apache Airflow Issue Type: Bug Reporter: Trevor Edwards Due to [https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/models.py#L298] and [https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/models.py#L332,] Airflow will refuse to gather DAGs from a file which does not include the strings 'DAG' or 'airflow'. This can be problematic if you are using inheritance with your DAGs and thus don't import DAG in a file that generates a DAG. Worse, it stops scanning a zipfile altogether if any top-level files is missing the string. I'm not sure why these checks were included as it does not seem to provide any strong guarantee that the file is a DAG, so I suggest removing these odd checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2612) Enable some Hive-related tests on CI
Kengo Seki created AIRFLOW-2612:
---
Summary: Enable some Hive-related tests on CI
Key: AIRFLOW-2612
URL: https://issues.apache.org/jira/browse/AIRFLOW-2612
Project: Apache Airflow
Issue Type: Test
Components: tests
Reporter: Kengo Seki
Assignee: Kengo Seki
{{tests/operators/hive_operator.py}} has test cases called {{HiveServer2Test}}
and {{HivePrestoTest}}, which are disabled by default. Enabling them on CI
improves test coverage and contributes to avoiding regression.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-2611) Fix wrong dag volume mount path for kubernetes executor
[
https://issues.apache.org/jira/browse/AIRFLOW-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-2611 started by roc chan.
-
> Fix wrong dag volume mount path for kubernetes executor
> ---
>
> Key: AIRFLOW-2611
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2611
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 2.0.0, 1.10
>Reporter: roc chan
>Assignee: roc chan
>Priority: Major
>
> If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
> volume mount path that I want ([see
> code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
> {code}
> volume_mounts = [{
> 'name': dags_volume_name,
> 'mountPath': os.path.join(
> self.worker_airflow_dags,
> self.kube_config.git_subpath
> ),
> 'readOnly': True
> }, {
> 'name': logs_volume_name,
> 'mountPath': self.worker_airflow_logs
> }]
> {code}
> There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
> the priority of *pvc* should higher than *git-sync*, because some beginers
> use the template config file at the beginning ([like
> this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/scripts/ci/kubernetes/kube/configmaps.yaml#L184]),
> when they want to use pvc to mount dags, they may don't know they should
> remove the *git_subpath* config. So I think when *dags_volume_claim* is set,
> the *mountPath* shuold not join the *git_subpath*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2611) Fix wrong dag volume mount path for kubernetes executor
[
https://issues.apache.org/jira/browse/AIRFLOW-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
roc chan updated AIRFLOW-2611:
--
Description:
If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
volume mount path that I want ([see
code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
{code}
volume_mounts = [{
'name': dags_volume_name,
'mountPath': os.path.join(
self.worker_airflow_dags,
self.kube_config.git_subpath
),
'readOnly': True
}, {
'name': logs_volume_name,
'mountPath': self.worker_airflow_logs
}]
{code}
There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
the priority of *pvc* should higher than *git-sync*, because some beginers use
the template config file at the beginning ([like
this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/scripts/ci/kubernetes/kube/configmaps.yaml#L184]),
when they want to use pvc to mount dags, they may don't know they should
remove the *git_subpath* config. So I think when *dags_volume_claim* is set,
the *mountPath* shuold not join the *git_subpath*
was:
If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
volume mount path that I want ([see
code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
{code}
volume_mounts = [{
'name': dags_volume_name,
'mountPath': os.path.join(
self.worker_airflow_dags,
self.kube_config.git_subpath
),
'readOnly': True
}, {
'name': logs_volume_name,
'mountPath': self.worker_airflow_logs
}]
{code}
There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
the priority of *pvc* should higher than *git-sync*, because some beginers use
the template config file at the beginning ([like
this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/scripts/ci/kubernetes/kube/configmaps.yaml#L184]),
when they want to use pvc to mount dags, they may don't know they should
remove the *git_subpath* config.
> Fix wrong dag volume mount path for kubernetes executor
> ---
>
> Key: AIRFLOW-2611
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2611
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 2.0.0, 1.10
>Reporter: roc chan
>Assignee: roc chan
>Priority: Major
>
> If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
> volume mount path that I want ([see
> code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
> {code}
> volume_mounts = [{
> 'name': dags_volume_name,
> 'mountPath': os.path.join(
> self.worker_airflow_dags,
> self.kube_config.git_subpath
> ),
> 'readOnly': True
> }, {
> 'name': logs_volume_name,
> 'mountPath': self.worker_airflow_logs
> }]
> {code}
> There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
> the priority of *pvc* should higher than *git-sync*, because some beginers
> use the template config file at the beginning ([like
> this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/scripts/ci/kubernetes/kube/configmaps.yaml#L184]),
> when they want to use pvc to mount dags, they may don't know they should
> remove the *git_subpath* config. So I think when *dags_volume_claim* is set,
> the *mountPath* shuold not join the *git_subpath*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2611) Fix wrong dag volume mount path for kubernetes executor
[
https://issues.apache.org/jira/browse/AIRFLOW-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
roc chan updated AIRFLOW-2611:
--
Description:
If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
volume mount path that I want ([see
code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
{code}
volume_mounts = [{
'name': dags_volume_name,
'mountPath': os.path.join(
self.worker_airflow_dags,
self.kube_config.git_subpath
),
'readOnly': True
}, {
'name': logs_volume_name,
'mountPath': self.worker_airflow_logs
}]
{code}
There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
the priority of *pvc* should higher than *git-sync*, because some beginers use
the template config file at the beginning ([like
this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/scripts/ci/kubernetes/kube/configmaps.yaml#L184]),
when they want to use pvc to mount dags, they may don't know they should
remove the *git_subpath* config.
was:
If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
volume mount path that I want ([see
code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
{code}
volume_mounts = [{
'name': dags_volume_name,
'mountPath': os.path.join(
self.worker_airflow_dags,
self.kube_config.git_subpath
),
'readOnly': True
}, {
'name': logs_volume_name,
'mountPath': self.worker_airflow_logs
}]
{code}
There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
the priority of *pvc* should higher than *git-sync*, because some beginers use
the template config file at the beginning ([like
this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]),
when they want to use pvc to mount dags, they may don't know they should
remove the *git_subpath* config.
> Fix wrong dag volume mount path for kubernetes executor
> ---
>
> Key: AIRFLOW-2611
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2611
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 2.0.0, 1.10
>Reporter: roc chan
>Assignee: roc chan
>Priority: Major
>
> If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
> volume mount path that I want ([see
> code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
> {code}
> volume_mounts = [{
> 'name': dags_volume_name,
> 'mountPath': os.path.join(
> self.worker_airflow_dags,
> self.kube_config.git_subpath
> ),
> 'readOnly': True
> }, {
> 'name': logs_volume_name,
> 'mountPath': self.worker_airflow_logs
> }]
> {code}
> There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
> the priority of *pvc* should higher than *git-sync*, because some beginers
> use the template config file at the beginning ([like
> this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/scripts/ci/kubernetes/kube/configmaps.yaml#L184]),
> when they want to use pvc to mount dags, they may don't know they should
> remove the *git_subpath* config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2611) Fix wrong dag volume mount path for kubernetes executor
roc chan created AIRFLOW-2611:
-
Summary: Fix wrong dag volume mount path for kubernetes executor
Key: AIRFLOW-2611
URL: https://issues.apache.org/jira/browse/AIRFLOW-2611
Project: Apache Airflow
Issue Type: Bug
Components: contrib
Affects Versions: 2.0.0, 1.10
Reporter: roc chan
Assignee: roc chan
If I set both *dags_volume_claim* and *git_subpath*, well, I won't get the
volume mount path that I want ([see
code|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]):
{code}
volume_mounts = [{
'name': dags_volume_name,
'mountPath': os.path.join(
self.worker_airflow_dags,
self.kube_config.git_subpath
),
'readOnly': True
}, {
'name': logs_volume_name,
'mountPath': self.worker_airflow_logs
}]
{code}
There are two way of syncing dags, *pvc* and *git-sync*, I think if set both,
the priority of *pvc* should higher than *git-sync*, because some beginers use
the template config file at the beginning ([like
this|https://github.com/apache/incubator-airflow/blob/0f4d681f6f6e15acd1399dede146e75cb688d536/airflow/contrib/kubernetes/worker_configuration.py#L139]),
when they want to use pvc to mount dags, they may don't know they should
remove the *git_subpath* config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Description: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 11.08.26 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? was: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 11.05.13 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-13 at 11.05.13 AM.png, Screen Shot > 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 PM.png, > Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 11.08.26 AM.png! > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Description: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 11.05.13 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? was: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Screen Shot 2018-06-13 at 11.05.13 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-13 at 11.05.13 AM.png, Screen Shot > 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 PM.png, > Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Description: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Screen Shot 2018-06-13 at 11.05.13 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? was: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 2.38.37 PM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-13 at 11.05.13 AM.png, Screen Shot > 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 PM.png, > Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Screen Shot 2018-06-13 at 11.05.13 AM.png! > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Description: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 2.38.37 PM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? was: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 10.50.35 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-13 at 11.05.13 AM.png, Screen Shot > 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 PM.png, > Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Attachment: (was: Screen Shot 2018-06-12 at 9.40.31 AM.png) > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-13 at 11.05.13 AM.png, Screen Shot > 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 PM.png, > Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 10.50.35 AM.png! > > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Attachment: (was: Screen Shot 2018-06-12 at 9.35.55 AM.png) > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-12 at 9.40.31 AM.png, Screen Shot > 2018-06-13 at 11.05.13 AM.png, Screen Shot 2018-06-13 at 11.08.26 AM.png, > Screen Shot 2018-06-13 at 2.38.37 PM.png, Screen Shot 2018-06-13 at 3.22.43 > PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 10.50.35 AM.png! > > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Description: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 10.50.35 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? was: I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 10.50.35 AM.png!!Screen Shot 2018-06-13 at 11.08.26 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-12 at 9.35.55 AM.png, Screen Shot > 2018-06-12 at 9.40.31 AM.png, Screen Shot 2018-06-13 at 11.05.13 AM.png, > Screen Shot 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 > PM.png, Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 10.50.35 AM.png! > > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Attachment: Screen Shot 2018-06-13 at 11.08.26 AM.png > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-12 at 9.35.55 AM.png, Screen Shot > 2018-06-12 at 9.40.31 AM.png, Screen Shot 2018-06-13 at 11.05.13 AM.png, > Screen Shot 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 > PM.png, Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 10.50.35 AM.png! > > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Attachment: (was: Screen Shot 2018-06-13 at 11.08.26 AM.png) > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-12 at 9.35.55 AM.png, Screen Shot > 2018-06-12 at 9.40.31 AM.png, Screen Shot 2018-06-13 at 11.05.13 AM.png, > Screen Shot 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 > PM.png, Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 10.50.35 AM.png! > > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio Palma updated AIRFLOW-2610: --- Attachment: Screen Shot 2018-06-13 at 11.08.26 AM.png > AIRFLOW PythonBranchOperator > > > Key: AIRFLOW-2610 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Alessio Palma >Priority: Major > Attachments: Screen Shot 2018-06-12 at 9.35.55 AM.png, Screen Shot > 2018-06-12 at 9.40.31 AM.png, Screen Shot 2018-06-13 at 11.05.13 AM.png, > Screen Shot 2018-06-13 at 11.08.26 AM.png, Screen Shot 2018-06-13 at 2.38.37 > PM.png, Screen Shot 2018-06-13 at 3.22.43 PM.png > > > I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do > not seem to work. I'm getting incomplete executions, here there are some > examples: > !Screen Shot 2018-06-13 at 10.50.35 AM.png!!Screen Shot 2018-06-13 at > 11.08.26 AM.png! > Usually, when a dag_run completes, the scheduler writes a row where states > the status which has been set for it; something like: > INFO - Marking run scheduled__2018-06-13T12:45:00, externally triggered: False> successful > for the above Dags, this does not happen. > I noticed that when the Dag is manually started or triggered by another Dag, > it works perfectly. In this case, the PythonBranchOperator has no issues. > I added a tree view of the executions. > On the left, there are the executions which have been triggered by another > Dag, in the center, there are the scheduled executions, on the right, there > are the executions triggered manually. > You can note some scheduled execution which completed with success, actually, > these have been cleared after they failed. > !Screen Shot 2018-06-13 at 2.38.37 PM.png! > I think there is an issue on the scheduler; but currently, it is hard for me > to debug the issue. > > For the sake of the completeness, show chains work also in scheduled mode, > for example: > !Screen Shot 2018-06-13 at 11.05.13 AM.png! > works perfectly: > !Screen Shot 2018-06-13 at 3.22.43 PM.png! > Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2610) AIRFLOW PythonBranchOperator
Alessio Palma created AIRFLOW-2610: -- Summary: AIRFLOW PythonBranchOperator Key: AIRFLOW-2610 URL: https://issues.apache.org/jira/browse/AIRFLOW-2610 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.9.0 Reporter: Alessio Palma Attachments: Screen Shot 2018-06-12 at 9.35.55 AM.png, Screen Shot 2018-06-12 at 9.40.31 AM.png, Screen Shot 2018-06-13 at 11.05.13 AM.png, Screen Shot 2018-06-13 at 2.38.37 PM.png, Screen Shot 2018-06-13 at 3.22.43 PM.png I'm running the PythonBranchOperator inside a scheduled DAG, sadly things do not seem to work. I'm getting incomplete executions, here there are some examples: !Screen Shot 2018-06-13 at 10.50.35 AM.png!!Screen Shot 2018-06-13 at 11.08.26 AM.png! Usually, when a dag_run completes, the scheduler writes a row where states the status which has been set for it; something like: INFO - Marking run successful for the above Dags, this does not happen. I noticed that when the Dag is manually started or triggered by another Dag, it works perfectly. In this case, the PythonBranchOperator has no issues. I added a tree view of the executions. On the left, there are the executions which have been triggered by another Dag, in the center, there are the scheduled executions, on the right, there are the executions triggered manually. You can note some scheduled execution which completed with success, actually, these have been cleared after they failed. !Screen Shot 2018-06-13 at 2.38.37 PM.png! I think there is an issue on the scheduler; but currently, it is hard for me to debug the issue. For the sake of the completeness, show chains work also in scheduled mode, for example: !Screen Shot 2018-06-13 at 11.05.13 AM.png! works perfectly: !Screen Shot 2018-06-13 at 3.22.43 PM.png! Any idea, any quick fix to this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2607) test_trigger_dag need to be fixed
[ https://issues.apache.org/jira/browse/AIRFLOW-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Verdan Mahmood reassigned AIRFLOW-2607: --- Assignee: Verdan Mahmood > test_trigger_dag need to be fixed > - > > Key: AIRFLOW-2607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2607 > Project: Apache Airflow > Issue Type: Bug > Components: api, tests >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Major > Labels: test-fail > > test_trigger_dag is failing, and need to be fixed. (used docker with python3 > environment for testing) > File: tests/api/client/test_local_client.py > Class: TestLocalClient > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2608) Improved Exceptions handling in APIs
[ https://issues.apache.org/jira/browse/AIRFLOW-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Verdan Mahmood reassigned AIRFLOW-2608: --- Assignee: Verdan Mahmood > Improved Exceptions handling in APIs > > > Key: AIRFLOW-2608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2608 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Major > Labels: exception-handling > > The error handling in APIs is very poor, and most of the exceptions are > raised directly from AirflowExceptions which is the base exceptions class. > Also, some of the custom exceptions are created locally under the scope of > each file making it duplicated for other files. > We should have a centralized location for Exception classes along with the > standard default message for each exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2609) Fix small issue with the BranchPythonOperator. It currently is skipping tasks it should not.
[ https://issues.apache.org/jira/browse/AIRFLOW-2609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510854#comment-16510854 ] Sandro Luck commented on AIRFLOW-2609: -- Made change in this pull request https://github.com/apache/incubator-airflow/pull/3495 > Fix small issue with the BranchPythonOperator. It currently is skipping tasks > it should not. > > > Key: AIRFLOW-2609 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2609 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Sandro Luck >Assignee: Sandro Luck >Priority: Minor > > Current behavior: When you Branch from A e.g. the BranchPythonOperator '->' > (B or C,), and you make some B '->' C as well. The current behavior is that C > will be skipped even though it's a downstream task of B. Wishes behavior only > skip downstream tasks which are not in the list of the branch_taken > downstream tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2609) Fix small issue with the BranchPythonOperator. It currently is skipping tasks it should not.
[ https://issues.apache.org/jira/browse/AIRFLOW-2609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandro Luck closed AIRFLOW-2609. Resolution: Done https://github.com/apache/incubator-airflow/pull/3495 > Fix small issue with the BranchPythonOperator. It currently is skipping tasks > it should not. > > > Key: AIRFLOW-2609 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2609 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Sandro Luck >Assignee: Sandro Luck >Priority: Minor > > Current behavior: When you Branch from A e.g. the BranchPythonOperator '->' > (B or C,), and you make some B '->' C as well. The current behavior is that C > will be skipped even though it's a downstream task of B. Wishes behavior only > skip downstream tasks which are not in the list of the branch_taken > downstream tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2609) Fix small issue with the BranchPythonOperator. It currently is skipping tasks it should not.
Sandro Luck created AIRFLOW-2609: Summary: Fix small issue with the BranchPythonOperator. It currently is skipping tasks it should not. Key: AIRFLOW-2609 URL: https://issues.apache.org/jira/browse/AIRFLOW-2609 Project: Apache Airflow Issue Type: Bug Components: operators Reporter: Sandro Luck Assignee: Sandro Luck Current behavior: When you Branch from A e.g. the BranchPythonOperator '->' (B or C,), and you make some B '->' C as well. The current behavior is that C will be skipped even though it's a downstream task of B. Wishes behavior only skip downstream tasks which are not in the list of the branch_taken downstream tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2608) Improved Exceptions handling in APIs
Verdan Mahmood created AIRFLOW-2608: --- Summary: Improved Exceptions handling in APIs Key: AIRFLOW-2608 URL: https://issues.apache.org/jira/browse/AIRFLOW-2608 Project: Apache Airflow Issue Type: Improvement Reporter: Verdan Mahmood The error handling in APIs is very poor, and most of the exceptions are raised directly from AirflowExceptions which is the base exceptions class. Also, some of the custom exceptions are created locally under the scope of each file making it duplicated for other files. We should have a centralized location for Exception classes along with the standard default message for each exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2607) test_trigger_dag need to be fixed
[ https://issues.apache.org/jira/browse/AIRFLOW-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Verdan Mahmood updated AIRFLOW-2607: Description: test_trigger_dag is failing, and need to be fixed. (used docker with python3 environment for testing) File: tests/api/client/test_local_client.py Class: TestLocalClient was: test_trigger_dag is failing, and need to be fixed. File: tests/api/client/test_local_client.py Class: TestLocalClient > test_trigger_dag need to be fixed > - > > Key: AIRFLOW-2607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2607 > Project: Apache Airflow > Issue Type: Bug > Components: api, tests >Reporter: Verdan Mahmood >Priority: Major > Labels: test-fail > > test_trigger_dag is failing, and need to be fixed. (used docker with python3 > environment for testing) > File: tests/api/client/test_local_client.py > Class: TestLocalClient > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2607) test_trigger_dag need to be fixed
Verdan Mahmood created AIRFLOW-2607: --- Summary: test_trigger_dag need to be fixed Key: AIRFLOW-2607 URL: https://issues.apache.org/jira/browse/AIRFLOW-2607 Project: Apache Airflow Issue Type: Bug Components: api, tests Reporter: Verdan Mahmood test_trigger_dag is failing, and need to be fixed. File: tests/api/client/test_local_client.py Class: TestLocalClient -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2606) Test needed to ensure database schema always match SQLAlchemy model types
[ https://issues.apache.org/jira/browse/AIRFLOW-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joy Gao updated AIRFLOW-2606: - Summary: Test needed to ensure database schema always match SQLAlchemy model types (was: Test needed to ensure database schema always match SQLAlchemy models) > Test needed to ensure database schema always match SQLAlchemy model types > - > > Key: AIRFLOW-2606 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2606 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Joy Gao >Priority: Major > > An issue was discovered by [this > PR|https://github.com/apache/incubator-airflow/pull/3492#issuecomment-396815203] > where database schema does not match its corresponding SQLAlchemy model > declaration. We should add generic unit test for this to prevent similar bugs > from occurring in the future. (Alternatively, we can add the policing logic > to `airflow upgradedb` command so each migrations can do the check) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2606) Test needed to ensure database schema always match SQLAlchemy models
Joy Gao created AIRFLOW-2606: Summary: Test needed to ensure database schema always match SQLAlchemy models Key: AIRFLOW-2606 URL: https://issues.apache.org/jira/browse/AIRFLOW-2606 Project: Apache Airflow Issue Type: Improvement Reporter: Joy Gao An issue was discovered by [this PR|https://github.com/apache/incubator-airflow/pull/3492#issuecomment-396815203] where database schema does not match its corresponding SQLAlchemy model declaration. We should add generic unit test for this to prevent similar bugs from occurring in the future. (Alternatively, we can add the policing logic to `airflow upgradedb` command so each migrations can do the check) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2605) MySqlHook().run() will not commit if autocommit is set to True.
Kevin Yang created AIRFLOW-2605: --- Summary: MySqlHook().run() will not commit if autocommit is set to True. Key: AIRFLOW-2605 URL: https://issues.apache.org/jira/browse/AIRFLOW-2605 Project: Apache Airflow Issue Type: Bug Reporter: Kevin Yang Assignee: Kevin Yang MySql [set autocommit in a different way|https://github.com/PyMySQL/mysqlclient-python/blob/master/MySQLdb/connections.py#L249-L256]. Thus setting it by doing `conn.autocommit = True` as we currently do will not set autocommit correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
