[jira] [Commented] (AIRFLOW-1298) Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616491#comment-16616491
]
ASF GitHub Bot commented on AIRFLOW-1298:
-
Fokko closed pull request #3886: [AIRFLOW-1298] Fix 'clear only_failed'
URL: https://github.com/apache/incubator-airflow/pull/3886
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/models.py b/airflow/models.py
index 86a9cc2fc6..4b4554ff59 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -3821,7 +3821,9 @@ def clear(
if end_date:
tis = tis.filter(TI.execution_date <= end_date)
if only_failed:
-tis = tis.filter(TI.state == State.FAILED)
+tis = tis.filter(or_(
+TI.state == State.FAILED,
+TI.state == State.UPSTREAM_FAILED))
if only_running:
tis = tis.filter(TI.state == State.RUNNING)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
> ---
>
> Key: AIRFLOW-1298
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1298
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.8.0
> Environment: Ubuntu 12.04, Kernel: 3.13.0-113-generic, python 2.7.10
>Reporter: Aaditya Ramesh
>Assignee: Aaditya Ramesh
>Priority: Critical
> Fix For: 2.0.0
>
>
> We are unable to clear airflow tasks that are in the UPSTREAM_FAILED state
> using the command line. The fix is simple - just change `clear` function in
> models.py to also clear tasks in UPSTREAM_FAILED state, not just FAILED.
> Diff:
> {noformat}
> diff --git a/airflow/models.py b/airflow/models.py
> index 30e18a44..e60d2918 100755
> --- a/airflow/models.py
> +++ b/airflow/models.py
> @@ -3180,7 +3180,7 @@ class DAG(BaseDag, LoggingMixin):
> if end_date:
> tis = tis.filter(TI.execution_date <= end_date)
> if only_failed:
> -tis = tis.filter(TI.state == State.FAILED)
> +tis = tis.filter(TI.state == State.FAILED or TI.state ==
> State.UPSTREAM_FAILED)
> if only_running:
> tis = tis.filter(TI.state == State.RUNNING)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1298) Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611489#comment-16611489
]
ASF GitHub Bot commented on AIRFLOW-1298:
-
ubermen opened a new pull request #3886: [AIRFLOW-1298] Fix 'clear only_failed'
URL: https://github.com/apache/incubator-airflow/pull/3886
[Description]
In general, when users clear failed tasks in CLI, they expect that all
down_streams which is upstream_failed should be cleared together. But, current
clear -f option could clear only 'failed' tasks, so operators have to clear
downstream tasks (upstream_failed) manually.
I often use 'airflow clear -f' for reprocessing massive failed tasks. So,
modified models.py like this
[ASIS] origin
if only_failed: tis = tis.filter(TI.state == State.FAILED)
[TOBE] modified
if only_failed: tis = tis.filter(or_(TI.state == State.FAILED, TI.state ==
State.UPSTREAM_FAILED))
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
> ---
>
> Key: AIRFLOW-1298
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1298
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.8.0
> Environment: Ubuntu 12.04, Kernel: 3.13.0-113-generic, python 2.7.10
>Reporter: Aaditya Ramesh
>Assignee: Aaditya Ramesh
>Priority: Critical
>
> We are unable to clear airflow tasks that are in the UPSTREAM_FAILED state
> using the command line. The fix is simple - just change `clear` function in
> models.py to also clear tasks in UPSTREAM_FAILED state, not just FAILED.
> Diff:
> {noformat}
> diff --git a/airflow/models.py b/airflow/models.py
> index 30e18a44..e60d2918 100755
> --- a/airflow/models.py
> +++ b/airflow/models.py
> @@ -3180,7 +3180,7 @@ class DAG(BaseDag, LoggingMixin):
> if end_date:
> tis = tis.filter(TI.execution_date <= end_date)
> if only_failed:
> -tis = tis.filter(TI.state == State.FAILED)
> +tis = tis.filter(TI.state == State.FAILED or TI.state ==
> State.UPSTREAM_FAILED)
> if only_running:
> tis = tis.filter(TI.state == State.RUNNING)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1298) Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606631#comment-16606631
]
ASF GitHub Bot commented on AIRFLOW-1298:
-
ubermen closed pull request #3667: [AIRFLOW-1298] Add clear option
'only_failed_or_upstream_failed'
URL: https://github.com/apache/incubator-airflow/pull/3667
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
> ---
>
> Key: AIRFLOW-1298
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1298
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.8.0
> Environment: Ubuntu 12.04, Kernel: 3.13.0-113-generic, python 2.7.10
>Reporter: Aaditya Ramesh
>Assignee: Aaditya Ramesh
>Priority: Critical
>
> We are unable to clear airflow tasks that are in the UPSTREAM_FAILED state
> using the command line. The fix is simple - just change `clear` function in
> models.py to also clear tasks in UPSTREAM_FAILED state, not just FAILED.
> Diff:
> {noformat}
> diff --git a/airflow/models.py b/airflow/models.py
> index 30e18a44..e60d2918 100755
> --- a/airflow/models.py
> +++ b/airflow/models.py
> @@ -3180,7 +3180,7 @@ class DAG(BaseDag, LoggingMixin):
> if end_date:
> tis = tis.filter(TI.execution_date <= end_date)
> if only_failed:
> -tis = tis.filter(TI.state == State.FAILED)
> +tis = tis.filter(TI.state == State.FAILED or TI.state ==
> State.UPSTREAM_FAILED)
> if only_running:
> tis = tis.filter(TI.state == State.RUNNING)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1298) Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569786#comment-16569786
]
ASF GitHub Bot commented on AIRFLOW-1298:
-
ubermen opened a new pull request #3667: [AIRFLOW-1298] Add clear option
'only_failed_or_upstream_failed'
URL: https://github.com/apache/incubator-airflow/pull/3667
### JIRA
- [x] My PR addresses the following [Airflow Jira]
- https://issues.apache.org/jira/browse/AIRFLOW-1298
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
In my operations team, it is necessary to clear all tasks in one command
line (because of so many schedules should be reprocessed cause of some reasons)
But, 'clear -cdf ...' is only clearing failed tasks without upstream_failed.
It is not fit for our operator's needs.
So, I want to add new options to clear failed or upstream_failed jobs all at
once.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
1. clear only one schedule of this task : airflow clear -cp -s
2018-07-22T05:00:00 -e 2018-07-22T05:00:00 -t ^task_name$ schedule_name
2. clear only multiple schedules of this task : airflow clear -cp -s
2018-07-22T05:00:00 -e 2018-07-22T09:00:00 -t ^task_name$ schedule_name
3. clear multiple schedules of this task with downstream : airflow clear
-cdp -s 2018-07-22T05:00:00 -e 2018-07-22T09:00:00 -t ^task_name$ schedule_name
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
new option of clear command
option is 'p' of upstream_failed's p character.
Usage sample is like below :
airflow clear -cdp -s 2018-07-22T05:00:00 -e 2018-07-22T09:00:00 -t
^task_name$ schedule_name
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
> ---
>
> Key: AIRFLOW-1298
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1298
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: Airflow 1.8
> Environment: Ubuntu 12.04, Kernel: 3.13.0-113-generic, python 2.7.10
>Reporter: Aaditya Ramesh
>Assignee: Aaditya Ramesh
>Priority: Critical
>
> We are unable to clear airflow tasks that are in the UPSTREAM_FAILED state
> using the command line. The fix is simple - just change `clear` function in
> models.py to also clear tasks in UPSTREAM_FAILED state, not just FAILED.
> Diff:
> {noformat}
> diff --git a/airflow/models.py b/airflow/models.py
> index 30e18a44..e60d2918 100755
> --- a/airflow/models.py
> +++ b/airflow/models.py
> @@ -3180,7 +3180,7 @@ class DAG(BaseDag, LoggingMixin):
> if end_date:
> tis = tis.filter(TI.execution_date <= end_date)
> if only_failed:
> -tis = tis.filter(TI.state == State.FAILED)
> +tis = tis.filter(TI.state == State.FAILED or TI.state ==
> State.UPSTREAM_FAILED)
> if only_running:
> tis = tis.filter(TI.state == State.RUNNING)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1298) Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569785#comment-16569785
]
ASF GitHub Bot commented on AIRFLOW-1298:
-
ubermen closed pull request #3667: [AIRFLOW-1298] Add clear option
'only_failed_or_upstream_failed'
URL: https://github.com/apache/incubator-airflow/pull/3667
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index b56e325327..8e329bd074 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -627,6 +627,7 @@ def clear(args):
start_date=args.start_date,
end_date=args.end_date,
only_failed=args.only_failed,
+only_failed_or_upstream_failed=args.only_failed_or_upstream_failed,
only_running=args.only_running,
confirm_prompt=not args.no_confirm,
include_subdags=not args.exclude_subdags)
@@ -1399,6 +1400,9 @@ class CLIFactory(object):
("-u", "--upstream"), "Include upstream tasks", "store_true"),
'only_failed': Arg(
("-f", "--only_failed"), "Only failed jobs", "store_true"),
+'only_failed_or_upstream_failed': Arg(
+("-p", "--only_failed_or_upstream_failed"),
+"Only failed or upstream_failed jobs", "store_true"),
'only_running': Arg(
("-r", "--only_running"), "Only running jobs", "store_true"),
'downstream': Arg(
@@ -1728,6 +1732,7 @@ class CLIFactory(object):
'args': (
'dag_id', 'task_regex', 'start_date', 'end_date', 'subdir',
'upstream', 'downstream', 'no_confirm', 'only_failed',
+'only_failed_or_upstream_failed',
'only_running', 'exclude_subdags', 'dag_regex'),
}, {
'func': pause,
diff --git a/airflow/models.py b/airflow/models.py
index b789fd1732..797ff69578 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -3751,6 +3751,7 @@ def set_dag_runs_state(
def clear(
self, start_date=None, end_date=None,
only_failed=False,
+only_failed_or_upstream_failed=False,
only_running=False,
confirm_prompt=False,
include_subdags=True,
@@ -3783,6 +3784,9 @@ def clear(
tis = tis.filter(TI.execution_date <= end_date)
if only_failed:
tis = tis.filter(TI.state == State.FAILED)
+if only_failed_or_upstream_failed:
+tis = tis.filter(or_(TI.state == State.FAILED,
+ TI.state == State.UPSTREAM_FAILED))
if only_running:
tis = tis.filter(TI.state == State.RUNNING)
@@ -3826,6 +3830,7 @@ def clear_dags(
start_date=None,
end_date=None,
only_failed=False,
+only_failed_or_upstream_failed=False,
only_running=False,
confirm_prompt=False,
include_subdags=True,
@@ -3838,6 +3843,7 @@ def clear_dags(
start_date=start_date,
end_date=end_date,
only_failed=only_failed,
+only_failed_or_upstream_failed=only_failed_or_upstream_failed,
only_running=only_running,
confirm_prompt=False,
include_subdags=include_subdags,
@@ -3866,6 +3872,7 @@ def clear_dags(
dag.clear(start_date=start_date,
end_date=end_date,
only_failed=only_failed,
+
only_failed_or_upstream_failed=only_failed_or_upstream_failed,
only_running=only_running,
confirm_prompt=False,
include_subdags=include_subdags,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
> ---
>
> Key: AIRFLOW-1298
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1298
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: Airflow 1.8
> Environment: Ubuntu 12.04, Kernel: 3.13.0-113-generic, python 2.7.10
>Reporter: Aaditya Ramesh
>Assignee: Aaditya Ramesh
>Priority: Critical
>
> We are unable to clear airflow tasks that are in the
[jira] [Commented] (AIRFLOW-1298) Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563151#comment-16563151
]
ASF GitHub Bot commented on AIRFLOW-1298:
-
ubermen opened a new pull request #3667: [AIRFLOW-1298] Add clear option
'only_failed_or_upstream_failed'
URL: https://github.com/apache/incubator-airflow/pull/3667
### Description
In my operations team, it is necessary to clear all tasks in one command
line (because of so many schedules will be reprocessed)
But, 'clear -cdf ...' is only clearing failed tasks without upstream_failed.
It is not fit for our operator's needs.
So, I want to add new options to clear failed or upstream_failed jobs all at
once.
### Tests
1. clear only one schedule of this task : airflow clear -cp -s
2018-07-22T05:00:00 -e 2018-07-22T05:00:00 -t ^task_name$ schedule_name
2. clear only multiple schedules of this task : airflow clear -cp -s
2018-07-22T05:00:00 -e 2018-07-22T09:00:00 -t ^task_name$ schedule_name
3. clear multiple schedules of this task with downstream : airflow clear
-cdp -s 2018-07-22T05:00:00 -e 2018-07-22T09:00:00 -t ^task_name$ schedule_name
### Documentation
new option of clear command
option is 'p' of upstream_failed's p character.
Usage sample is like below :
airflow clear -cdp -s 2018-07-22T05:00:00 -e 2018-07-22T09:00:00 -t
^task_name$ schedule_name
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow Clear Command does not clear tasks in UPSTREAM_FAILED state
> ---
>
> Key: AIRFLOW-1298
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1298
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: Airflow 1.8
> Environment: Ubuntu 12.04, Kernel: 3.13.0-113-generic, python 2.7.10
>Reporter: Aaditya Ramesh
>Assignee: Aaditya Ramesh
>Priority: Critical
>
> We are unable to clear airflow tasks that are in the UPSTREAM_FAILED state
> using the command line. The fix is simple - just change `clear` function in
> models.py to also clear tasks in UPSTREAM_FAILED state, not just FAILED.
> Diff:
> {noformat}
> diff --git a/airflow/models.py b/airflow/models.py
> index 30e18a44..e60d2918 100755
> --- a/airflow/models.py
> +++ b/airflow/models.py
> @@ -3180,7 +3180,7 @@ class DAG(BaseDag, LoggingMixin):
> if end_date:
> tis = tis.filter(TI.execution_date <= end_date)
> if only_failed:
> -tis = tis.filter(TI.state == State.FAILED)
> +tis = tis.filter(TI.state == State.FAILED or TI.state ==
> State.UPSTREAM_FAILED)
> if only_running:
> tis = tis.filter(TI.state == State.RUNNING)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
