[
https://issues.apache.org/jira/browse/AIRFLOW-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915053#comment-16915053
]
ASF GitHub Bot commented on AIRFLOW-4797:
-----------------------------------------
KevinYang21 commented on pull request #5908: [WIP]Revert "[AIRFLOW-4797]
Improve performance and behaviour of zombie de…
URL: https://github.com/apache/airflow/pull/5908
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Original reason stated in the PR why zombie detection was moved
```Zombie tasks will be calculate by DAG parsing manager and send to DAG
parsing processor to kill. This is to reduce DB CPU load( identified to produce
80% of CPU load during stress test, CPU usage went down from 80%+ to ~40% after
this change).```
from https://github.com/apache/airflow/pull/3873.
I see no point sending a query joining two biggest tables in every DAG
parsing. Establishing new connections is much more expensive than sending an
aggregated query. It doesn't seem to deliver any immediate value: the DB load
in a smaller cluster was not changing. And if we want to compare the running
time diff we compare the aggregated query running time on all DAG file
processors vs. the old query running time instead of compare the individual
query. We parse a couple thoudsand files in 2 mins and it will generate heavy
load to the DB, which I believe is the biggest bottelneck of Airflow
scalibility.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Reverting PR
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Zombie detection and killing is not deterministic
> -------------------------------------------------
>
> Key: AIRFLOW-4797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4797
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.3
> Reporter: Stefan Seelmann
> Assignee: Stefan Seelmann
> Priority: Major
> Fix For: 1.10.4
>
>
> Zombie detection and killing is done within the DAG file processing loop.
> Within one iteration only a subset of the DAG files are processed (config
> scheduler.max_threads). The loop sleeps for the rest of the second, until the
> next iteration runs which processes the next subset of DAG files. The
> function to get zombie task instancs only returns zombies once within 10
> seconds, otherwise an empty list is returned.
> That means only in every 10th iteration of the DAG file processing loop
> zombies are detected. And only if the zombie task belong to one of the DAG
> files of the current iteration they are killed.
> We run into the worst case scenario with max_threads=2 and 20 DAGs. In such a
> scenario only zombies of the same 2 DAGs are killed. (as loop iterations are
> not exactly 1s it shifts slowly and eventually the zomies are killed, but in
> one example it took 33 minutes).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
