[
https://issues.apache.org/jira/browse/AIRFLOW-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042196#comment-17042196
]
ASF GitHub Bot commented on AIRFLOW-6871:
-----------------------------------------
houqp commented on pull request #7492: [AIRFLOW-6871] optimize tree view for
large DAGs
URL: https://github.com/apache/airflow/pull/7492
This change reduces page size by more than 10X and reduces page load time by
3-5X. As a result, the tree view can now load large DAGs that were causing 5XX
error without the patch.
Another example: one of our DAGs' tree view had a page size of 200MB and
took 1 minute to load. With this patch, it now loads within 24s with page size
of 17MB.
List of optimizations applied to the view handler:
* only seralize used task instance attributes to json instead of the
whole ORM object
* encode task instance attributes as array instead of dict
* encode datetime in unix timestamp instead of iso formmat string
* push task instance attribute construction into client side JS
* remove redundant task instance attributes
* simplify reduce_nodes() logic, remove unnecessary if statements
* seralize JSON as string to be used with JSON.parse on the client side
to speed up browser JS parse time
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN =
JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with
`[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Tree view unusable for large DAGs
> ---------------------------------
>
> Key: AIRFLOW-6871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6871
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webserver
> Affects Versions: 2.0.0
> Reporter: QP Hou
> Assignee: QP Hou
> Priority: Major
>
> By default airflow loads 25 runs in the tree view. For our main DAG, it's
> just not usable because all we get is a 5xx error when hitting that page.
> Manually overriding number of runs to 15 makes the page to load again, but
> it's very slow and takes more than 1 minute to render.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
