[jira] [Commented] (AIRFLOW-2776) Tree view JSON is unnecessarily large
[ https://issues.apache.org/jira/browse/AIRFLOW-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601293#comment-16601293 ] Apache Spark commented on AIRFLOW-2776: --- User 'abdul-stripe' has created a pull request for this issue: https://github.com/apache/incubator-airflow/pull/3620 > Tree view JSON is unnecessarily large > - > > Key: AIRFLOW-2776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2776 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > The tree view generates JSON that can be massive for bigger DAGs –– some of > our tree views at stripe have 10s of MBs of JSON. > The [generated JSON is > prettified|https://github.com/apache/incubator-airflow/blob/52c745da71a6da798f7322956967b5e818b56e48/airflow/www/views.py#L1480], > which both takes up more CPU time during serialization, as well as slowing > everything else that uses it. We patched this on stripe's fork by removing > all whitespace and had a ton of speedup for bigger tree views. Considering > the JSON is only meant to be used programmatically, this is probably an easy > win. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2776) Tree view JSON is unnecessarily large
[ https://issues.apache.org/jira/browse/AIRFLOW-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558765#comment-16558765 ] ASF subversion and git services commented on AIRFLOW-2776: -- Commit 68535d1886f22b0fa833727aac4d0ee3ce12edf1 in incubator-airflow's branch refs/heads/master from [~abdul-stripe] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=68535d1 ] [AIRFLOW-2776] Compress tree view JSON The tree view generates JSON that can be massive for bigger DAGs, up to 10s of MBs. The JSON is currently prettified, which both takes up more CPU time during serialization, and slows down everything else that uses it. Considering the JSON is only meant to be used programmatically, this is an easy win Closes #3620 from abdul-stripe/smaller-tree-view- json > Tree view JSON is unnecessarily large > - > > Key: AIRFLOW-2776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2776 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > The tree view generates JSON that can be massive for bigger DAGs –– some of > our tree views at stripe have 10s of MBs of JSON. > The [generated JSON is > prettified|https://github.com/apache/incubator-airflow/blob/52c745da71a6da798f7322956967b5e818b56e48/airflow/www/views.py#L1480], > which both takes up more CPU time during serialization, as well as slowing > everything else that uses it. We patched this on stripe's fork by removing > all whitespace and had a ton of speedup for bigger tree views. Considering > the JSON is only meant to be used programmatically, this is probably an easy > win. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2776) Tree view JSON is unnecessarily large
[ https://issues.apache.org/jira/browse/AIRFLOW-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551356#comment-16551356 ] Abdul Nimeri commented on AIRFLOW-2776: --- https://github.com/apache/incubator-airflow/pull/3620 > Tree view JSON is unnecessarily large > - > > Key: AIRFLOW-2776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2776 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > > The tree view generates JSON that can be massive for bigger DAGs –– some of > our tree views at stripe have 10s of MBs of JSON. > The [generated JSON is > prettified|https://github.com/apache/incubator-airflow/blob/52c745da71a6da798f7322956967b5e818b56e48/airflow/www/views.py#L1480], > which both takes up more CPU time during serialization, as well as slowing > everything else that uses it. We patched this on stripe's fork by removing > all whitespace and had a ton of speedup for bigger tree views. Considering > the JSON is only meant to be used programmatically, this is probably an easy > win. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
