[jira] [Commented] (AIRFLOW-5210) Resolving Template Files for large DAGs hurts performance
[ https://issues.apache.org/jira/browse/AIRFLOW-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932578#comment-16932578 ] ASF subversion and git services commented on AIRFLOW-5210: -- Commit 85ac7f748f13cb71ba989acb6800dbcea383541e in airflow's branch refs/heads/v1-10-test from Daniel Frank [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=85ac7f7 ] [AIRFLOW-5210] Make finding template files more efficient (#5815) For large DAGs, iterating over template fields to find template files can be time intensive. Save this time for tasks that do not specify a template file extension. (cherry picked from commit eeac82318a6440b2d65f9a35b3437b91813945f4) > Resolving Template Files for large DAGs hurts performance > -- > > Key: AIRFLOW-5210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5210 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.4 >Reporter: Daniel Frank >Priority: Major > Fix For: 1.10.6 > > > During task execution, "resolve_template_files" runs for all tasks in a > given DAG. For large DAGs this takes a long time and is not necessary for > tasks that do not use the template_ext field -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5210) Resolving Template Files for large DAGs hurts performance
[ https://issues.apache.org/jira/browse/AIRFLOW-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907336#comment-16907336 ] ASF subversion and git services commented on AIRFLOW-5210: -- Commit 577970c210c9160be9e2382ecfd3ae79b01e4d88 in airflow's branch refs/heads/revert-5815-df_resolve_template_files from Bas Harenslak [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=577970c ] Revert "[AIRFLOW-5210] Make finding template files more efficient (#5815)" This reverts commit eeac82318a6440b2d65f9a35b3437b91813945f4. > Resolving Template Files for large DAGs hurts performance > -- > > Key: AIRFLOW-5210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5210 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.4 >Reporter: Daniel Frank >Priority: Major > Fix For: 1.10.5 > > > During task execution, "resolve_template_files" runs for all tasks in a > given DAG. For large DAGs this takes a long time and is not necessary for > tasks that do not use the template_ext field -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-5210) Resolving Template Files for large DAGs hurts performance
[ https://issues.apache.org/jira/browse/AIRFLOW-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907254#comment-16907254 ] ASF subversion and git services commented on AIRFLOW-5210: -- Commit eeac82318a6440b2d65f9a35b3437b91813945f4 in airflow's branch refs/heads/master from Daniel Frank [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=eeac823 ] [AIRFLOW-5210] Make finding template files more efficient (#5815) For large DAGs, iterating over template fields to find template files can be time intensive. Save this time for tasks that do not specify a template file extension. > Resolving Template Files for large DAGs hurts performance > -- > > Key: AIRFLOW-5210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5210 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.4 >Reporter: Daniel Frank >Priority: Major > > During task execution, "resolve_template_files" runs for all tasks in a > given DAG. For large DAGs this takes a long time and is not necessary for > tasks that do not use the template_ext field -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-5210) Resolving Template Files for large DAGs hurts performance
[ https://issues.apache.org/jira/browse/AIRFLOW-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907253#comment-16907253 ] ASF GitHub Bot commented on AIRFLOW-5210: - BasPH commented on pull request #5815: [AIRFLOW-5210] Make finding template files more efficient URL: https://github.com/apache/airflow/pull/5815 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Resolving Template Files for large DAGs hurts performance > -- > > Key: AIRFLOW-5210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5210 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.4 >Reporter: Daniel Frank >Priority: Major > > During task execution, "resolve_template_files" runs for all tasks in a > given DAG. For large DAGs this takes a long time and is not necessary for > tasks that do not use the template_ext field -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-5210) Resolving Template Files for large DAGs hurts performance
[ https://issues.apache.org/jira/browse/AIRFLOW-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906809#comment-16906809 ] ASF GitHub Bot commented on AIRFLOW-5210: - danfrankj commented on pull request #5815: [AIRFLOW-5210] Make finding template files more efficient URL: https://github.com/apache/airflow/pull/5815 Make sure you have checked _all_ steps below. ### Jira https://issues.apache.org/jira/browse/AIRFLOW-5210 ### Description For large DAGs (1000s of tasks) iterating over template_fields adds significant time to task execution and is not necessary for tasks that do not specify template_ext ### Tests I can add tests if you think necessary, but this is a _very_ simple change ### Code Quality - [ ] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Resolving Template Files for large DAGs hurts performance > -- > > Key: AIRFLOW-5210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5210 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.4 >Reporter: Daniel Frank >Priority: Major > > During task execution, "resolve_template_files" runs for all tasks in a > given DAG. For large DAGs this takes a long time and is not necessary for > tasks that do not use the template_ext field -- This message was sent by Atlassian JIRA (v7.6.14#76016)
