[ 
https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chen xianxin updated AIRFLOW-5639:
----------------------------------
    Description: 
 
h2. Code

[https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse

  was:
h2. Code

airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, 
safe_mode=True)

<

file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
 if only_if_updated \
     and filepath in self.file_last_changed \
     and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
     return found_dags

>

https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse


> DagFileProcessor: parse dag files every time, which consume lots of resources 
> and is unnecessary
> ------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5639
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5639
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 1.10.5
>            Reporter: chen xianxin
>            Priority: Major
>
>  
> h2. Code
> [https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
> h2. Problem description 
> *self.file_last_changed* doesn't work, beacause it's set to empty when 
> initializing. so even no changes were made to the file, it will also parse



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to