[
https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chen xianxin updated AIRFLOW-5639:
----------------------------------
Description:
h2. Code
airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True,
safe_mode=True)
<
file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
if only_if_updated \
and filepath in self.file_last_changed \
and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
return found_dags
>
h2. Problem description
*self.file_last_changed* doesn't work, beacause it's set to empty when
initializing. so even no changes were made to the file, it will also parse
was:
airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True,
safe_mode=True)
file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
if only_if_updated \
and filepath in self.file_last_changed \
and file_last_changed_on_disk == self.file_last_changed[filepath]:
return found_dags
self.file_last_changed doesn't work, beacause it's set to empty when
initializing. so even no changes were made to the file, it will parse again
> DagFileProcessor: parse dag files every time, which consume lots of resources
> and is unnecessary
> ------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5639
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5639
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 1.10.5
> Reporter: chen xianxin
> Priority: Major
>
> h2. Code
> airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True,
> safe_mode=True)
> <
> file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
> if only_if_updated \
> and filepath in self.file_last_changed \
> and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
> return found_dags
> >
> h2. Problem description
> *self.file_last_changed* doesn't work, beacause it's set to empty when
> initializing. so even no changes were made to the file, it will also parse
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
