[
https://issues.apache.org/jira/browse/AIRFLOW-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sébastien GENESTA updated AIRFLOW-3267:
---------------------------------------
Description:
Hi,
We are encountering strange behaviour with Airflow Scheduler since our upgrade
from Apache Airflow 1.8 to 1.10.
We have configured 4 workers on 4 servers which all can treat 6 tasks
simultaneously.
We notice that after a period (not yet determined), the scheduler seems to stop
scheduling at is maximum capacity of this 24 tasks so delayed our process.
To troubleshoot this behaviour we've configured the scheduler reboot every 3
hours.
I've also notice following informations in
[https://github.com/apache/incubator-airflow/blob/master/UPDATING.md]
[...]
_The scheduler.min_file_parsing_loop_time config option has been temporarily
removed due to some bugs._
[...]
and find corresponding value uncommented and set to 1 in airflow.cfg in
[scheduler] section.
Does this bug keep pending? Is-it related to our issue? Do we have to comment
it?
Thanks for your help.
was:
Hi,
We are encountering strange behaviour with Airflow Scheduler since our upgrade
from Apache Airflow 1.8 to 1.10.
We have configured 4 workers on 4 servers which all can treat 6 tasks
simultaneously.
We notice that after a period (not yet determined), the scheduler seems to stop
scheduling at is maximum capacity of this 24 tasks so delayed our process.
To troubleshoot this behaviour we've configured the scheduler reboot every 3
hours.
Thanks for your help.
> Airflow Scheduler instable after Webserver upgrade from 1.8 to 1.10
> -------------------------------------------------------------------
>
> Key: AIRFLOW-3267
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3267
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.0
> Environment: Debian 8
> Apache-Airflow 1.10
> Reporter: Sébastien GENESTA
> Priority: Major
>
> Hi,
> We are encountering strange behaviour with Airflow Scheduler since our
> upgrade from Apache Airflow 1.8 to 1.10.
> We have configured 4 workers on 4 servers which all can treat 6 tasks
> simultaneously.
> We notice that after a period (not yet determined), the scheduler seems to
> stop scheduling at is maximum capacity of this 24 tasks so delayed our
> process.
> To troubleshoot this behaviour we've configured the scheduler reboot every 3
> hours.
> I've also notice following informations in
> [https://github.com/apache/incubator-airflow/blob/master/UPDATING.md]
> [...]
> _The scheduler.min_file_parsing_loop_time config option has been temporarily
> removed due to some bugs._
> [...]
> and find corresponding value uncommented and set to 1 in airflow.cfg in
> [scheduler] section.
> Does this bug keep pending? Is-it related to our issue? Do we have to comment
> it?
> Thanks for your help.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
