[
https://issues.apache.org/jira/browse/AIRFLOW-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842535#comment-15842535
]
ASF subversion and git services commented on AIRFLOW-812:
---------------------------------------------------------
Commit 1060ade28119a5e1eb81fc331689bffd37b351b3 in incubator-airflow's branch
refs/heads/v1-8-test from [~farlue]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1060ade ]
[AIRFLOW-812] Fix the scheduler termination bug.
When checking max-runs for dag_files the scheduler
would stop immediately if there were no files.
Closes #2027 from fenglu-g/master
(cherry picked from commit f8d8ddac08dbec90b240a6b8670be9bd650dfb3b)
Signed-off-by: Bolke de Bruin <bo...@xs4all.nl>
> Scheduler job terminates when there is no dag file despite run_numbers and
> run_duration set to unlimited.
> ---------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-812
> URL: https://issues.apache.org/jira/browse/AIRFLOW-812
> Project: Apache Airflow
> Issue Type: Bug
> Components: utils
> Reporter: Feng Lu
> Assignee: Feng Lu
> Fix For: 1.8.1
>
>
> When scheduler run_numbers and run_duration are set to -1 (unlimited), we
> expect the scheduler to run continuously. However, when dag folder is empty,
> the scheduler terminates immediately. This is due to a bug in
> utils/dag_processing when checking max_runs_reached.
> Details:
> airflow scheduler
> [2017-01-27 08:20:50,433] {__init__.py:57} INFO - Using executor
> SequentialExecutor
> [2017-01-27 08:20:50,508] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-01-27 08:20:50,529] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> ____________ _____________
> ____ |__( )_________ __/__ /________ __
> ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
> ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
> _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
> [2017-01-27 08:20:50,771] {jobs.py:514} ERROR - Cannot use more than 1 thread
> when using sqlite. Setting max_threads to 1
> [2017-01-27 08:20:50,786] {jobs.py:1222} INFO - Starting the scheduler
> [2017-01-27 08:20:50,787] {jobs.py:1238} INFO - Processing files using up to
> 1 processes at a time
> [2017-01-27 08:20:50,787] {jobs.py:1240} INFO - Running execute loop for -1
> seconds
> [2017-01-27 08:20:50,787] {jobs.py:1242} INFO - Processing each file at most
> None times
> [2017-01-27 08:20:50,787] {jobs.py:1244} INFO - Process each file at most
> once every 0 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1246} INFO - Checking for new files in
> /home/fenglu/airflow/dags every 300 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1249} INFO - Searching for files in
> /home/fenglu/airflow/dags
> [2017-01-27 08:20:50,787] {jobs.py:1252} INFO - There are 0 files in
> /home/fenglu/airflow/dags
> [2017-01-27 08:20:50,787] {jobs.py:1314} INFO - Resetting state for orphaned
> tasks
> [2017-01-27 08:20:50,789] {jobs.py:1326} INFO - Removing old import errors
> [2017-01-27 08:20:50,795] {jobs.py:1363} INFO - Heartbeating the process
> manager
> [2017-01-27 08:20:50,796] {jobs.py:1400} INFO - Heartbeating the executor
> [2017-01-27 08:20:51,797] {jobs.py:1432} INFO - Exiting loop as all files
> have been processed None times
> [2017-01-27 08:20:51,797] {jobs.py:1448} INFO - Deactivating DAGs that
> haven't been touched since 2017-01-27T08:20:50.795782
> [2017-01-27 08:20:51,801] {jobs.py:1271} INFO - Exited execute loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
