[jira] [Commented] (AIRFLOW-2763) No precheck mechanism in place during worker initialisation for the connection to metadata database
[
https://issues.apache.org/jira/browse/AIRFLOW-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575617#comment-16575617
]
ASF GitHub Bot commented on AIRFLOW-2763:
-
r39132 closed pull request #3726: [AIRFLOW-2763] Add check to validate worker
connectivity to metadata …
URL: https://github.com/apache/incubator-airflow/pull/3726
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 45b7903d3e..e22427cf40 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -955,6 +955,11 @@ def worker(args):
env = os.environ.copy()
env['AIRFLOW_HOME'] = settings.AIRFLOW_HOME
+if not settings.validate_session():
+log = LoggingMixin().log
+log.error("Worker exiting... database connection precheck failed! ")
+sys.exit(1)
+
# Celery worker
from airflow.executors.celery_executor import app as celery_app
from celery.bin import worker
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 4f1f0df383..9d240b8323 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -173,6 +173,9 @@ killed_task_cleanup_time = 60
# `airflow trigger_dag -c`, the key-value pairs will override the existing
ones in params.
dag_run_conf_overrides_params = False
+# Worker initialisation check to validate Metadata Database connection
+worker_precheck = False
+
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
diff --git a/airflow/config_templates/default_test.cfg
b/airflow/config_templates/default_test.cfg
index 01696c6906..06937452b0 100644
--- a/airflow/config_templates/default_test.cfg
+++ b/airflow/config_templates/default_test.cfg
@@ -51,6 +51,7 @@ enable_xcom_pickling = False
killed_task_cleanup_time = 5
secure_mode = False
hostname_callable = socket:getfqdn
+worker_precheck = False
[cli]
api_client = airflow.api.client.local_client
diff --git a/airflow/settings.py b/airflow/settings.py
index 7d660ab5e6..098164cdc7 100644
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -28,7 +28,7 @@
import pendulum
import socket
-from sqlalchemy import create_engine
+from sqlalchemy import create_engine, exc
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.pool import NullPool
@@ -216,6 +216,26 @@ def configure_adapters():
pass
+def validate_session():
+try:
+worker_precheck = conf.getboolean('core', 'worker_precheck')
+except conf.AirflowConfigException:
+worker_precheck = False
+if not worker_precheck:
+return True
+else:
+check_session = sessionmaker(bind=engine)
+session = check_session()
+try:
+session.execute("select 1")
+conn_status = True
+except exc.DBAPIError as err:
+log.error(err)
+conn_status = False
+session.close()
+return conn_status
+
+
def configure_action_logging():
"""
Any additional configuration (register callback) for
airflow.utils.action_loggers
diff --git a/tests/cli/test_worker_initialisation.py
b/tests/cli/test_worker_initialisation.py
new file mode 100644
index 00..477221693a
--- /dev/null
+++ b/tests/cli/test_worker_initialisation.py
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+import sqlalchemy
+import airflow
+from argparse import Namespace
+
+try:
+from unittest import mock
+except ImportError:
+try:
+import mock
+except ImportError:
+mock = None
+from mock import patch
+
+patch('airflow.utils.cli.action_logging', lambda x: x).start()
+from airflow.bin import cli # noqa
+mock_args = Namespace(queues=1, concurrency=1)
+
+
+class
[jira] [Commented] (AIRFLOW-2763) No precheck mechanism in place during worker initialisation for the connection to metadata database
[ https://issues.apache.org/jira/browse/AIRFLOW-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575618#comment-16575618 ] ASF subversion and git services commented on AIRFLOW-2763: -- Commit c7551f680a52b8b0242aa18872758693bda72955 in incubator-airflow's branch refs/heads/master from [~aimohan] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c7551f6 ] [AIRFLOW-2763] Add check to validate worker connectivity to metadata Database > No precheck mechanism in place during worker initialisation for the > connection to metadata database > --- > > Key: AIRFLOW-2763 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2763 > Project: Apache Airflow > Issue Type: Improvement > Components: db, worker >Reporter: Aishwarya Mohan >Assignee: Aishwarya Mohan >Priority: Major > > Currently when airflow worker is started, no initialization check is in place > to see if the connection to metadata database is healthy or not. The worker > threads remain alive(after throwing exceptions - OperationalError) leading to > a misconception that the worker is up and running the tasks allocated. > In the case of worker interaction with the broker, a reconnection mechanism > exists the exits the worker due to timeout operational error if it failed to > establish connection within the configured number of > attempts(connection_max_retries). > An exit/shutdown mechanism needs to be included during worker initialization > incase the connection to metadata DB is not established. This will assure > the airflow users that the connection to database is in place before they > proceed with task scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2763) No precheck mechanism in place during worker initialisation for the connection to metadata database
[
https://issues.apache.org/jira/browse/AIRFLOW-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575506#comment-16575506
]
ASF GitHub Bot commented on AIRFLOW-2763:
-
amohan34 opened a new pull request #3726: [AIRFLOW-2763] Add check to validate
worker connectivity to metadata …
URL: https://github.com/apache/incubator-airflow/pull/3726
…Database
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-2763\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-2763
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-2763\], code changes always need a Jira issue.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Run the following commands from within tests/cli
nosetests test_worker_initialisation
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> No precheck mechanism in place during worker initialisation for the
> connection to metadata database
> ---
>
> Key: AIRFLOW-2763
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2763
> Project: Apache Airflow
> Issue Type: Improvement
> Components: db, worker
>Reporter: Aishwarya Mohan
>Assignee: Aishwarya Mohan
>Priority: Major
>
> Currently when airflow worker is started, no initialization check is in place
> to see if the connection to metadata database is healthy or not. The worker
> threads remain alive(after throwing exceptions - OperationalError) leading to
> a misconception that the worker is up and running the tasks allocated.
> In the case of worker interaction with the broker, a reconnection mechanism
> exists the exits the worker due to timeout operational error if it failed to
> establish connection within the configured number of
> attempts(connection_max_retries).
> An exit/shutdown mechanism needs to be included during worker initialization
> incase the connection to metadata DB is not established. This will assure
> the airflow users that the connection to database is in place before they
> proceed with task scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
