robintian001 commented on issue #38252:
URL: https://github.com/apache/airflow/issues/38252#issuecomment-2006169931
**The Scheduler's logs**
2024-03-16 17:11:08,194 INFO - Adopting or resetting orphaned tasks for
active dag runs
2024-03-16 17:11:22,864 ERROR - Exception when executing
SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3371, in _wrap_pool_connect
return fn()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 327, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 498, in checkout
rec._checkin_failed(err, _fairy_was_created=False)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
dbapi_connection = rec.get_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 630, in get_connection
self.__connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 691, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py",
line 574, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/__init__.py",
line 121, in Connect
return Connection(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py",
line 193, in __init__
super().__init__(*args, **kwargs2)
MySQLdb.OperationalError: (2003, "Can't connect to MySQL server on
'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 845, in _execute
self._run_scheduler_loop()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 979, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1053, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py",
line 91, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger,
**retry_kwargs):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/__init__.py",
line 347, in __iter__
do = self.iter(retry_state=retry_state)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/__init__.py",
line 325, in iter
raise retry_exc.reraise()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/__init__.py",
line 158, in reraise
raise self.last_attempt.result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in
result
return self.__get_result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in
__get_result
raise self._exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py",
line 100, in wrapped_function
return func(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1120, in _create_dagruns_for_dags
query, dataset_triggered_dag_info =
DagModel.dags_needing_dagruns(session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/models/dag.py",
line 3615, in dags_needing_dagruns
for x in session.execute(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1716, in execute
conn = self._connection_for_bind(bind)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1555, in _connection_for_bind
return self._transaction._connection_for_bind(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 750, in _connection_for_bind
conn = bind.connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/future/engine.py",
line 406, in connect
return super(Engine, self).connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 96, in __init__
else engine.raw_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3374, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 2208, in _handle_dbapi_exception_noconnection
util.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3371, in _wrap_pool_connect
return fn()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 327, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 498, in checkout
rec._checkin_failed(err, _fairy_was_created=False)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
dbapi_connection = rec.get_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 630, in get_connection
self.__connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 691, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py",
line 574, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/__init__.py",
line 121, in Connect
return Connection(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py",
line 193, in __init__
super().__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2003, "Can't
connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306'
(111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2024-03-16 17:11:22,868 INFO - Shutting down LocalExecutor; waiting for
running tasks to finish. Signal again if you don't want to wait.
2024-03-16 17:11:22,910 INFO - Sending Signals.SIGTERM to group 1732789.
PIDs of all processes in the group: []
2024-03-16 17:11:22,910 INFO - Sending the signal Signals.SIGTERM to group
1732789
2024-03-16 17:11:22,910 INFO - Sending the signal Signals.SIGTERM to process
1732789 as process group is missing.
2024-03-16 17:11:22,910 INFO - Exited execute loop
2024-03-16 17:11:23,161 ERROR - Exception when running scheduler job
Traceback (most recent call last):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3371, in _wrap_pool_connect
return fn()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 327, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 498, in checkout
rec._checkin_failed(err, _fairy_was_created=False)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
dbapi_connection = rec.get_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 630, in get_connection
self.__connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 691, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py",
line 574, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/__init__.py",
line 121, in Connect
return Connection(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py",
line 193, in __init__
super().__init__(*args, **kwargs2)
MySQLdb.OperationalError: (2003, "Can't connect to MySQL server on
'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py",
line 289, in run_job
return execute_job(job, execute_callable=execute_callable)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py",
line 318, in execute_job
ret = execute_callable()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 845, in _execute
self._run_scheduler_loop()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 979, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1053, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py",
line 91, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger,
**retry_kwargs):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/__init__.py",
line 347, in __iter__
do = self.iter(retry_state=retry_state)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/__init__.py",
line 325, in iter
raise retry_exc.reraise()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/__init__.py",
line 158, in reraise
raise self.last_attempt.result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in
result
return self.__get_result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in
__get_result
raise self._exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py",
line 100, in wrapped_function
return func(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1120, in _create_dagruns_for_dags
query, dataset_triggered_dag_info =
DagModel.dags_needing_dagruns(session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/models/dag.py",
line 3615, in dags_needing_dagruns
for x in session.execute(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1716, in execute
conn = self._connection_for_bind(bind)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1555, in _connection_for_bind
return self._transaction._connection_for_bind(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 750, in _connection_for_bind
conn = bind.connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/future/engine.py",
line 406, in connect
return super(Engine, self).connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 96, in __init__
else engine.raw_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3374, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 2208, in _handle_dbapi_exception_noconnection
util.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3371, in _wrap_pool_connect
return fn()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 327, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 498, in checkout
rec._checkin_failed(err, _fairy_was_created=False)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
dbapi_connection = rec.get_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 630, in get_connection
self.__connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 691, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py",
line 574, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/__init__.py",
line 121, in Connect
return Connection(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py",
line 193, in __init__
super().__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2003, "Can't
connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306'
(111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3371, in _wrap_pool_connect
return fn()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 327, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 498, in checkout
rec._checkin_failed(err, _fairy_was_created=False)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
dbapi_connection = rec.get_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 630, in get_connection
self.__connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 691, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py",
line 574, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/__init__.py",
line 121, in Connect
return Connection(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py",
line 193, in __init__
super().__init__(*args, **kwargs2)
MySQLdb.OperationalError: (2003, "Can't connect to MySQL server on
'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py",
line 47, in _run_scheduler_job
run_job(job=job_runner.job, execute_callable=job_runner._execute)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/session.py",
line 77, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py",
line 291, in run_job
job.complete_execution(session=session)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/session.py",
line 74, in wrapper
return func(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py",
line 237, in complete_execution
session.merge(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3056, in merge
return self._merge(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3136, in _merge
merged = self.get(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 2853, in get
return self._get_impl(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 2975, in _get_impl
return db_load_fn(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py",
line 530, in load_on_pk_identity
session.execute(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1716, in execute
conn = self._connection_for_bind(bind)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1555, in _connection_for_bind
return self._transaction._connection_for_bind(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 750, in _connection_for_bind
conn = bind.connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/future/engine.py",
line 406, in connect
return super(Engine, self).connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 96, in __init__
else engine.raw_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3374, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 2208, in _handle_dbapi_exception_noconnection
util.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 3371, in _wrap_pool_connect
return fn()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 327, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 498, in checkout
rec._checkin_failed(err, _fairy_was_created=False)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
dbapi_connection = rec.get_connection()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 630, in get_connection
self.__connect()
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 691, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py",
line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py",
line 574, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/__init__.py",
line 121, in Connect
return Connection(*args, **kwargs)
File
"/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py",
line 193, in __init__
super().__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2003, "Can't
connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306'
(111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
---------------------------------------------------------------------------------------------
**this message from microsoft support team**
It can be seen from the background connection log that your standby server
restarts at 17:00 UTC on 3/16 for regular maintenance, and restarts at 17:07
UTC on 3/16 for about 7 minutes. In this case, your standby server will be
failover to the new primary server for you to connect. At the same time, your
original primary server restarts at 17:11 UTC on 3/16 and completes the restart
at 17:19 UTC on 3/16, which takes about 8 minutes. From the connection log,
your initial standby server restarts before the initial primary server, so your
standby server that has restarted can connect before your initial primary
server restarts.
From the figure below about the conversion of HA server, you can see that at
17:12 UTC on 3/16, your HA server has been switched, which means that at this
time your original standby server has become your new primary server available
for connection. As you can see from the following figure, your total failover
time is about 2 minutes. If you feel that you are not connected for about 20
minutes, it may be related to the retry mechanism of your client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
