rodrigo-morais commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876388007
Pods before to deploy a new scheduler
```
NAME READY
STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1
Running 0 3m57s
airflow-web-74f56b9df-hvpzz 2/2
Running 0 3m57s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1
Running 0 40s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1
Running 0 40s
dbt-docs-647db48b7d-vpj4p 1/1
Running 0 3m57s
```
Pods after to deploy the scheduler
```
NAME READY
STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1
Running 0 4m6s
airflow-scheduler-776f94956b-s4dfv 0/1
ContainerCreating 0 5s
airflow-web-5c667d85db-kfcx6 0/2
ContainerCreating 0 2s
airflow-web-74f56b9df-hvpzz 2/2
Running 0 4m6s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1
Running 0 49s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1
Running 0 49s
dbt-docs-647db48b7d-vpj4p 1/1
Running 0 4m6s
```
Pods after the current scheduler have been terminated and the new one has
been created
```
NAME
READY STATUS RESTARTS AGE
airflow-scheduler-776f94956b-s4dfv
1/1 Running 0 3m6s
airflow-web-5c667d85db-kfcx6
2/2 Running 0 3m3s
ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
0/1 Error 0 2m16s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532
0/1 Completed 0 3m50s
dbt-docs-647db48b7d-vpj4p
1/1 Running 0 7m7s
```
<details><summary>Pod with error</summary>Running <TaskInstance:
as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]>
on host ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line
40, in main
args.func(args)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line
91, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 237, in task_run
_run_task_by_selected_method(args, dag, ti)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py",
line 120, in _run_task_by_local_task_job
run_job.run()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py",
line 237, in run
self._execute()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py",
line 133, in _execute
self.heartbeat()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py",
line 218, in heartbeat
self.heartbeat_callback(session=session)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py",
line 67, in wrapper
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py",
line 207, in heartbeat_callback
ti._run_finished_callback(error=error) # pylint:
disable=protected-access
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1367, in _run_finished_callback
task.on_failure_callback(context)</details>
<details><summary>Pod with scheduler</summary>
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:850:
DeprecationWarning: Specifying both AIRFLOW_HOME environment variable and
airflow_home in the config file is deprecated. Please use only the AIRFLOW_HOME
environment variable and remove the config file entry.
warnings.warn(msg, category=DeprecationWarning)
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-08 11:44:16,768] {dagrun.py:429} ERROR - Marking run <DagRun
as_check_counts_sf_vs_pg @ 2021-07-08 11:42:09.208095+00:00:
manual__2021-07-08T11:42:09.208095+00:00, externally triggered: True> failed
[2021-07-08 11:44:26,873] {kubernetes_executor.py:202} ERROR - Event:
ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 Failed</details>
<details><summary>Log of the task</summary>[2021-07-08 11:43:57,254]
{taskinstance.py:876} INFO - Dependencies all met for <TaskInstance:
as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]>
[2021-07-08 11:43:57,268] {taskinstance.py:876} INFO - Dependencies all met
for <TaskInstance: as_check_counts_sf_vs_pg.contentdb
2021-07-08T11:42:09.208095+00:00 [queued]>
[2021-07-08 11:43:57,269] {taskinstance.py:1067} INFO -
--------------------------------------------------------------------------------
[2021-07-08 11:43:57,269] {taskinstance.py:1068} INFO - Starting attempt 2
of 1
[2021-07-08 11:43:57,269] {taskinstance.py:1069} INFO -
--------------------------------------------------------------------------------
[2021-07-08 11:43:57,293] {taskinstance.py:1087} INFO - Executing
<Task(jw_PgVsSfCountCheck): contentdb> on 2021-07-08T11:42:09.208095+00:00
[2021-07-08 11:43:57,298] {standard_task_runner.py:52} INFO - Started
process 149 to run task
[2021-07-08 11:43:57,305] {standard_task_runner.py:76} INFO - Running:
['airflow', 'tasks', 'run', 'as_check_counts_sf_vs_pg', 'contentdb',
'2021-07-08T11:42:09.208095+00:00', '--job-id', '583317', '--pool',
'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/gen_dags.py', '--cfg-path',
'/tmp/tmpz_sm9qlb', '--error-file', '/tmp/tmpmsipgv0p']
[2021-07-08 11:43:57,307] {standard_task_runner.py:77} INFO - Job 583317:
Subtask contentdb
[2021-07-08 11:43:57,415] {logging_mixin.py:104} INFO - Running
<TaskInstance: as_check_counts_sf_vs_pg.contentdb
2021-07-08T11:42:09.208095+00:00 [running]> on host
ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
[2021-07-08 11:43:57,550] {taskinstance.py:1280} INFO - Exporting the
following env vars:
AIRFLOW_CTX_DAG_OWNER=Rodrigo
AIRFLOW_CTX_DAG_ID=as_check_counts_sf_vs_pg
AIRFLOW_CTX_EXECUTION_DATE=2021-07-08T11:42:09.208095+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-08T11:42:09.208095+00:00
[2021-07-08 11:43:57,611] {connection.py:257} INFO - Snowflake Connector for
Python Version: 2.4.6, Python Version: 3.8.10, Platform:
Linux-5.4.89+-x86_64-with-glibc2.2.5
[2021-07-08 11:43:57,612] {connection.py:868} INFO - This connection is in
OCSP Fail Open Mode. TLS Certificates would be checked for validity and
revocation status. Any other Certificate Revocation related exceptions or OCSP
Responder failures would be disregarded in favor of connectivity.
[2021-07-08 11:43:57,612] {connection.py:886} INFO - Setting
use_openssl_only mode to False
[2021-07-08 11:43:58,015] {cursor.py:637} INFO - query: [SELECT
T.TABLE_NAME, IFF(C.COLUMN_NAME IS NULL, 0, 1) AS HAS_CREATED_AT FROM SHA...]
[2021-07-08 11:43:59,060] {cursor.py:661} INFO - query execution done
[2021-07-08 11:43:59,061] {arrow_result.pyx:0} INFO - fetching data done
[2021-07-08 11:43:59,077] {cursor.py:637} INFO - query: [SELECT
'AGE_CERTIFICATION' AS TABLE_NAME, COUNT(1) AS COUNT, NULL AS CREATED_AT ...]
[2021-07-08 11:43:59,575] {cursor.py:661} INFO - query execution done
[2021-07-08 11:43:59,576] {arrow_result.pyx:0} INFO - fetching data done
[2021-07-08 11:44:17,664] {local_task_job.py:196} WARNING - State of this
instance has been externally set to failed. Terminating instance.
[2021-07-08 11:44:17,666] {process_utils.py:100} INFO - Sending
Signals.SIGTERM to GPID 149
[2021-07-08 11:44:25,218] {taskinstance.py:1264} ERROR - Received SIGTERM.
Terminating subprocesses.
[2021-07-08 11:44:25,276] {process_utils.py:66} INFO - Process
psutil.Process(pid=149, status='terminated', exitcode=0, started='11:43:57')
(149) terminated with exit code 0</details>
The UI:
The previous and these logs don't have any error in the DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
