minnieshi edited a comment on issue #13542:
URL: https://github.com/apache/airflow/issues/13542#issuecomment-821356514
@SalmonTimo i have access to the workers, as well as the dag logs folder
(which is saved in the file share -PVC, and can be viewed via azure storage
explorer).
ps. The environment is set up new, and migrated a few tables listed below
from old environment. during debug of this stuck situation, the table 'dag',
'task_*', 'celery_*' had been truncated.
```
- celeray_taskmeta
- dag
- dag_run
- log
- task_fail
- task_instance
- task_reschedule
- connections
```
How would like me to fetch the log? Just the dag log run?
update: the one i can view via storage explorer - it is empty, since the
task was not executed. i will check the worker itself. and update this comment.
there are no errors
```
$ kubectl -n airflow get pods
NAME READY STATUS RESTARTS
AGE
airflow-scheduler-55549d985f-dbw7b 1/1 Running 0
134m
airflow-web-6c8467fd74-9zkj5 1/1 Running 0
134m
airflow-worker-0 1/1 Running 0
115m
airflow-worker-1 1/1 Running 0
125m
nginx-ingress-nginx-controller-6d5794678d-8zdsl 1/1 Running 0
5h16m
nginx-ingress-nginx-controller-6d5794678d-fzhwb 1/1 Running 0
5h15m
telegraf-54cd7f8578-fgdrn 1/1 Running 0
5h16m
DKCPHMAC137:instructions mishi$ kubectl -n airflow logs airflow-worker-0
*** installing global extra pip packages...
Collecting flask_oauthlib==0.9.6
Downloading Flask_OAuthlib-0.9.6-py3-none-any.whl (40 kB)
Collecting cachelib
Downloading cachelib-0.1.1-py3-none-any.whl (13 kB)
Collecting requests-oauthlib<1.2.0,>=0.6.2
Downloading requests_oauthlib-1.1.0-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: Flask in
/home/airflow/.local/lib/python3.7/site-packages (from flask_oauthlib==0.9.6)
(1.1.2)
Collecting oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2
Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
Requirement already satisfied: requests>=2.0.0 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6) (2.23.0)
Requirement already satisfied: click>=5.1 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (6.7)
Requirement already satisfied: Werkzeug>=0.15 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (0.16.1)
Requirement already satisfied: itsdangerous>=0.24 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (2.11.2)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6)
(1.25.11)
Requirement already satisfied: certifi>=2017.4.17 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6)
(2020.11.8)
Requirement already satisfied: idna<3,>=2.5 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6) (2.8)
Requirement already satisfied: chardet<4,>=3.0.2 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6) (3.0.4)
Requirement already satisfied: MarkupSafe>=0.23 in
/home/airflow/.local/lib/python3.7/site-packages (from
Jinja2>=2.10.1->Flask->flask_oauthlib==0.9.6) (1.1.1)
Installing collected packages: cachelib, oauthlib, requests-oauthlib,
flask-oauthlib
Attempting uninstall: oauthlib
Found existing installation: oauthlib 3.1.0
Uninstalling oauthlib-3.1.0:
Successfully uninstalled oauthlib-3.1.0
Attempting uninstall: requests-oauthlib
Found existing installation: requests-oauthlib 1.3.0
Uninstalling requests-oauthlib-1.3.0:
Successfully uninstalled requests-oauthlib-1.3.0
Successfully installed cachelib-0.1.1 flask-oauthlib-0.9.6 oauthlib-2.1.0
requests-oauthlib-1.1.0
WARNING: You are using pip version 20.2.4; however, version 21.0.1 is
available.
You should consider upgrading via the '/usr/local/bin/python -m pip install
--upgrade pip' command.
*** running worker...
[2021-04-16 16:03:50,914] {settings.py:233} DEBUG - Setting up DB connection
pool (PID 7)
[2021-04-16 16:03:50,914] {settings.py:300} DEBUG -
settings.prepare_engine_args(): Using pool settings. pool_size=5,
max_overflow=10, pool_recycle=1800, pid=7
[2021-04-16 16:03:51,233] {sentry.py:179} DEBUG - Could not configure
Sentry: No module named 'blinker', using DummySentry instead.
[2021-04-16 16:03:51,286] {__init__.py:45} DEBUG - Cannot import due to
doesn't look like a module path
[2021-04-16 16:03:51,685] {cli_action_loggers.py:42} DEBUG - Adding
<function default_action_log at 0x7f7cda6afd40> to pre execution callback
The 'worker' command is deprecated and removed in Airflow 2.0, please use
'celery worker' instead
[2021-04-16 16:03:52,608] {cli_action_loggers.py:68} DEBUG - Calling
callbacks: [<function default_action_log at 0x7f7cda6afd40>]
[2021-04-16 16:03:53,288: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2021-04-16 16:03:53,293: DEBUG/MainProcess] | Worker: Building graph...
[2021-04-16 16:03:53,293: DEBUG/MainProcess] | Worker: New boot order:
{Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2021-04-16 16:03:53,316: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2021-04-16 16:03:53,317: DEBUG/MainProcess] | Consumer: Building graph...
[2021-04-16 16:03:53,393: DEBUG/MainProcess] | Consumer: New boot order:
{Connection, Events, Mingle, Gossip, Agent, Tasks, Control, Heart, event loop}
[2021-04-16 16:03:53,427: DEBUG/MainProcess] | Worker: Starting Hub
[2021-04-16 16:03:53,427: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:53,427: DEBUG/MainProcess] | Worker: Starting Pool
[2021-04-16 16:03:53,749] {settings.py:233} DEBUG - Setting up DB connection
pool (PID 17)
[2021-04-16 16:03:53,750] {settings.py:300} DEBUG -
settings.prepare_engine_args(): Using pool settings. pool_size=5,
max_overflow=10, pool_recycle=1800, pid=17
[2021-04-16 16:03:53,951] {sentry.py:179} DEBUG - Could not configure
Sentry: No module named 'blinker', using DummySentry instead.
[2021-04-16 16:03:53,998] {__init__.py:45} DEBUG - Cannot import due to
doesn't look like a module path
[2021-04-16 16:03:54,013: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:54,013: DEBUG/MainProcess] | Worker: Starting Autoscaler
[2021-04-16 16:03:54,014: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:54,014: DEBUG/MainProcess] | Worker: Starting Consumer
[2021-04-16 16:03:54,014: DEBUG/MainProcess] | Consumer: Starting Connection
[2021-04-16 16:03:54,081: INFO/MainProcess] Connected to
redis://redis:**@redis-airflow-osweu-dev.redis.cache.windows.net:6379//
[2021-04-16 16:03:54,081: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:54,081: DEBUG/MainProcess] | Consumer: Starting Events
[2021-04-16 16:03:54,101: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:54,101: DEBUG/MainProcess] | Consumer: Starting Mingle
[2021-04-16 16:03:54,102: INFO/MainProcess] mingle: searching for neighbors
[2021-04-16 16:03:54,328] {cli_action_loggers.py:42} DEBUG - Adding
<function default_action_log at 0x7fb8a5c12290> to pre execution callback
[2021-04-16 16:03:54,603] {cli_action_loggers.py:68} DEBUG - Calling
callbacks: [<function default_action_log at 0x7fb8a5c12290>]
Starting flask
* Serving Flask app "airflow.bin.cli" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production
deployment.
Use a production WSGI server instead.
* Debug mode: off
[2021-04-16 16:03:54,954] {_internal.py:122} INFO - * Running on
http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2021-04-16 16:03:55,194: INFO/MainProcess] mingle: sync with 1 nodes
[2021-04-16 16:03:55,195: DEBUG/MainProcess] mingle: processing reply from
celery@airflow-worker-1
[2021-04-16 16:03:55,195: INFO/MainProcess] mingle: sync complete
[2021-04-16 16:03:55,195: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:55,195: DEBUG/MainProcess] | Consumer: Starting Gossip
[2021-04-16 16:03:55,229: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:55,230: DEBUG/MainProcess] | Consumer: Starting Tasks
[2021-04-16 16:03:55,240: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:55,240: DEBUG/MainProcess] | Consumer: Starting Control
[2021-04-16 16:03:55,266: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:55,267: DEBUG/MainProcess] | Consumer: Starting Heart
[2021-04-16 16:03:55,280: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 16:03:55,280: DEBUG/MainProcess] | Consumer: Starting event loop
[2021-04-16 16:03:55,281: DEBUG/MainProcess] | Worker: Hub.register
Autoscaler...
[2021-04-16 16:03:55,281: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2021-04-16 16:03:55,281: INFO/MainProcess] celery@airflow-worker-0 ready.
[2021-04-16 16:03:55,281: DEBUG/MainProcess] basic.qos: prefetch_count->8
[2021-04-16 16:03:56,928: DEBUG/MainProcess] celery@airflow-worker-1 joined
the party
**$ kubectl -n airflow logs airflow-worker-1**
*** installing global extra pip packages...
Collecting flask_oauthlib==0.9.6
Downloading Flask_OAuthlib-0.9.6-py3-none-any.whl (40 kB)
Collecting requests-oauthlib<1.2.0,>=0.6.2
Downloading requests_oauthlib-1.1.0-py2.py3-none-any.whl (21 kB)
Collecting oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2
Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
Collecting cachelib
Downloading cachelib-0.1.1-py3-none-any.whl (13 kB)
Requirement already satisfied: Flask in
/home/airflow/.local/lib/python3.7/site-packages (from flask_oauthlib==0.9.6)
(1.1.2)
Requirement already satisfied: requests>=2.0.0 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6) (2.23.0)
Requirement already satisfied: itsdangerous>=0.24 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (2.11.2)
Requirement already satisfied: Werkzeug>=0.15 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (0.16.1)
Requirement already satisfied: click>=5.1 in
/home/airflow/.local/lib/python3.7/site-packages (from
Flask->flask_oauthlib==0.9.6) (6.7)
Requirement already satisfied: certifi>=2017.4.17 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6)
(2020.11.8)
Requirement already satisfied: idna<3,>=2.5 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6)
(1.25.11)
Requirement already satisfied: chardet<4,>=3.0.2 in
/home/airflow/.local/lib/python3.7/site-packages (from
requests>=2.0.0->requests-oauthlib<1.2.0,>=0.6.2->flask_oauthlib==0.9.6) (3.0.4)
Requirement already satisfied: MarkupSafe>=0.23 in
/home/airflow/.local/lib/python3.7/site-packages (from
Jinja2>=2.10.1->Flask->flask_oauthlib==0.9.6) (1.1.1)
Installing collected packages: oauthlib, requests-oauthlib, cachelib,
flask-oauthlib
Attempting uninstall: oauthlib
Found existing installation: oauthlib 3.1.0
Uninstalling oauthlib-3.1.0:
Successfully uninstalled oauthlib-3.1.0
Attempting uninstall: requests-oauthlib
Found existing installation: requests-oauthlib 1.3.0
Uninstalling requests-oauthlib-1.3.0:
Successfully uninstalled requests-oauthlib-1.3.0
Successfully installed cachelib-0.1.1 flask-oauthlib-0.9.6 oauthlib-2.1.0
requests-oauthlib-1.1.0
WARNING: You are using pip version 20.2.4; however, version 21.0.1 is
available.
You should consider upgrading via the '/usr/local/bin/python -m pip install
--upgrade pip' command.
*** running worker...
[2021-04-16 15:53:39,557] {settings.py:233} DEBUG - Setting up DB connection
pool (PID 6)
[2021-04-16 15:53:39,558] {settings.py:300} DEBUG -
settings.prepare_engine_args(): Using pool settings. pool_size=5,
max_overflow=10, pool_recycle=1800, pid=6
[2021-04-16 15:53:39,926] {sentry.py:179} DEBUG - Could not configure
Sentry: No module named 'blinker', using DummySentry instead.
[2021-04-16 15:53:39,982] {__init__.py:45} DEBUG - Cannot import due to
doesn't look like a module path
[2021-04-16 15:53:40,446] {cli_action_loggers.py:42} DEBUG - Adding
<function default_action_log at 0x7f424221bd40> to pre execution callback
The 'worker' command is deprecated and removed in Airflow 2.0, please use
'celery worker' instead
[2021-04-16 15:53:41,458] {cli_action_loggers.py:68} DEBUG - Calling
callbacks: [<function default_action_log at 0x7f424221bd40>]
[2021-04-16 15:53:42,387: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2021-04-16 15:53:42,393: DEBUG/MainProcess] | Worker: Building graph...
[2021-04-16 15:53:42,393: DEBUG/MainProcess] | Worker: New boot order:
{StateDB, Beat, Timer, Hub, Pool, Autoscaler, Consumer}
[2021-04-16 15:53:42,418: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2021-04-16 15:53:42,419: DEBUG/MainProcess] | Consumer: Building graph...
[2021-04-16 15:53:42,517: DEBUG/MainProcess] | Consumer: New boot order:
{Connection, Events, Mingle, Gossip, Heart, Tasks, Control, Agent, event loop}
[2021-04-16 15:53:42,556: DEBUG/MainProcess] | Worker: Starting Hub
[2021-04-16 15:53:42,556: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:42,556: DEBUG/MainProcess] | Worker: Starting Pool
[2021-04-16 15:53:42,904] {settings.py:233} DEBUG - Setting up DB connection
pool (PID 16)
[2021-04-16 15:53:42,905] {settings.py:300} DEBUG -
settings.prepare_engine_args(): Using pool settings. pool_size=5,
max_overflow=10, pool_recycle=1800, pid=16
[2021-04-16 15:53:43,085] {sentry.py:179} DEBUG - Could not configure
Sentry: No module named 'blinker', using DummySentry instead.
[2021-04-16 15:53:43,144] {__init__.py:45} DEBUG - Cannot import due to
doesn't look like a module path
[2021-04-16 15:53:43,172: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:43,172: DEBUG/MainProcess] | Worker: Starting Autoscaler
[2021-04-16 15:53:43,172: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:43,172: DEBUG/MainProcess] | Worker: Starting Consumer
[2021-04-16 15:53:43,173: DEBUG/MainProcess] | Consumer: Starting Connection
[2021-04-16 15:53:43,253: INFO/MainProcess] Connected to
redis://redis:**@redis-airflow-osweu-dev.redis.cache.windows.net:6379//
[2021-04-16 15:53:43,253: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:43,253: DEBUG/MainProcess] | Consumer: Starting Events
[2021-04-16 15:53:43,300: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:43,300: DEBUG/MainProcess] | Consumer: Starting Mingle
[2021-04-16 15:53:43,300: INFO/MainProcess] mingle: searching for neighbors
[2021-04-16 15:53:43,533] {cli_action_loggers.py:42} DEBUG - Adding
<function default_action_log at 0x7fe2c52ec290> to pre execution callback
[2021-04-16 15:53:43,860] {cli_action_loggers.py:68} DEBUG - Calling
callbacks: [<function default_action_log at 0x7fe2c52ec290>]
Starting flask
* Serving Flask app "airflow.bin.cli" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production
deployment.
Use a production WSGI server instead.
* Debug mode: off
[2021-04-16 15:53:44,132] {_internal.py:122} INFO - * Running on
http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2021-04-16 15:53:44,382: INFO/MainProcess] mingle: sync with 1 nodes
[2021-04-16 15:53:44,383: DEBUG/MainProcess] mingle: processing reply from
celery@airflow-worker-0
[2021-04-16 15:53:44,383: INFO/MainProcess] mingle: sync complete
[2021-04-16 15:53:44,383: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:44,383: DEBUG/MainProcess] | Consumer: Starting Gossip
[2021-04-16 15:53:44,413: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:44,414: DEBUG/MainProcess] | Consumer: Starting Heart
[2021-04-16 15:53:44,431: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:44,431: DEBUG/MainProcess] | Consumer: Starting Tasks
[2021-04-16 15:53:44,439: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:44,440: DEBUG/MainProcess] | Consumer: Starting Control
[2021-04-16 15:53:44,473: DEBUG/MainProcess] ^-- substep ok
[2021-04-16 15:53:44,473: DEBUG/MainProcess] | Consumer: Starting event loop
[2021-04-16 15:53:44,473: DEBUG/MainProcess] | Worker: Hub.register
Autoscaler...
[2021-04-16 15:53:44,473: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2021-04-16 15:53:44,474: INFO/MainProcess] celery@airflow-worker-1 ready.
[2021-04-16 15:53:44,474: DEBUG/MainProcess] basic.qos: prefetch_count->8
[2021-04-16 15:53:45,888: DEBUG/MainProcess] celery@airflow-worker-0 joined
the party
[2021-04-16 16:03:34,318: DEBUG/MainProcess] celery@airflow-worker-0 left
[2021-04-16 16:03:54,170: DEBUG/MainProcess] pidbox received method
hello(from_node='celery@airflow-worker-0', revoked={}) [reply_to:{'exchange':
'reply.celery.pidbox', 'routing_key': '86623118-5415-3478-a9be-f48f1670b61b'}
ticket:07d58af6-f3f2-4404-a926-94eab1e2955c]
[2021-04-16 16:03:54,170: INFO/MainProcess] sync with celery@airflow-worker-0
[2021-04-16 16:03:55,281: DEBUG/MainProcess] celery@airflow-worker-0 joined
the party
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
