KulykDmytro opened a new issue #11629:
URL: https://github.com/apache/airflow/issues/11629
**Apache Airflow version**: 1.10.12
**Kubernetes version**: v1.18.6
**What happened**:
Using KubernetesExecutor with **pod_template** to run tasks
pod log do not contains same info as shown in task log
**What you expected to happen**:
Pod logs should contain all information from respective logs
**Anything else we need to know**:
Pod log
```
[2020-10-18 10:36:25,449] {__init__.py:50} INFO - Using executor
LocalExecutor
[2020-10-18 10:36:25,450] {dagbag.py:417} INFO - Filling up the DagBag from
/opt/airflow/dags/aggregates_daily.py
```
Task log
```
airflow@aggregatesloaderdailycheckpaymentlegacy-a437af6584ed4006a0c62b0:/opt/airflow$
cat
/opt/airflow/logs/aggregates_loader_daily/check_payment_legacy/2020-09-13T02\:00\:00+00\:00/8.log
[2020-10-18 10:36:25,834] {taskinstance.py:670} INFO - Dependencies all met
for <TaskInstance: aggregates_loader_daily.check_payment_legacy
2020-09-13T02:00:00+00:00 [queued]>
[2020-10-18 10:36:25,867] {taskinstance.py:670} INFO - Dependencies all met
for <TaskInstance: aggregates_loader_daily.check_payment_legacy
2020-09-13T02:00:00+00:00 [queued]>
[2020-10-18 10:36:25,868] {taskinstance.py:880} INFO -
--------------------------------------------------------------------------------
[2020-10-18 10:36:25,868] {taskinstance.py:881} INFO - Starting attempt 8 of
9
[2020-10-18 10:36:25,868] {taskinstance.py:882} INFO -
--------------------------------------------------------------------------------
[2020-10-18 10:36:25,890] {taskinstance.py:901} INFO - Executing
<Task(SqlSensor): check_payment_legacy> on 2020-09-13T02:00:00+00:00
[2020-10-18 10:36:25,893] {standard_task_runner.py:54} INFO - Started
process 15 to run task
[2020-10-18 10:36:25,915] {standard_task_runner.py:77} INFO - Running:
['airflow', 'run', 'aggregates_loader_daily', 'check_payment_legacy',
'2020-09-13T02:00:00+00:00', '--job_id', '10678', '--pool', 'default_pool',
'--raw', '-sd', 'DAGS_FOLDER/aggregates_daily.py', '--cfg_path',
'/tmp/tmp6cm9p7nx']
[2020-10-18 10:36:25,916] {standard_task_runner.py:78} INFO - Job 10678:
Subtask check_payment_legacy
[2020-10-18 10:36:25,992] {logging_mixin.py:112} INFO - Running %s on host
%s <TaskInstance: aggregates_loader_daily.check_payment_legacy
2020-09-13T02:00:00+00:00 [running]>
aggregatesloaderdailycheckpaymentlegacy-a437af6584ed4006a0c62b0
[2020-10-18 10:36:26,061] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:36:26,062] {sql_sensor.py:89} INFO - Poking:
select 1
from rawdata.bi_getpaymentssummary
where datepayment >= '2020-09-13'::timestamp
and datepayment < '2020-09-13'::timestamp + '1
day'::interval
limit 1
(with parameters None)
[2020-10-18 10:36:26,080] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:41:26,220] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:41:26,220] {sql_sensor.py:89} INFO - Poking:
select 1
from rawdata.bi_getpaymentssummary
where datepayment >= '2020-09-13'::timestamp
and datepayment < '2020-09-13'::timestamp + '1
day'::interval
limit 1
(with parameters None)
[2020-10-18 10:41:26,245] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:46:26,397] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:46:26,397] {sql_sensor.py:89} INFO - Poking:
select 1
from rawdata.bi_getpaymentssummary
where datepayment >= '2020-09-13'::timestamp
and datepayment < '2020-09-13'::timestamp + '1
day'::interval
limit 1
(with parameters None)
[2020-10-18 10:46:26,419] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:51:26,566] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:51:26,566] {sql_sensor.py:89} INFO - Poking:
select 1
from rawdata.bi_getpaymentssummary
where datepayment >= '2020-09-13'::timestamp
and datepayment < '2020-09-13'::timestamp + '1
day'::interval
limit 1
(with parameters None)
[2020-10-18 10:51:26,586] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:56:26,729] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
[2020-10-18 10:56:26,729] {sql_sensor.py:89} INFO - Poking:
select 1
from rawdata.bi_getpaymentssummary
where datepayment >= '2020-09-13'::timestamp
and datepayment < '2020-09-13'::timestamp + '1
day'::interval
limit 1
(with parameters None)
[2020-10-18 10:56:26,749] {base_hook.py:89} INFO - Using connection to: id:
dwh. Host: dev.pgsql-dwh.betlab.private, Port: 5432, Schema: dwh, Login:
airflow_user, Password: XXXXXXXX, extra: XXXXXXXX
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
