[
https://issues.apache.org/jira/browse/AIRFLOW-7078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063565#comment-17063565
]
Oleg Korsak commented on AIRFLOW-7078:
--------------------------------------
After my investigation I've found that logs are not being overwritten after
task is completed. My logging records are just missing even while sleep(5*60)
and of course after that. Here is the sample DAG:
{code:java}
import logging.config
import os
from datetime import datetime, timedelta
from time import sleepfrom airflow import DAG
from airflow.operators.python_operator import PythonOperator
LOGGING_CONFIG = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '%(asctime)s [%(levelname)s]
PID=%(process)d:TID=%(thread)d '
'%(name)s.%(funcName)s:%(lineno)d: %(message)s'
},
},
'handlers': {
'default': {
'level': os.getenv('LOGGING_LEVEL', 'INFO'),
'class': 'logging.StreamHandler',
'stream': 'ext://sys.stdout',
'formatter': 'standard'
},
},
'loggers': {
'': {
'handlers': ['default'],
'level': os.getenv('LOGGING_LEVEL', 'INFO'),
'propagate': True
}
}
}
logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger(__name__)default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2018, 6, 10),
'catchup': False,
'email': ['[email protected]'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 3,
'retry_delay': timedelta(seconds=5),
'max_retry_delay': timedelta(seconds=60),
'retry_exponential_backoff': False,
}
dag = DAG(dag_id='aaa_test_logging', catchup=False, default_args=default_args,
schedule_interval=None, max_active_runs=1)
def aaa_test_logging(**kwargs):
logger.info('aaa_test_logging START before sleep')
sleep(5*60)
logger.info('aaa_test_logging END after sleep')
def generate():
PythonOperator(
task_id=f'aaa_test_logging',
python_callable=aaa_test_logging,
provide_context=True,
dag=dag,
)
generate()
{code}
> logger / print gets overwritten with custom Operator after version upgrade
> --------------------------------------------------------------------------
>
> Key: AIRFLOW-7078
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7078
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
> Affects Versions: 1.10.9
> Reporter: seth
> Priority: Major
> Attachments: airflow_post.PNG, airflow_pre.PNG
>
>
> Hi, I have customized `AWSAthenaOperator` to both log the executed query, and
> log the query_id.
> I am using a simple print statement, which should be logged as INFO by the
> logger
> After upgrade to airflow==1.10.9 these logs are not persisted anymore, and in
> fact seems to be overwritten.
> I'm attaching 2 screenshots, 1 of the log output while the task is still
> running. As you can see, the expected output is present in the logs.
> The 2nd, the log is taken after the task has completed successfully .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
