[ https://issues.apache.org/jira/browse/AIRFLOW-7078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063565#comment-17063565 ]
Oleg Korsak commented on AIRFLOW-7078: -------------------------------------- After my investigation I've found that logs are not being overwritten after task is completed. My logging records are just missing even while sleep(5*60) and of course after that. Here is the sample DAG: {code:java} import logging.config import os from datetime import datetime, timedelta from time import sleepfrom airflow import DAG from airflow.operators.python_operator import PythonOperator LOGGING_CONFIG = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'standard': { 'format': '%(asctime)s [%(levelname)s] PID=%(process)d:TID=%(thread)d ' '%(name)s.%(funcName)s:%(lineno)d: %(message)s' }, }, 'handlers': { 'default': { 'level': os.getenv('LOGGING_LEVEL', 'INFO'), 'class': 'logging.StreamHandler', 'stream': 'ext://sys.stdout', 'formatter': 'standard' }, }, 'loggers': { '': { 'handlers': ['default'], 'level': os.getenv('LOGGING_LEVEL', 'INFO'), 'propagate': True } } } logging.config.dictConfig(LOGGING_CONFIG) logger = logging.getLogger(__name__)default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2018, 6, 10), 'catchup': False, 'email': ['okors...@magic.com'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 3, 'retry_delay': timedelta(seconds=5), 'max_retry_delay': timedelta(seconds=60), 'retry_exponential_backoff': False, } dag = DAG(dag_id='aaa_test_logging', catchup=False, default_args=default_args, schedule_interval=None, max_active_runs=1) def aaa_test_logging(**kwargs): logger.info('aaa_test_logging START before sleep') sleep(5*60) logger.info('aaa_test_logging END after sleep') def generate(): PythonOperator( task_id=f'aaa_test_logging', python_callable=aaa_test_logging, provide_context=True, dag=dag, ) generate() {code} > logger / print gets overwritten with custom Operator after version upgrade > -------------------------------------------------------------------------- > > Key: AIRFLOW-7078 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7078 > Project: Apache Airflow > Issue Type: Bug > Components: logging > Affects Versions: 1.10.9 > Reporter: seth > Priority: Major > Attachments: airflow_post.PNG, airflow_pre.PNG > > > Hi, I have customized `AWSAthenaOperator` to both log the executed query, and > log the query_id. > I am using a simple print statement, which should be logged as INFO by the > logger > After upgrade to airflow==1.10.9 these logs are not persisted anymore, and in > fact seems to be overwritten. > I'm attaching 2 screenshots, 1 of the log output while the task is still > running. As you can see, the expected output is present in the logs. > The 2nd, the log is taken after the task has completed successfully . -- This message was sent by Atlassian Jira (v8.3.4#803005)