raj-manvar commented on issue #9610: URL: https://github.com/apache/airflow/issues/9610#issuecomment-689941118
Few new findings: * was able to reproduce this issue for task running more than a minute by setting `visibility_timeout = 60` in [celery_broker_transport_options] in airflow.cfg file. * Therefore, this is happening because Celery expects the task to complete within an hour and if not assigns another worker for the task, during this transition, worker uploads the logs with "Task is not able to run" to S3 * Can see another worker getting same task from logs `Received task: airflow.executors.celery_executor.execute_command[b40cacbb-9dd3-4681-8454-0e1df2dbc910]` with same id seconding that Celery is assigning this task to another worker. * Modifying "visibility_timeout = 86400 # 1day" in airflow.cfg doesn't resolve this issue and logs in UI are corrupted after an hour * Even tried "visibility_timeout = 7200 # 2 hours" in airflow.cfg but can still see this issue after an hour. * Seems the issue is similar to https://github.com/celery/celery/issues/5935, but according to this it should be resolved in Celery version 4.4.5 but, we still see the same issue even though Airflow 1.10.10 uses Celery version 4.4.6 ( CC: @chrismclennon @dimberman ) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
