chrismclennon opened a new issue #9610:
URL: https://github.com/apache/airflow/issues/9610
**Apache Airflow version**: 1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`): 1.17.2
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): CentOS Linux
- **Kernel** (e.g. `uname -a`): `Linux airflow-worker-0
5.6.13-1.el7.elrepo.x86_64 #1 SMP Thu May 14 08:05:24 EDT 2020 x86_64 x86_64
x86_64 GNU/Linux`
- **Install tools**:
- **Others**:
**What happened**:
I run Airflow as a way to orchestrate jobs on Kubernetes using the
KubernetesPodOperator. While most of the time logs appear correctly in the
Airflow webserver, I do notice increasingly that the logs do not appear and
instead just show a message, "Task is not able to be run", such as in the
snippet below:
```
*** Reading remote log from
s3://*****/****/******/2020-06-30T10:00:00+00:00/1.log.
[2020-06-30 23:07:40,362] {taskinstance.py:663} INFO - Dependencies not met
for <TaskInstance: ****.***** 2020-06-30T10:00:00+00:00 [running]>, dependency
'Task Instance State' FAILED: Task is in the 'running' state which is not a
valid state for execution. The task must be cleared in order to be run.
[2020-06-30 23:07:40,363] {logging_mixin.py:112} INFO - [2020-06-30
23:07:40,363] {local_task_job.py:91} INFO - Task is not able to be run
```
Unusually, when I go check what is happening on the Kubernetes cluster, the
pod is actually running and emitting logs when I run a `kubectl logs` command.
When the pod is complete, Airflow will reflect that the task has completed as
well.
**What you expected to happen**: I expected pod logs to be printed out.
**How to reproduce it**: Very unfortunately, I am unsure what circumstances
cause this error and am currently trying to gather evidence to replicate.
**Anything else we need to know**:
* I have remote logging set to an S3 bucket.
* I've noticed this issue increasingly with the 1.10.10 update, and I find
this error daily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
