rafidka edited a comment on issue #13824:
URL: https://github.com/apache/airflow/issues/13824#issuecomment-1055787983
@andormarkus , hmm this is interesting. I didn't particularly test Airflow
2.2.4, but I tested 2.0.2 + watchtower 2.0.0 and beyond and I couldn't
reproduce the issue, so I assumed since Airflow 2.2.4 is using watchtower
2.0.1, then the issue should be resolved. I will see if I can do some testing
and try to reproduce this issue.
> Where should I find the State of this instance has been externally set to
success. Terminating instance. warning Message? I can not find in the worker or
scheduler logs.
This is a bit tricky unfortunately. The thing is that when you configure
Airflow to use CloudWatch logging, but the logging itself has issues, you are
likely to miss some logs. On the other hand, when stop using CloudWatch
logging, the issue itself disappears (i.e. this is a Heisenbug situation.) In
my case, I had to modify Airflow source code locally and then use file based
logging like this:
```
with open('/tmp/local_task_job.py.log', 'a') as f:
print(f"State of this instance has been externally set to
{ti.state}. ", file=f)
print(f'Dumping stack trace:', file=f)
frame = inspect.currentframe()
stack_trace = traceback.format_stack(frame)
print('\n'.join(stack_trace), file=f)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
