jroachgolf84 commented on issue #52969:
URL: https://github.com/apache/airflow/issues/52969#issuecomment-3048564826
This is something that I see a good bit. This normally happens when your
Task instance is killed unexpectedly (OOMKill) and isn't able to write logs to
S3/blob storage. Here's the snippet in the codebase that handles this.
Can you validate that there are in fact logs in S3 for this Task instance?
```python
def _read_from_logs_server(self, ti, worker_log_rel_path) ->
tuple[LogSourceInfo, LogMessages]:
sources = []
logs = []
try:
log_type = LogType.TRIGGER if ti.triggerer_job else
LogType.WORKER
url, rel_path = self._get_log_retrieval_url(ti,
worker_log_rel_path, log_type=log_type)
response = _fetch_logs_from_service(url, rel_path)
if response.status_code == 403:
sources.append(
"!!!! Please make sure that all your Airflow components
(e.g. "
"schedulers, webservers, workers and triggerer) have "
"the same 'secret_key' configured in 'webserver' section
and "
"time is synchronized on all your machines (for example
with ntpd)\n"
"See more at
https://airflow.apache.org/docs/apache-airflow/";
"stable/configurations-ref.html#secret-key"
)
else:
# Check if the resource was properly fetched
response.raise_for_status()
if response.text:
sources.append(url)
logs.append(response.text)
except Exception as e:
from requests.exceptions import InvalidURL
if isinstance(e, InvalidURL) and
ti.task.inherits_from_empty_operator is True:
sources.append(self.inherits_from_empty_operator_log_message)
else:
sources.append(f"Could not read served logs: {e}")
logger.exception("Could not read served logs")
return sources, logs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
