GitHub user potiuk added a comment to the discussion: Unable to see logs in the web UI when the job is running
> As you can see in the stacktrace in the middle of my issue, the executor is > kubernetes_executor Thanks. that saves a bit of searching through a stack trace. In the future might be better to specify it explicitly rather than leave a chnce that somoene will find it. It allows for people who look at it and try to help to quickly assess whether they can help or whether the case "rings a bell" without actually spending time and looking at such details. It simply optimizes for time of those who try to help you to solve your problem > and as for the volume used for logging, here's the configuration (as shown > in the issue): I was more thinking - what are properties of the volume you have. Something that you can look at your K8S way of handling volumes of the specific kind you use. The error indicates, that somewhere during receiving logs you get "resource unavailable" error. After looking at this - it seems that somewhere the k8s reads logs from remote pod and something does not let it read it. And I think in this case it's something in your K8S configuration., There is a similar issue https://github.com/kubernetes/kubernetes/issues/105928 which indicates that somewhere logs are filing space - for example containerd version used has a problem. And yes - I think the way how logs are read has changed between versions of k8s provider - you can take a look at the changelog - so maybe you had uncovered a configuration or another issue in your K8S. Maybe you can try to see your k8s logs correlating with the events and see if you have some other errors in other components of K8S that indicate what is a root cause. Unfortunately k8s has 100s of moving parts and sometimes you need to dig deeper to find out the root causes (for example often problems - very strange) might occur when your DNS does not have enough resources to respond on time, and the only way to see what's going on is to generally look at what happens in your K8S and see potential issues that are correlated with the event. But I am mostly guessing here - I wanted to help and direct the discussion but I have no deep knowledge on this particular part. GitHub link: https://github.com/apache/airflow/discussions/45624#discussioncomment-11823674 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
