george-zubrienko commented on issue #21387: URL: https://github.com/apache/airflow/issues/21387#issuecomment-1033002973
> I think in many installations you will not be able to exec into running container from webserver. This is a common security practice that you are not allowed to do it in production. This assumes you have rbac in place that blocks exec. But airflow default pod security context has fsGroup 0, which is also not a common practice in production. Moreover, this is internal call, compared to `exec`s mainly used by outside callers. I'm not sure if opening a pod-pod websocket connection, which is what an SDK will do, is the same as `exec`? Even if we put this aside, there there is still an issue with pod log != task log. And k8s pod logs are usually scraped to central logging processing system (datadog , cloudwatch etc.). Given the chaotic nature of airflow logs, it is not a great practice to feed them into pod stdout as some solutions on the internet suggest. So we are left with elastic/custom logger as the only viable options. But custom loggers are not guaranteed to survive next update and elastic is not the only log processing solution. Can we then change the way logs are written to remote, aka they are streamed instead of moved when done? What are the potential pitfalls there? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
