Sinsin1367 commented on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod… URL: https://github.com/apache/airflow/pull/6261#issuecomment-543251695 > My initial thought: Just use prometheus/cadvisor for this. There are other tools that are better suited to capturing metrics than Airflow. > > What about the KubeExecutor? If (and it's still an if to me) we are to add this it should work with pods launched by the executor too. The reason for fetching usage in this feature is to report the usage interleaved with application logs. This way the user would know at what stage in their job their usage is going up or down and will watch the usage as it is going. Also it eliminates necessity of using metric monitoring frameworks. Like in our case we use datadog for our metric monitoring which is not that easy to integrate in this case. KubeExecutor uses PodLauncher to launch pods also. However it seems it calls the async method of podLauncher which means it would not log.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
