parthchandra commented on PR #44021: URL: https://github.com/apache/spark/pull/44021#issuecomment-1863766109
> If async profiler does not allow us to map the native thread to its java thread (please validate this) - we cannot map stack traces to the corresponding task threads - and so limits usability of this integration in spark. `AsyncGetCallTrace` is used precisely to map calls in the native thread to calls in the java thread. Not sure exactly what you are looking for here. Are you looking to profile individual tasks? It certainly can be done, but would require some changes similar to SPARK-45151 and some additional work if you want the profile available thru the UI. Or are you looking to enhance SPARK-45151 and get a stack trace that includes native calls? This is a little harder via async_profiler since there is no API to get a snapshot. Note that getting a profile needs to be collected over a period of time and so is different from getting a snapshot as SPARK-45151 is doing. > Simply dumping per executor flamegraphs or stack traces has limited utility (and can be done today). I would suggest that this PR makes it trivially simple to profile with no setup required. On K8s, with ephemeral storage, it is not a simple task to dump a profile to disk and get it off the pod before the pod is destroyed (it was in fact the original motivation behind doing this). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
