2010YOUY01 commented on PR #21152: URL: https://github.com/apache/datafusion/pull/21152#issuecomment-4131086948
I have some concerns about these low-level (kernel-profiling) metrics, so I’m sharing a few suggestions. (Not trying to block this, given this is useful to solve real problems—just offering additional perspectives and possible improvements.) Metrics are typically used for query tuning at the application level, while these low-level ones are mainly for internal debugging and are less frequently used. They may also introduce execution overhead that’s hard to observe, and bring maintenance overhead. In general, it might be better to keep metrics that directly help application tuning, are frequently used, or are difficult to capture with external profilers. I suspect some of them can be directly observed with profilers/flamegraphs, maybe they can be simplified? Additionally, we could consider introducing a new analyze level `Internal` in `datafusion.explain.analyze_level` (https://datafusion.apache.org/user-guide/configs.html) to hide these metrics from regular output. It might also be worth exploring ways to conditionally disable certain metrics tracking to reduce runtime overhead, then those low level metrics can get added more easily I think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
