dongjoon-hyun commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1888868394
Thank you all. Especially, @mridulm for many ideas and advices on this PR
reviews.
- Ya, as you mentioned, `honest-profiler` has been abandoned about two years
unfortunately. We should not depend on it.
- Like `Safari` (2019), there has been many profiler approaches outside the
community because it's valuable. However, IIUC, it's not easily available to
the Apache Spark users in these days.
- I believe the value of this PR is capable of providing `Apache Spark
4.0.0` users the foundation of easy pre-defined profiling with additional help
like `spark.executor.profiling.fraction`.
- To be clear, we don't recommend users to enable this for all executors
of all jobs.
- Lastly, this plugin way is non-intrusive and has no maintain cost because
it's based on the Spark standard. For the proposed integrated and detailed
analysis, I believe we can achieve them in the future in a way because we
agreed lots of values of that integrated analysis.
I hope we can merge this as a part of Apache Spark 4.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
