Github user squito commented on the issue:
https://github.com/apache/spark/pull/18935
thanks for the added info @jerryshao.
(a) ok makes sense now about the use of this -- it is not exposed now, you
plan to expose it in future changes. That is fine, however I would like to at
least see a *plan* for how you want to expose this before we merge this. I
think past efforts have gotten tripped on how we'd expose it in the driver;
perhaps the simplest thing to do is expose it to the metric system? There just
isn't any point in this unless we're going to do something with it.
(b) thanks for including the example of the metrics. But one minor thing I
noticed -- are you at all surprised that the direct & heap memory are exactly
the same? Are things getting mixed up somewhere? Or maybe those are just the
values from the initialization netty always does. I definitely think that is
more metrics than most users care about, and would want verbose off by default.
I am wondering if those metrics are useful at all ... but I guess they are
pretty cheap, and maybe there is some scenario where a user would care about
some of them, eg. `numHugeAllocations`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
