Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/4749#issuecomment-75876725
@ksakellis I did this in my own branch here:
https://github.com/kayousterhout/spark-1/commit/eb84a0a2201d392a275179293f8a1c4c23981b24
and can submit a PR for Spark if there is sufficient interest. We can't do it
in unpackBlock; the problem is that the data gets read record-by-record, so
we'd have to add how long it took to read each record (in unpackBlock, we still
have have an iterator that points to a memory-mapped region, but the data still
isn't necessarily read from disk there). Any kind of per-record logging ends
up being too high overhead. The way I did it in the change I referred to is to
add buffering to the reading of input data, and then you can log the time to
read each chunk of the buffer (this is the same thing we do to track the time
to write shuffle files to disk).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
