Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/5868#issuecomment-101112252
I just pushed a commit (4f0b770) that attempts to implement reasonable
shuffle metrics reporting. @kayousterhout, if you have any spare review cycles
I'd love to get your feedback regarding my interpretations of the current
semantics of shuffle write time and shuffle spill time. I think that I was
fairly careful in terms of where I chose to record shuffle write time, since I
was trying to match the current confusing semantics by ignoring IO time during
spill writing (see https://issues.apache.org/jira/browse/SPARK-3577). It's
possible that I've overlooked something, though, since some of my IO time
benchmark measurements seem a little odd. I'm pretty sure that the shuffle
write bytes / shuffle spill bytes calculations are good, though, since I have
unit tests for them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
