Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147493632
I see the underlying problem posed in the JIRA - it's difficult to assess
duration since it currently includes the time spent waiting on dependent
stages. However, this patch doesn't seem like the obvious way to fix that. I
think there are some alternatives that would make more sense:
1. Re-define duration so that it's only defined starting when the first
task in a stage launches (some concerns here about changing semantics, though).
2. Add a new field that represents the time spent servicing the stage
"service time" (?)
3. Add a new field that represents the time spent queuing before any tasks
launched "queue time" (?)
Those all seem better ways to address the issue in the JIRA. This way of
showing the max task time, it seems indirect. And also not always helpful since
max task time doesn't have a simple relationship with "duration" as desired
here... for instance the max task could be pretty short but the duration is
anyways really long for the stage.
/cc @rxin @kayousterhout for any thoughts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
