Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/981#issuecomment-45977166
Hey @darabos - this is a cool feature. I had two main pieces of feedback.
1. We need to deal with serailizing/deserailializing your new `StageInfo`
field into JSON. See the note above.
2. This patch only provides a detailed call site for result stages, not for
shuffle map stages. One awkward thing is that we previously had the call site
be an `Option` for a stage and we just didn't define it for shuffle map stages.
There is no reason we shouldn't make `callSite` a required field of `Stage`.
I've written up a patch for that, do you mind including that in your code? I
think this will get easier to follow if we make that change. The abundance of
`Option`'s in the current code makes it pretty confusing.
https://github.com/pwendell/spark/commit/0efddefbf33a094453dd105cf73e61cc330b17b5
3. It would be good if the `long` field included the entire stack trace,
inclusive of what is in the `short` field. It didn't seem to when I did some
local testing. The reason is that in some cases users override the description
that is shown in the UI for a stage. In these cases the `short` field won't
ever be displayed (only the `long` field). If the `long` field lacks the first
line it will be really confusing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
