Github user darabos commented on the pull request:
https://github.com/apache/spark/pull/981#issuecomment-45321679
Wow, 40k stages? :open_mouth: To estimate the memory use, I guess a large
stack trace could be ~10 kB, so it would be 400 MB total. Would that be
noticeable compared to the base memory use from the 40k stages?
How about I limit the stack trace to 1 kB? That would still be 20 lines, if
there is an average of 50 characters per line. I expect it would be enough in
most cases, and it would limit the memory use to 40 MB in your giant job.
(I could make the cutoff a system property. Should I?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
