Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/981#issuecomment-45318235
What is the impact on memory footprint for this change ?
We have jobs which result in 40k stages, and we have jobs which have stages
with 50k tasks.
I would assume the former would be impacted : but by how much ?
And if the impact is non trivial (the master is already at limits of its
memory), how can the user go about reducing the memory footprint of this change
? Possibly disabling it ? Reducing amount of info stored ? Something
better/different ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
