Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/10966#issuecomment-176502936
Thanks a lot @zsxwing for your comments. Yes I agree with you that current
approach of replaying event logs suffers a lot for streaming application. The
reason I brought out the patch here is to hear more feedbacks about this
approach or what is the right way to address it. Below is what I though about:
* This problem is not only streaming specific problem, but also affects all
the long running applications like thrift server. Unless we change the current
replay mechanism for long running service, this problem will always exist.
* Beside this scalability issue, I think the modification of Streaming UI
and `StreamingJobProgressListener` is still valid to decouple from live
streaming context, like what other UI did.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
