Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5491#issuecomment-92566099
@vanzin Erhhh, It seems like another solution, but there are few questions:
1.It adds logic to event logger(more codes and more action)
2.It increase the pressure to HDFS, not sure if it is namenode friendly.
3.How should we set the frequency to call setTimes?
Think my solution is better except one issue left:
If user doesn't call sc.stop() in their app, then its event log will be
never deleted. I am not sure if it is our responsibility to take care of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
