Github user kanzhang commented on the pull request:
https://github.com/apache/spark/pull/366#issuecomment-40031940
Thanks for getting rid of the 100ms wait. I noticed it but didn't have
better idea.
I think this patch addresses SPARK-1407. Without this patch, even if
sc.stop() is called, it will only close the logger and flush whatever is
already in its write buffer. If, at this point, some events are yet to be
drained from event queue, when listenerThread tries to post them to logger
later on, the same Filesystem is closed exception will be encountered.
Consequently event log file will be incomplete (same as in SPARK-1407).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
