Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5886#discussion_r29955157
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -184,15 +183,14 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
*/
private[history] def checkForLogs(): Unit = {
try {
+ val newLastScanTime = getNewLastScanTime()
--- End diff --
That's what I mentioned that my suggestion would make SPARK-7189 worse.
There are ways to fix it, such as those discussed in the bug, that don't
require too much extra memory in the HS - basically keep track of all log files
that have `lastModified > newLastScanTime` and, if they haven't changed in the
next poll, don't re-parse them. That's a lot less state to keep track of than
the previous approach in this PR - in fact most of the time there will be
nothing to keep track of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
