Github user devaraj-kavali commented on a diff in the pull request:
https://github.com/apache/spark/pull/22752#discussion_r226416951
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/config.scala
---
@@ -64,4 +64,11 @@ private[spark] object config {
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("1m")
+ val IN_PROGRESS_ABSOLUTE_LENGTH_CHECK =
+ ConfigBuilder("spark.history.fs.inProgressAbsoluteLengthCheck.enabled")
--- End diff --
This new check adds overhead to the checkForLogs thread, made it disabled
by default since most of the users may not want to see the history ui for the
running applications, they can enable it explicitly if they want to see
progress apps in history ui. I can remove this config if you think not much
useful.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
