Github user devaraj-kavali commented on a diff in the pull request:
https://github.com/apache/spark/pull/22752#discussion_r226429600
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/config.scala
---
@@ -64,4 +64,11 @@ private[spark] object config {
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("1m")
+ val IN_PROGRESS_ABSOLUTE_LENGTH_CHECK =
+ ConfigBuilder("spark.history.fs.inProgressAbsoluteLengthCheck.enabled")
--- End diff --
This call makes two invocations, one for getting blocks info from the
NameNode and another for getting the last block info from DataNode. I agree
this is not a performance critical path, I will remove the config in the update.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
