HeartSaVioR commented on a change in pull request #27557:
URL: https://github.com/apache/spark/pull/27557#discussion_r413469060
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
##########
@@ -268,6 +288,7 @@ abstract class CompactibleFileStreamLog[T <: AnyRef :
ClassTag](
object CompactibleFileStreamLog {
val COMPACT_FILE_SUFFIX = ".compact"
+ val COMPACT_LATENCY_WARN_THRESHOLD_MS = 2000
Review comment:
Yeah it's a heuristic - I think a batch spending more than 2 seconds
only for compacting metadata should be noticed to the end users, as the latency
here is opaque to end user if we don't log it and they will be questioning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
