[GitHub] [kafka] ijuma commented on a change in pull request #8970: MINOR: Improve logging around initial log loading

[GitBox]

ijuma commented on a change in pull request #8970:
URL: https://github.com/apache/kafka/pull/8970#discussion_r448743973



##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -321,25 +325,32 @@ class LogManager(logDirs: Seq[File],
           recoveryPoints = this.recoveryPointCheckpoints(dir).read()
         } catch {
           case e: Exception =>
-            warn(s"Error occurred while reading 
recovery-point-offset-checkpoint file of directory $dir", e)
-            warn("Resetting the recovery checkpoint to 0")
+            warn(s"Error occurred while reading 
recovery-point-offset-checkpoint file of directory $dir, " +
+              "resetting the recovery checkpoint to 0", e)
         }
 
         var logStartOffsets = Map[TopicPartition, Long]()
         try {
           logStartOffsets = this.logStartOffsetCheckpoints(dir).read()
         } catch {
           case e: Exception =>
-            warn(s"Error occurred while reading log-start-offset-checkpoint 
file of directory $dir", e)
+            warn(s"Error occurred while reading log-start-offset-checkpoint 
file of directory $dir, " +
+              "resetting to the base offset of the first segment", e)
         }
 
-        val jobsForDir = for {
-          dirContent <- Option(dir.listFiles).toList
-          logDir <- dirContent if logDir.isDirectory
-        } yield {
+        val logsToLoad = 
Option(dir.listFiles).getOrElse(Array.empty).filter(_.isDirectory)
+        val numRemainingLogsToLoad = new AtomicInteger(logsToLoad.length)
+
+        val jobsForDir = logsToLoad.map { logDir =>
           val runnable: Runnable = () => {
             try {
-              loadLog(logDir, recoveryPoints, logStartOffsets)
+              debug(s"Loading log $logDir")
+              val logLoadStartMs = time.milliseconds()
+              val log = loadLog(logDir, recoveryPoints, logStartOffsets)
+              numRemainingLogsToLoad.decrementAndGet()
+              info(s"Completed load of $log with ${log.numberOfSegments} 
segments " +
+                s"in ${time.milliseconds() - logLoadStartMs}ms " +
+                s"(${numRemainingLogsToLoad.get} logs remaining to load in 
$dir)")

Review comment:
       Nit: an alternative would be to say something like `3 out of 5 logs have 
been loaded in $dir`. It gives a slightly better sense of progression.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org