(spark) branch branch-3.4 updated: [SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once and reuse the result

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.4 by this push:
     new d1277836354 [SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once 
and reuse the result
d1277836354 is described below

commit d1277836354161bdbb45a3151635bc33905b26a1
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Tue Nov 21 17:51:10 2023 -0800

    [SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once and reuse the 
result
    
    ### What changes were proposed in this pull request?
    
    This PR is a follow-up of #43914 and aims to invoke `fs.listStatus` once 
and reuse the result.
    
    ### Why are the changes needed?
    
    This will prevent the increase of the number of `listStatus` invocation .
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Pass the CIs with the existing test case.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #43944 from dongjoon-hyun/SPARK-46012-2.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
    (cherry picked from commit 6be4a0358265fb81f68a27589f9940bd726c8ee7)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../scala/org/apache/spark/deploy/history/EventLogFileReaders.scala  | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git 
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala 
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
index 714987a8eb8..8c3dda47277 100644
--- 
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
+++ 
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
@@ -119,8 +119,9 @@ object EventLogFileReader extends Logging {
     if (isSingleEventLog(status)) {
       Some(new SingleFileEventLogFileReader(fs, status.getPath, 
Option(status)))
     } else if (isRollingEventLogs(status)) {
-      if 
(fs.listStatus(status.getPath).exists(RollingEventLogFilesWriter.isEventLogFile)
 &&
-          
fs.listStatus(status.getPath).exists(RollingEventLogFilesWriter.isAppStatusFile))
 {
+      val files = fs.listStatus(status.getPath)
+      if (files.exists(RollingEventLogFilesWriter.isEventLogFile) &&
+          files.exists(RollingEventLogFilesWriter.isAppStatusFile)) {
         Some(new RollingEventLogFilesFileReader(fs, status.getPath))
       } else {
         logDebug(s"Rolling event log directory have no event log file at 
${status.getPath}")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org