[
https://issues.apache.org/jira/browse/SPARK-47622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831831#comment-17831831
]
Srinivasu Majeti commented on SPARK-47622:
------------------------------------------
CCing [~vanzin] to look at it and guide on the next proceedings. Thank you!
> Spark creates lot of tiny blocks for a single driverLog file of size less
> than a dfs.blocksize
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-47622
> URL: https://issues.apache.org/jira/browse/SPARK-47622
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Spark Submit
> Affects Versions: 3.3.2
> Reporter: Srinivasu Majeti
> Priority: Major
>
> Upon reviewing the spark code, found that /user/spark/driverLogs are synced
> to HDFS with hsync option as shown below.
> {code:java}
> hdfsStream.hsync(EnumSet.allOf(classOf[HdfsDataOutputStream.SyncFlag]))
> Ref:
> https://github.com/apache/spark/blob/a3c04ec1145662e4227d57cd953bffce96b8aad7/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala{code}
> As a result of this we see lot of tiny blocks getting synced every 5 seconds
> with a new block. So we see a small HDFS file with 8 blocks as shown in the
> below example.
> {code:java}
> [r...@ccycloud-3.smajeti.root.comops.site subdir0]# hdfs fsck
> /user/spark/driverLogs/application_1710495774861_0002_driver.log
> Connecting to namenode via
> https://ccycloud-3.smajeti.root.comops.site:20102/fsck?ugi=hdfs&path=%2Fuser%2Fspark%2FdriverLogs%2Fapplication_1710495774861_0002_driver.log
> FSCK started by hdfs (auth:KERBEROS_SSL) from /10.140.136.139 for path
> /user/spark/driverLogs/application_1710495774861_0002_driver.log at Thu Mar
> 28 06:37:29 UTC 2024
> Status: HEALTHY
> Number of data-nodes: 4
> Number of racks: 1
> Total dirs: 0
> Total symlinks: 0
> Replicated Blocks:
> Total size: 157574 B
> Total files: 1
> Total blocks (validated): 8 (avg. block size 19696 B)
> Minimally replicated blocks: 8 (100.0 %) {code}
> HdfsDataOutputStream.SyncFlag includes two flags UPDATE_LENGTH and END_BLOCK.
> This has been an expected behavior for some time now and these flags will
> help visualize the latest size of the HDFS Driver log file and to achieve
> that, blocks are being ended/closed every 5-second sync. Every new sync will
> create a new block for the same HDFS driver log file. This hysnc behavior was
> started after fixing SPARK-29105 (SHS may delete driver log file of
> in-progress application) 5 years back.
> But this leaves Namenode to manage a lot of meta and becomes an overhead at
> times in large clusters.
> {code:java}
> public static enum SyncFlag {
> UPDATE_LENGTH,
> END_BLOCK;
> private SyncFlag() {
> }
> }
> {code}
> I don't see any configurable option to avoid this and avoiding this type of
> hsync may have some side effects in spark as we saw SPARK-29105 bug.
> We only have two options that needs manual intevention
> 1. Keep cleaning these Driver logs after some time
> 2. Keep merging these small block files into files with 128MB
> Can we provide some customizable option to merge these blocks while closing
> the spark-shell or during closing the driver log file?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
