jhu-chang opened a new pull request, #36872: URL: https://github.com/apache/spark/pull/36872
In this PR, the DriverRunner has been enhanced to use the same rolling policy as ExecutorRunner. There are two opions for this enhancement: 1. use the same configuration as executor 2. create a new set of configuration prefixed by spark.driver.logs.rolling To minimize the configurations and, the PR chooses #1. ### What changes were proposed in this pull request? The DriverRunner will now use the following configuration to roll the stdout and stderr logs ``` spark.executor.logs.rolling.maxRetainedFiles spark.executor.logs.rolling.enableCompression spark.executor.logs.rolling.maxSize spark.executor.logs.rolling.strategy spark.executor.logs.rolling.time.interval ``` ### Why are the changes needed? The log file (stdout/stderr) of driver in cluster mode on Spark standalone cluster will never get cleaned and will use all the disk space ### Does this PR introduce _any_ user-facing change? If user doesn't set "spark.executor.logs.rolling.*" on spark worker, there is no impaction, if user did specify those settings, then the driver's stdout/stderr will be rolling, user may not see the full log as before. ### How was this patch tested? Hard to add unit test since we need a complete application -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
