[ https://issues.apache.org/jira/browse/SPARK-36252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jack Hu updated SPARK-36252: ---------------------------- Description: For a long running driver in cluster mode, there is no rolling policy, the log (stdout/stderr) may accupy lots of space, user needs an external tool to clean the old logs, it's not user friendly. For executor, following 5 configurations is used to control the log file rolling policy: {code:java} spark.executor.logs.rolling.maxRetainedFiles spark.executor.logs.rolling.enableCompression spark.executor.logs.rolling.maxSize spark.executor.logs.rolling.strategy spark.executor.logs.rolling.time.interval {code} For driver running in cluster mode: 1. reuse the executor settings 2. similar to executor: add following configurations (only works for stderr/stdout for driver in cluster mode) {code:java} spark.driver.logs.rolling.maxRetainedFiles spark.driver.logs.rolling.enableCompression spark.driver.logs.rolling.maxSize spark.driver.logs.rolling.strategy spark.driver.logs.rolling.time.interval {code} #2 seems better, do you agree? was: For a long running driver in cluster mode, there is no rolling policy, the log (stdout/stderr) may accupy lots of space, user needs a external tool to clean the old logs, it's not friendly. For executor, following 5 configurations is used to control the log file rolling policy: {code:java} spark.executor.logs.rolling.maxRetainedFiles spark.executor.logs.rolling.enableCompression spark.executor.logs.rolling.maxSize spark.executor.logs.rolling.strategy spark.executor.logs.rolling.time.interval {code} For driver running in cluster mode: 1. reuse the executor settings 2. similar to executor: add following configurations (only works for stderr/stdout for driver in cluster mode) {code:java} spark.driver.logs.rolling.maxRetainedFiles spark.driver.logs.rolling.enableCompression spark.driver.logs.rolling.maxSize spark.driver.logs.rolling.strategy spark.driver.logs.rolling.time.interval {code} #2 seems better, do you agree? > Add log files rolling policy for driver running in cluster mode with spark > standalone cluster > --------------------------------------------------------------------------------------------- > > Key: SPARK-36252 > URL: https://issues.apache.org/jira/browse/SPARK-36252 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.1.2 > Reporter: Jack Hu > Priority: Major > > For a long running driver in cluster mode, there is no rolling policy, the > log (stdout/stderr) may accupy lots of space, user needs an external tool to > clean the old logs, it's not user friendly. > For executor, following 5 configurations is used to control the log file > rolling policy: > {code:java} > spark.executor.logs.rolling.maxRetainedFiles > spark.executor.logs.rolling.enableCompression > spark.executor.logs.rolling.maxSize > spark.executor.logs.rolling.strategy > spark.executor.logs.rolling.time.interval > {code} > For driver running in cluster mode: > 1. reuse the executor settings > 2. similar to executor: add following configurations (only works for > stderr/stdout for driver in cluster mode) > {code:java} > spark.driver.logs.rolling.maxRetainedFiles > spark.driver.logs.rolling.enableCompression > spark.driver.logs.rolling.maxSize > spark.driver.logs.rolling.strategy > spark.driver.logs.rolling.time.interval > {code} > #2 seems better, do you agree? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org