The logs from the executor are redirected to stdout only because there is a default log4j.properties that is configured to do so. If you put your log4j.properties with rolling file appender in the classpath (refer to Spark docs for that), all the logs will get redirected to a separate files that will get rolled over. So even though there will be a stdout, nothing much would get printed in that. Though the side effect of this modification is that you wont be able to see the logs from the Spark web UI as that only shows stdout.
On Mon, Mar 24, 2014 at 10:48 PM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > Hi TD, > > I thought about that but was not sure whether this will have any impact in > spark UI/ Executor runner as it redirects stream to stderr/stdout. But > ideally it should not as it will fetch the log record from stderr file > (which is latest).. > > Is my understanding correct? > > Thanks, > Sourav > > > On Tue, Mar 25, 2014 at 3:26 AM, Tathagata Das < > tathagata.das1...@gmail.com> wrote: > >> You can use RollingFileAppenders in log4j.properties. >> >> http://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html >> >> You can have other scripts delete old logs. >> >> TD >> >> >> On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra < >> sourav.chan...@livestream.com> wrote: >> >>> Hi, >>> >>> I have few questions regarding log file management in spark: >>> >>> 1. Currently I did not find any way to modify the lof file name for >>> executor/drivers). Its hardcoded as stdout and stderr. Also there is no log >>> rotation. >>> >>> In case of streaming application this will grow forever and become >>> unmanageable. Is there any way to overcome this? >>> >>> Thanks, >>> -- >>> >>> Sourav Chandra >>> >>> Senior Software Engineer >>> >>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · >>> >>> sourav.chan...@livestream.com >>> >>> o: +91 80 4121 8723 >>> >>> m: +91 988 699 3746 >>> >>> skype: sourav.chandra >>> >>> Livestream >>> >>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd >>> Block, Koramangala Industrial Area, >>> >>> Bangalore 560034 >>> >>> www.livestream.com >>> >> >> > > > -- > > Sourav Chandra > > Senior Software Engineer > > · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · > > sourav.chan...@livestream.com > > o: +91 80 4121 8723 > > m: +91 988 699 3746 > > skype: sourav.chandra > > Livestream > > "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd > Block, Koramangala Industrial Area, > > Bangalore 560034 > > www.livestream.com >
