I am comparing performances between different implementations of a spark job and I am testing a chunk of code which prints partial results and info to sdtout. I can surely replace all the prints with logger calls and collect them. I just wanted to know if there was a way to avoid this or if this functionality was of easier implementation.
*Alessandro Liparoti* 2018-06-19 10:52 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: > > Not sure what kind of analysis you want to do, is the logging info in the > interpreter log file enough for you ? (You can update the log level in > log4j.properties to get more logs) > > Alessandro Liparoti <alessandro.l...@gmail.com>于2018年6月19日周二 下午4:47写道： > >> I would like to post-analyze the output of verbose jobs in the notebook >> and save them, avoiding to relaunch the jobs again. It would be also good >> to have the stderr logged to file. >> >> Thanks >> >> *Alessandro Liparoti* >> >> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: >> >>> >>> I am not afraid it is not possible now. The stdout of notebooks is not >>> based on log4j. If you want it output to file as well, you might need to >>> change the code of the interpreter itself. >>> Usually it is not necessary to log it to log file as well, could you >>> tell why you want that ? Thanks >>> >>> >>> >>> alessandro.l...@gmail.com <alessandro.l...@gmail.com>于2018年6月19日周二 >>> 下午3:52写道： >>> >>>> Good morning, >>>> I would like to have stdout of notebooks both printed out to console >>>> and file. How can I achieve that? I tried to play around with log4j but >>>> without any success; it seems it requires a custom appender implementation. >>>> Any other simpler idea? >>>> >>> >>
