Hi Zhan,
Yes I am running spark on yarn, is there any alternative you came across to get that logs in files. Thanks, Kalpesh Jadhav From: Zhan Zhang [mailto:zzh...@hortonworks.com] Sent: Tuesday, December 22, 2015 5:27 AM To: Kalpesh Jadhav Cc: user@spark.apache.org Subject: Re: Spark with log4j Hi Kalpesh, If you are using spark on yarn, it may not work. Because you write log to files other than stdout/stderr, which yarn log aggregation may not work. As I understand, yarn only aggregate log in stdout/stderr, and local cache will be deleted (in configured timeframe). To check it, at application run time, you can log into the container's box, and check the local cache of the container to find whether the log file exists or not (after app terminate, these local cache files will be deleted as well). Thanks. Zhan Zhang On Dec 18, 2015, at 7:23 AM, Kalpesh Jadhav <kalpesh.jad...@citiustech.com> wrote: Hi all, I am new to spark, I am trying to use log4j for logging my application. But any how the logs are not getting written at specified file. I have created application using maven, and kept log.properties file at resources folder. Application written in scala . If there is any alternative instead of log4j then also it will work, but I wanted to see logs in file. If any changes need to be done in <https://www.google.co.in/search?client=firefox-a&rls=org.mozilla:en-US:offi cial&channel=fflb&q=hortonworks&spell=1&sa=X&ved=0ahUKEwj5k4Gq2-XJAhXUB44KHY U-C6MQvwUIGSgA> hortonworks for spark configuration, please mentioned that as well. If anyone has done before or on github any source available please respond. Thanks, Kalpesh Jadhav ============================================================================ ============================================================================ =================================== DISCLAIMER: The information contained in this message (including any attachments) is confidential and may be privileged. If you have received it by mistake please notify the sender by return e-mail and permanently delete this message and any attachments from your system. Any dissemination, use, review, distribution, printing or copying of this message in whole or in part is strictly prohibited. Please note that e-mails are susceptible to change. CitiusTech shall not be liable for the improper or incomplete transmission of the information contained in this communication nor for any delay in its receipt or damage to your system. CitiusTech does not guarantee that the integrity of this communication has been maintained or that this communication is free of viruses, interceptions or interferences. ============================================================================ ============================================================================ ============ =========================================================================================================================================================================================== DISCLAIMER: The information contained in this message (including any attachments) is confidential and may be privileged. If you have received it by mistake please notify the sender by return e-mail and permanently delete this message and any attachments from your system. Any dissemination, use, review, distribution, printing or copying of this message in whole or in part is strictly prohibited. Please note that e-mails are susceptible to change. CitiusTech shall not be liable for the improper or incomplete transmission of the information contained in this communication nor for any delay in its receipt or damage to your system. CitiusTech does not guarantee that the integrity of this communication has been maintained or that this communication is free of viruses, interceptions or interferences. ====================================================================================================================================================================
