Hi Kalpseh, Just to add, you could use "yarn logs -applicationId <application id>" to see aggregated logs once application is finished.
Thanks, Sivakumar Bhavanari. On Mon, Dec 21, 2015 at 3:56 PM, Zhan Zhang <zzh...@hortonworks.com> wrote: > Hi Kalpesh, > > If you are using spark on yarn, it may not work. Because you write log to > files other than stdout/stderr, which yarn log aggregation may not work. As > I understand, yarn only aggregate log in stdout/stderr, and local cache > will be deleted (in configured timeframe). > > To check it, at application run time, you can log into the container’s > box, and check the local cache of the container to find whether the log > file exists or not (after app terminate, these local cache files will be > deleted as well). > > Thanks. > > Zhan Zhang > > On Dec 18, 2015, at 7:23 AM, Kalpesh Jadhav <kalpesh.jad...@citiustech.com> > wrote: > > Hi all, > > I am new to spark, I am trying to use log4j for logging my application. > But any how the logs are not getting written at specified file. > > I have created application using maven, and kept log.properties file at > resources folder. > Application written in scala . > > If there is any alternative instead of log4j then also it will work, but I > wanted to see logs in file. > > If any changes need to be done in hortonworks > <https://www.google.co.in/search?client=firefox-a&rls=org.mozilla:en-US:official&channel=fflb&q=hortonworks&spell=1&sa=X&ved=0ahUKEwj5k4Gq2-XJAhXUB44KHYU-C6MQvwUIGSgA> > for spark configuration, please mentioned that as well. > > If anyone has done before or on github any source available please respond. > > > Thanks, > Kalpesh Jadhav > =========================================================================================================================================================================================== > DISCLAIMER: The information contained in this message (including any > attachments) is confidential and may be privileged. If you have received it > by mistake please notify the sender by return e-mail and permanently delete > this message and any attachments from your system. Any dissemination, use, > review, distribution, printing or copying of this message in whole or in > part is strictly prohibited. Please note that e-mails are susceptible to > change. CitiusTech shall not be liable for the improper or incomplete > transmission of the information contained in this communication nor for any > delay in its receipt or damage to your system. CitiusTech does not > guarantee that the integrity of this communication has been maintained or > that this communication is free of viruses, interceptions or interferences. > ==================================================================================================================================================================== > > >
