Re: View all user's application logs in history server
Hmm...all files under the event log folder has permission 770 but strangely my account cannot read other user's files. Permission denied. I'll sort it out with our Hadoop admin. Thanks for then help! Jianshi On Thu, May 28, 2015 at 12:13 AM, Marcelo Vanzin wrote: > Then: > - Are all files readable by the user running the history server? > - Did all applications call sc.stop() correctly (i.e. files do not have > the ".inprogress" suffix)? > > Other than that, always look at the logs first, looking for any errors > that may be thrown. > > > On Wed, May 27, 2015 at 9:10 AM, Jianshi Huang > wrote: > >> Yes, all written to the same directory on HDFS. >> >> Jianshi >> >> On Wed, May 27, 2015 at 11:57 PM, Marcelo Vanzin >> wrote: >> >>> You may be the only one not seeing all the logs. Are you sure all the >>> users are writing to the same log directory? The HS can only read from a >>> single log directory. >>> >>> On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang >>> wrote: >>> No one using History server? :) Am I the only one need to see all user's logs? Jianshi On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang >>> > wrote: > Hi, > > I'm using Spark 1.4.0-rc1 and I'm using default settings for history > server. > > But I can only see my own logs. Is it possible to view all user's > logs? The permission is fine for the user group. > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/ >>> >>> >>> >>> -- >>> Marcelo >>> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > > > -- > Marcelo > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
Re: View all user's application logs in history server
BTW, is there an option to set file permission for spark event logs? Jianshi On Thu, May 28, 2015 at 11:25 AM, Jianshi Huang wrote: > Hmm...all files under the event log folder has permission 770 but > strangely my account cannot read other user's files. Permission denied. > > I'll sort it out with our Hadoop admin. Thanks for then help! > > Jianshi > > On Thu, May 28, 2015 at 12:13 AM, Marcelo Vanzin > wrote: > >> Then: >> - Are all files readable by the user running the history server? >> - Did all applications call sc.stop() correctly (i.e. files do not have >> the ".inprogress" suffix)? >> >> Other than that, always look at the logs first, looking for any errors >> that may be thrown. >> >> >> On Wed, May 27, 2015 at 9:10 AM, Jianshi Huang >> wrote: >> >>> Yes, all written to the same directory on HDFS. >>> >>> Jianshi >>> >>> On Wed, May 27, 2015 at 11:57 PM, Marcelo Vanzin >>> wrote: >>> You may be the only one not seeing all the logs. Are you sure all the users are writing to the same log directory? The HS can only read from a single log directory. On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang >>> > wrote: > No one using History server? :) > > Am I the only one need to see all user's logs? > > Jianshi > > On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang < > jianshi.hu...@gmail.com> wrote: > >> Hi, >> >> I'm using Spark 1.4.0-rc1 and I'm using default settings for history >> server. >> >> But I can only see my own logs. Is it possible to view all user's >> logs? The permission is fine for the user group. >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Marcelo >>> >>> >>> >>> -- >>> Jianshi Huang >>> >>> LinkedIn: jianshi >>> Twitter: @jshuang >>> Github & Blog: http://huangjs.github.com/ >>> >> >> >> >> -- >> Marcelo >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
Re: View all user's application logs in history server
Then: - Are all files readable by the user running the history server? - Did all applications call sc.stop() correctly (i.e. files do not have the ".inprogress" suffix)? Other than that, always look at the logs first, looking for any errors that may be thrown. On Wed, May 27, 2015 at 9:10 AM, Jianshi Huang wrote: > Yes, all written to the same directory on HDFS. > > Jianshi > > On Wed, May 27, 2015 at 11:57 PM, Marcelo Vanzin > wrote: > >> You may be the only one not seeing all the logs. Are you sure all the >> users are writing to the same log directory? The HS can only read from a >> single log directory. >> >> On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang >> wrote: >> >>> No one using History server? :) >>> >>> Am I the only one need to see all user's logs? >>> >>> Jianshi >>> >>> On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang >>> wrote: >>> Hi, I'm using Spark 1.4.0-rc1 and I'm using default settings for history server. But I can only see my own logs. Is it possible to view all user's logs? The permission is fine for the user group. -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/ >>> >>> >>> >>> -- >>> Jianshi Huang >>> >>> LinkedIn: jianshi >>> Twitter: @jshuang >>> Github & Blog: http://huangjs.github.com/ >>> >> >> >> >> -- >> Marcelo >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Marcelo
Re: View all user's application logs in history server
Yes, all written to the same directory on HDFS. Jianshi On Wed, May 27, 2015 at 11:57 PM, Marcelo Vanzin wrote: > You may be the only one not seeing all the logs. Are you sure all the > users are writing to the same log directory? The HS can only read from a > single log directory. > > On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang > wrote: > >> No one using History server? :) >> >> Am I the only one need to see all user's logs? >> >> Jianshi >> >> On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang >> wrote: >> >>> Hi, >>> >>> I'm using Spark 1.4.0-rc1 and I'm using default settings for history >>> server. >>> >>> But I can only see my own logs. Is it possible to view all user's logs? >>> The permission is fine for the user group. >>> >>> -- >>> Jianshi Huang >>> >>> LinkedIn: jianshi >>> Twitter: @jshuang >>> Github & Blog: http://huangjs.github.com/ >>> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > > > -- > Marcelo > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
Re: View all user's application logs in history server
You may be the only one not seeing all the logs. Are you sure all the users are writing to the same log directory? The HS can only read from a single log directory. On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang wrote: > No one using History server? :) > > Am I the only one need to see all user's logs? > > Jianshi > > On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang > wrote: > >> Hi, >> >> I'm using Spark 1.4.0-rc1 and I'm using default settings for history >> server. >> >> But I can only see my own logs. Is it possible to view all user's logs? >> The permission is fine for the user group. >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Marcelo
Re: View all user's application logs in history server
No one using History server? :) Am I the only one need to see all user's logs? Jianshi On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang wrote: > Hi, > > I'm using Spark 1.4.0-rc1 and I'm using default settings for history > server. > > But I can only see my own logs. Is it possible to view all user's logs? > The permission is fine for the user group. > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
