Re: Need help to understand hadoop.tmp.dir
On Mon, Mar 03, 2014 at 09:03:28AM -0500, JCAD Cell 1 wrote:
> With the services stopped you would change the setting in core-site.xml:
>
> hadoop.tmp.dir
> /var/hadoop/tmp
>
>
> Then move your /tmp/hadoop folder over to the new location:
> mv /tmp/hadoop /var/hadoop/tmp
Thank you, since the machine did a reboot, so all files in
/tmp/hadoop-user, the previous default direcotry of hadoop.tmp.dir, were
gone.
So that's the problem, no files to left, if the namenode can't re-create
its name space, then I think the only way is to format the namenode
again?
--
Thanks,
Chengwei
>
>
>
> On Mon, Mar 3, 2014 at 5:55 AM, Chengwei Yang
> wrote:
>
> On Mon, Mar 03, 2014 at 01:57:49PM +0530, shashwat shriparv wrote:
> > NO need to format just change the value and restart the cluster;
>
> Hmm, seems it doesn't work for me, if the only need to do is to change
> to another directory, then why it can not re-init the directory in /tmp
> just as another directory?
>
> If I changed to another directory, a new directory, the same error
> happen.
>
> --
> Thanks,
> Chengwei
>
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> twitter.com/
> > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> +ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 1:55 PM, Chengwei Yang
> >
> > wrote:
> >
> > On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> > > Ya its always better to change the temp dir path in hadoop, as it
> will
> > prevent
> > > deletion of file while the server reboots.
> >
> > Thanks, so is there anyway to recovery from this state? Or I have to
> format
> > namenode again?
> >
> > --
> > Thanks,
> > Chengwei
> >
> > >
> > >
> > > Warm Regards_∞_
> > > Shashwat Shriparv
> > > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> > twitter.com/
> > > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> > +ShashwatShriparv
> > > http://www.youtube.com/user/sShriparv/videoshttp://
> profile.yahoo.com/
> > > SWXSTW3DVSDTF2HHSRM47AV6DI/
> > >
> > >
> > >
> > > On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang <
> > chengwei.yang...@gmail.com>
> > > wrote:
> > >
> > > On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv
> wrote:
> > > > You can use any directory you like beside permissions are
> right.
> > >
> > > I mean if it's better if we change the default hadoop.tmp.dir?
> > Because it
> > > can not work cross reboot in default Linux environment.
> > >
> > > --
> > > Thanks,
> > > Chengwei
> > >
> > > >
> > > >
> > > > Warm Regards_∞_
> > > > Shashwat Shriparv
> > > > http://www.linkedin.com/pub/shashwat-shriparv/19/214/
> 2a9https://
> > > twitter.com/
> > > > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> > > +ShashwatShriparv
> > > > http://www.youtube.com/user/sShriparv/videoshttp://
> > profile.yahoo.com/
> > > > SWXSTW3DVSDTF2HHSRM47AV6DI/
> > > >
> > > >
> > > >
> > > > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> > > chengwei.yang...@gmail.com>
> > > > wrote:
> > > >
> > > > Hi List,
> > > >
> > > > I'm confusing by hadoop.tmp.dir currently because its
> default
> > value
> > > > "/tmp/hadoop-${user.name}" always means a directory in
> tmpfs in
> > > Linux.
Re: Need help to understand hadoop.tmp.dir
On Mon, Mar 03, 2014 at 01:57:49PM +0530, shashwat shriparv wrote:
> NO need to format just change the value and restart the cluster;
Hmm, seems it doesn't work for me, if the only need to do is to change
to another directory, then why it can not re-init the directory in /tmp
just as another directory?
If I changed to another directory, a new directory, the same error
happen.
--
Thanks,
Chengwei
>
>
> Warm Regards_∞_
> Shashwat Shriparv
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> shriparvhttps://www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> SWXSTW3DVSDTF2HHSRM47AV6DI/
>
>
>
> On Mon, Mar 3, 2014 at 1:55 PM, Chengwei Yang
> wrote:
>
> On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> > Ya its always better to change the temp dir path in hadoop, as it will
> prevent
> > deletion of file while the server reboots.
>
> Thanks, so is there anyway to recovery from this state? Or I have to
> format
> namenode again?
>
> --
> Thanks,
> Chengwei
>
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> twitter.com/
> > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> +ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang <
> chengwei.yang...@gmail.com>
> > wrote:
> >
> > On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv wrote:
> > > You can use any directory you like beside permissions are right.
> >
> > I mean if it's better if we change the default hadoop.tmp.dir?
> Because it
> > can not work cross reboot in default Linux environment.
> >
> > --
> > Thanks,
> > Chengwei
> >
> > >
> > >
> > > Warm Regards_∞_
> > > Shashwat Shriparv
> > > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> > twitter.com/
> > > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> > +ShashwatShriparv
> > > http://www.youtube.com/user/sShriparv/videoshttp://
> profile.yahoo.com/
> > > SWXSTW3DVSDTF2HHSRM47AV6DI/
> > >
> > >
> > >
> > > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> > chengwei.yang...@gmail.com>
> > > wrote:
> > >
> > > Hi List,
> > >
> > > I'm confusing by hadoop.tmp.dir currently because its default
> value
> > > "/tmp/hadoop-${user.name}" always means a directory in tmpfs
> in
> > Linux.
> > > So after the name node machine reboot, it gone away and then
> name
> > node
> > > fail to start.
> > >
> > > I found this was reported here.
> > > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/
> 201205.mbox
> > /
> > > %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> > >
> > > As I found from http://hadoop.apache.org/docs/r2.3.0/, there
> are a
> > lot
> > > properties are based on hadoop.tmp.dir, like
> > > dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name
> > >
> > > I'm wondering, if we can set the default value of
> hadoop.tmp.dir to
> > > a non-tmpfs direcotry if it doesn't work at all by using a
> real
> tmpfs
> > > directory?
> > >
> > > --
> > > Thanks,
> > > Chengwei
> > >
> > >
> >
> >
>
>
Re: Need help to understand hadoop.tmp.dir
On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> Ya its always better to change the temp dir path in hadoop, as it will prevent
> deletion of file while the server reboots.
Thanks, so is there anyway to recovery from this state? Or I have to format
namenode again?
--
Thanks,
Chengwei
>
>
> Warm Regards_∞_
> Shashwat Shriparv
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> shriparvhttps://www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> SWXSTW3DVSDTF2HHSRM47AV6DI/
>
>
>
> On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang
> wrote:
>
> On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv wrote:
> > You can use any directory you like beside permissions are right.
>
> I mean if it's better if we change the default hadoop.tmp.dir? Because it
> can not work cross reboot in default Linux environment.
>
> --
> Thanks,
> Chengwei
>
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> twitter.com/
> > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> +ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> chengwei.yang...@gmail.com>
> > wrote:
> >
> > Hi List,
> >
> > I'm confusing by hadoop.tmp.dir currently because its default value
> > "/tmp/hadoop-${user.name}" always means a directory in tmpfs in
> Linux.
> > So after the name node machine reboot, it gone away and then name
> node
> > fail to start.
> >
> > I found this was reported here.
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201205.mbox
> /
> > %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> >
> > As I found from http://hadoop.apache.org/docs/r2.3.0/, there are a
> lot
> > properties are based on hadoop.tmp.dir, like
> > dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name
> >
> > I'm wondering, if we can set the default value of hadoop.tmp.dir to
> > a non-tmpfs direcotry if it doesn't work at all by using a real
> tmpfs
> > directory?
> >
> > --
> > Thanks,
> > Chengwei
> >
> >
>
>
Re: Need help to understand hadoop.tmp.dir
On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv wrote:
> You can use any directory you like beside permissions are right.
I mean if it's better if we change the default hadoop.tmp.dir? Because it
can not work cross reboot in default Linux environment.
--
Thanks,
Chengwei
>
>
> Warm Regards_∞_
> Shashwat Shriparv
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> shriparvhttps://www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> SWXSTW3DVSDTF2HHSRM47AV6DI/
>
>
>
> On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang
> wrote:
>
> Hi List,
>
> I'm confusing by hadoop.tmp.dir currently because its default value
> "/tmp/hadoop-${user.name}" always means a directory in tmpfs in Linux.
> So after the name node machine reboot, it gone away and then name node
> fail to start.
>
> I found this was reported here.
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201205.mbox/
> %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
>
> As I found from http://hadoop.apache.org/docs/r2.3.0/, there are a lot
> properties are based on hadoop.tmp.dir, like
> dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name
>
> I'm wondering, if we can set the default value of hadoop.tmp.dir to
> a non-tmpfs direcotry if it doesn't work at all by using a real tmpfs
> directory?
>
> --
> Thanks,
> Chengwei
>
>
Need help to understand hadoop.tmp.dir
Hi List,
I'm confusing by hadoop.tmp.dir currently because its default value
"/tmp/hadoop-${user.name}" always means a directory in tmpfs in Linux.
So after the name node machine reboot, it gone away and then name node
fail to start.
I found this was reported here.
http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201205.mbox/%3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
As I found from http://hadoop.apache.org/docs/r2.3.0/, there are a lot
properties are based on hadoop.tmp.dir, like
dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name
I'm wondering, if we can set the default value of hadoop.tmp.dir to
a non-tmpfs direcotry if it doesn't work at all by using a real tmpfs
directory?
--
Thanks,
Chengwei
