To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
The dirs in that env-var are app-specific and are for the app's user to
utilize. You shouldn't have any permission issues working within them.
The LocalDirAllocator is still somewhat MR-bound but you can still
John
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, October 20, 2013 10:49 AM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the relative
./) thats auto
11:58 PM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Hi,
MR does use multiple disks when spilling. But the work directory is also
round-robined to spread I/O.
YARN sets an environment property thats a list (comma separated value) of
directories
John
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, October 20, 2013 10:49 AM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the
relative
Thanks, sounds like LOCAL_DIR_ENV is the way to go.
john
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Monday, October 21, 2013 12:11 PM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
The dirs in that env-var are app-specific
@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
This post might help a bit.
http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
Thanks,
Jian
On Mon, Oct 21, 2013 at 11:11 AM, Harsh J
ha...@cloudera.commailto:ha...@cloudera.com wrote:
The dirs
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the relative ./)
thats auto-cleaned up at the end of the container's life.
This is the best place to store the temporary files. This is not something you
need custom
-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, October 20, 2013 10:49 AM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the relative
./) thats auto-cleaned up at the end
