I have done chmod only for mapper/reducer/combiner for -file it just
checks for availability of file. Is it a good idea to make all files
in -file option executable?
Sanjay
On 08-Dec-06, at 1:23 AM, Arkady Borkovsky wrote:
chmod is fine.
In some cases streaming job is a script that uses executables that
are shipped using -file option.
The executables loose the x permissions and the script fails. (I
ran into this and it took me a while to undersatnd the reason of
failure).
This may be fixed or alleviated by
(a) adding an -executable option that is similar to -file but
does "chmod a+x"
(b) at least describing this in user documentation, so that the
user knows to do this on her own.
Of course, (a) is preferable...
-- ab
On Dec 7, 2006, at 7:42 AM, Sanjay Dahiya (JIRA) wrote:
[ http://issues.apache.org/jira/browse/HADOOP-476?page=all ]
Sanjay Dahiya updated HADOOP-476:
---------------------------------
Attachment: Hadoop-476.patch
This patch replaces commandline processing with apache CLI2.
Input validation is not done as part of this patch as thats moved
to JobClient now. It validates -mapper, -reducer, -combiner, -file
attributes to be files present and accessible on disk.
It doesn't validate a file to be an executable, that is not
available through java File API until Java 6. For now it runs
"chmod +x" on the file if its present and accessible.
comments?
Streaming should check for correctness of the task
--------------------------------------------------
Key: HADOOP-476
URL: http://issues.apache.org/jira/browse/HADOOP-476
Project: Hadoop
Issue Type: Bug
Components: contrib/streaming
Reporter: arkady borkovsky
Assigned To: Sanjay Dahiya
Attachments: Hadoop-476.patch
Currently, if anythin is wrong with streaming job, it dies
without any explanation.
Before creating and running actual MapReduce job, Streaming
should check if:
-- the executables (or scripts) for -mapper and -reducer are
available and have right permissions
-- the input fragments exist
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the
administrators: http://issues.apache.org/jira/secure/
Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/
software/jira
