On second thoughts if we plan to support file patterns for input
paths both in streaming and general map reduce. Then either we should
be match and see if some files matching the pattern exist in DFS or
not check at all. Otherwise this check will break functionality.
Sanjay
On 30-Nov-06, at 7:23 PM, Sanjay Dahiya wrote:
It does take care of existence input in DFS, scripts on local FS
and read permission for scripts. Execute permission check on a file
is coming in java 6, so for now I have chmod +x on the file after
checking other permissions.
On descriptive error messages there is another JIRA on that
HADOOP-676, that patch contains some changes to improve errors and
throwing new exceptions.
Sanjay
On 30-Nov-06, at 1:46 AM, Arkady Borkovsky wrote:
Sanjay,
does you implementation that ALL the scripts and executable
available and have right permissions, and that the input exists?
Also: what about descriptive error messages?
-- ab
On Nov 29, 2006, at 7:43 AM, Sanjay Dahiya (JIRA) wrote:
[ http://issues.apache.org/jira/browse/HADOOP-476?
page=comments#action_12454343 ]
Sanjay Dahiya commented on HADOOP-476:
--------------------------------------
I have the cli2 implementation in place, there is one issue with
cli2 which is holding it back. Command line arguments with -
jobconf and -env get merged in the same properties set in CLI's
parser and after parsing we dont know what property was passed
with which trigger.
I have raised on CLI jira CLI-126. I'll submit the patch once i
resolve this one..
(http://issues.apache.org/jira/browse/CLI-126) in case auto link
doesnt work for other projects.
Streaming should check for correctness of the task
--------------------------------------------------
Key: HADOOP-476
URL: http://issues.apache.org/jira/browse/
HADOOP-476
Project: Hadoop
Issue Type: Bug
Components: contrib/streaming
Reporter: arkady borkovsky
Assigned To: Sanjay Dahiya
Currently, if anythin is wrong with streaming job, it dies
without any explanation.
Before creating and running actual MapReduce job, Streaming
should check if:
-- the executables (or scripts) for -mapper and -reducer are
available and have right permissions
-- the input fragments exist
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the
administrators: http://issues.apache.org/jira/secure/
Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/
software/jira
