[
https://issues.apache.org/jira/browse/MAPREDUCE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510236#comment-13510236
]
Mariappan Asokan commented on MAPREDUCE-4847:
---------------------------------------------
Hi Peng Lei,
By invoking "sh" in Hadoop streaming code, you may be creating a dependency
on UNIX(or some ugly branching based on OS.) On Windows, there is no "sh"
unless you install additional software like cygwin. Why can't you put your
complex command in a file, give execute permission to the file, and then
specify the file name as mapper/reducer command? You can create this file on
the fly if you want. This way, the streaming command will work in its current
form on UNIX and Windows and there is no change needed.
-- Asokan
> Command Parsing in Hadoop Streaming
> -----------------------------------
>
> Key: MAPREDUCE-4847
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4847
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Reporter: Peng Lei
> Labels: features
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Hadoop streaming parse the mapper and reducer commands by itself, this is not
> a good choice, when I write a complex mapper/reducer script inline, such as
> 'perl -ne ...', it don't work.
> An alternative way is to send the command to the shell, simply create new
> process(sh -c "command_and_args"), this not also simplize the streaming code,
> but also improve its capability!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
