[
https://issues.apache.org/jira/browse/HADOOP-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591379#action_12591379
]
Rick Cox commented on HADOOP-3280:
----------------------------------
* Bash is currently an implementation detail of TaskLog. Defining the prologue
as something that runs in the shell locks bash into the API (as users will then
have bash-specific fragments in config files), which is a quite different level
of requirement.
* Task startup is a very powerful place to have a hook, and I think many
desired customizations could be framed as task wrappers. But many of them also
require a true wrapper, not just a command run before the real command. For
example, I've long wanted a hook to run commands after the task exits (e.g.
task log collection). A wrapper could provide that. Sameer mentioned wanting to
setuid to the user who submitted the job; that could also be done by supplying
a wrapper (it it had someway to get the submitter's username).
* For more complicated setup tasks, it may be desirable to use a language
besides bash.
For these reasons, I'm still in favor of the prologue being defined as a prefix
to the command rather than a fragment of bash that is run before the command.
On the latest patch: since TaskRunner applies the prologue to all tasks, will
it still be necessary in PipeMapRed and Application too?
> virtual address space limits break streaming apps
> -------------------------------------------------
>
> Key: HADOOP-3280
> URL: https://issues.apache.org/jira/browse/HADOOP-3280
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Rick Cox
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3280_0_20080418.patch, patch-3280.txt,
> patch-3280.txt
>
>
> HADOOP-2765 added a mandatory, hard virtual address space limit to streaming
> apps based on the Java process's -Xmx setting.
> This makes it impossible to run a 64-bit streaming app that needs large
> address spaces under a 32-bit JVM, even if one is otherwise willing to
> dramatically increase the -Xmx setting without cause. Also, unlike Java's
> -Xmx limit, the virtual address space limit for an arbitrary UNIX process
> does not necessarily correspond to RAM usage, so it's likely to be a
> relatively difficult to configure limit.
> 2765 was originally opened to allow an optional wrapper script around
> streaming tasks, one use case for which was setting a ulimit. That approach
> seems much less intrusive and more flexible than the final implementation.
> The ulimit can also be trivially set by the streaming task itself without any
> support from Hadoop.
> Marking this as an 0.17 blocker because it will break deployed apps and there
> is no workaround available.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
