[
https://issues.apache.org/jira/browse/HADOOP-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591478#action_12591478
]
Rick Cox commented on HADOOP-3280:
----------------------------------
I really don't think a default ulimit is a good idea. The solution seems very
specific to a particular problem, and, though we're heavy users of streaming,
we've never encountered this need. Will most streaming users run into processes
using too much virtual address space? And will they expect that hadoop, by
default, would limit that?
The flip side is that many streaming users will conservatively set Xmx (since
Java's behavior with that setting is not comparable to ulimit -v, and the real
work is being done by the streaming app anyway), and then wonder why their
streaming processes refuse to start or fail randomly. For example, here's what
{{ls}} does when it exceeds the {{ulimit -v}} on RHEL3:
{code}
$ bash -c 'ulimit -v 10; ls'
$
{code}
No error message is produced (bash -c doesn't even print the "Killed" message).
The exit status is 137, but otherwise, there's no indication of what happened
or why, and when the user tries to reproduce it outside of hadoop, it probably
won't happen unless they know to set the ulimit -v.
> virtual address space limits break streaming apps
> -------------------------------------------------
>
> Key: HADOOP-3280
> URL: https://issues.apache.org/jira/browse/HADOOP-3280
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Rick Cox
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3280_0_20080418.patch, patch-3280.txt,
> patch-3280.txt
>
>
> HADOOP-2765 added a mandatory, hard virtual address space limit to streaming
> apps based on the Java process's -Xmx setting.
> This makes it impossible to run a 64-bit streaming app that needs large
> address spaces under a 32-bit JVM, even if one is otherwise willing to
> dramatically increase the -Xmx setting without cause. Also, unlike Java's
> -Xmx limit, the virtual address space limit for an arbitrary UNIX process
> does not necessarily correspond to RAM usage, so it's likely to be a
> relatively difficult to configure limit.
> 2765 was originally opened to allow an optional wrapper script around
> streaming tasks, one use case for which was setting a ulimit. That approach
> seems much less intrusive and more flexible than the final implementation.
> The ulimit can also be trivially set by the streaming task itself without any
> support from Hadoop.
> Marking this as an 0.17 blocker because it will break deployed apps and there
> is no workaround available.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
