[
https://issues.apache.org/jira/browse/HADOOP-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591104#action_12591104
]
Sameer Paranjpye commented on HADOOP-3280:
------------------------------------------
> On Linux, we should be able to do this as part of /etc/security/limits.conf.
limits.conf is a blunt tool. Anything set there will apply to all processes
belonging to a particular user or group. Tasks in Hadoop run with the uid of
the tasktrackers, so restricting processes by username/group has no effect.
Even in a world with HoD it is reasonable to expect tasks to be constrained
differently from tasktrackers and jobtrackers.
Long term we will likely run tasktrackers as root and setuid to the submitting
user and at that point limits.conf will suffice.
> virtual address space limits break streaming apps
> -------------------------------------------------
>
> Key: HADOOP-3280
> URL: https://issues.apache.org/jira/browse/HADOOP-3280
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Rick Cox
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3280_0_20080418.patch
>
>
> HADOOP-2765 added a mandatory, hard virtual address space limit to streaming
> apps based on the Java process's -Xmx setting.
> This makes it impossible to run a 64-bit streaming app that needs large
> address spaces under a 32-bit JVM, even if one is otherwise willing to
> dramatically increase the -Xmx setting without cause. Also, unlike Java's
> -Xmx limit, the virtual address space limit for an arbitrary UNIX process
> does not necessarily correspond to RAM usage, so it's likely to be a
> relatively difficult to configure limit.
> 2765 was originally opened to allow an optional wrapper script around
> streaming tasks, one use case for which was setting a ulimit. That approach
> seems much less intrusive and more flexible than the final implementation.
> The ulimit can also be trivially set by the streaming task itself without any
> support from Hadoop.
> Marking this as an 0.17 blocker because it will break deployed apps and there
> is no workaround available.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
