virtual address space limits break streaming apps
-------------------------------------------------
Key: HADOOP-3280
URL: https://issues.apache.org/jira/browse/HADOOP-3280
Project: Hadoop Core
Issue Type: Bug
Components: contrib/streaming
Reporter: Rick Cox
Priority: Blocker
Fix For: 0.17.0
HADOOP-2765 added a mandatory, hard virtual address space limit to streaming
apps based on the Java process's -Xmx setting.
This makes it impossible to run a 64-bit streaming app that needs large address
spaces under a 32-bit JVM, even if one is otherwise willing to dramatically
increase the -Xmx setting without cause. Also, unlike Java's -Xmx limit, the
virtual address space limit for an arbitrary UNIX process does not necessarily
correspond to RAM usage, so it's likely to be a relatively difficult to
configure limit.
2765 was originally opened to allow an optional wrapper script around streaming
tasks, one use case for which was setting a ulimit. That approach seems much
less intrusive and more flexible than the final implementation. The ulimit can
also be trivially set by the streaming task itself without any support from
Hadoop.
Marking this as an 0.17 blocker because it will break deployed apps and there
is no workaround available.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
