[
https://issues.apache.org/jira/browse/S4-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503147#comment-13503147
]
Daniel Gómez Ferro commented on S4-25:
--------------------------------------
Thanks for the update.
I think I wasn't clear enough on my previous comment. My idea was to decouple
the "Container Memory" from the "Node/JVM Memory", since the JVM memory we set
through -Xmx has to be strictly less than the reserved memory at the container
level, otherwise we could run into the issues reported by Frank Zheng on the
mailing list (see
http://mail-archives.apache.org/mod_mbox/incubator-s4-user/201211.mbox/%3CCAAf2GfeQK_-zx7sCvpP_9euTsDJL0VbDTS08iXMyQw1%3D8%2B1tOg%40mail.gmail.com%3E
) What I propose is to have two separate parameters, one for the container
memory and another for the jvm memory.
Do you think that's too much complexity?
There's a minor issue in the S4ApplicationMaster, it's not taking into account
the -s4NodeMemory parameter to compute the container memory (it always picks
YARN's minimum container memory regardless of the passed parameters).
> Write S4 Application Master to deploy S4 in Yarn
> ------------------------------------------------
>
> Key: S4-25
> URL: https://issues.apache.org/jira/browse/S4-25
> Project: Apache S4
> Issue Type: New Feature
> Reporter: J Mohamed Zahoor
> Fix For: 0.6
>
> Attachments: S4-ApplicationMaster.diff, S4-Client.diff,
> S4-Constants.diff, S4-YARN-1.patch
>
>
> On the lines of s4PigWrapper, write a s4 application master to host s4 piper
> inside Hadoop Yarn. This could be useful not only for reading data stored in
> hadoop ( to build or train a model)... But we could make use of the resource
> manager to deploy s4 instances in remote machine and monitor them. In short,
> we could make use of most of the resource management , scheduling and other
> good stuff in Yarn.
> - Yarn is useful to deploy and launch s4 instances.
> - It still requires deploying node managers on each box which means it will
> be useful if one is running more than one s4 process on a node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
