[
https://issues.apache.org/jira/browse/S4-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463102#comment-13463102
]
Matthieu Morel commented on S4-25:
----------------------------------
Thanks Sean for contributing this patch!
I had a look today, and I was able to make things work. There were a few minor
(but blocking) issues in the Client and ApplicationMaster, and things we'll
have to polish for the classpath of the submitted S4 job.
The issue you reported is due to the following (and I agree the error message
from Yarn is not very helpful for non-initiated!):
* When creating a connection to the resource manager, the client needs to use
the resource manager ports, not the scheduler ones
(YarnConfiguration.RM_ADDRESS,YarnConfiguration.DEFAULT_RM_ADDRESS). The
scheduler is initialized differently and cannot understand the
"ClientRMProtocolPB"
I'll have a further look at how to provide a clean integration, but the
approach you took looks like a promising one!
> Write S4 Application Master to deploy S4 in Yarn
> ------------------------------------------------
>
> Key: S4-25
> URL: https://issues.apache.org/jira/browse/S4-25
> Project: Apache S4
> Issue Type: New Feature
> Reporter: J Mohamed Zahoor
> Fix For: 0.6
>
> Attachments: S4-ApplicationMaster.diff, S4-Client.diff,
> S4-Constants.diff, S4-YARN-1.patch
>
>
> On the lines of s4PigWrapper, write a s4 application master to host s4 piper
> inside Hadoop Yarn. This could be useful not only for reading data stored in
> hadoop ( to build or train a model)... But we could make use of the resource
> manager to deploy s4 instances in remote machine and monitor them. In short,
> we could make use of most of the resource management , scheduling and other
> good stuff in Yarn.
> - Yarn is useful to deploy and launch s4 instances.
> - It still requires deploying node managers on each box which means it will
> be useful if one is running more than one s4 process on a node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
