[
https://issues.apache.org/jira/browse/SAMZA-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated SAMZA-58:
-----------------------------
Attachment: SAMZA-58.1.patch
I've created a patch, which use AMRMClientAsync to replace AMRMClient. In this
case YarnAppMaster is no longer required, but I keep the loop, because I want
to block SamzaAppMaster termination until all the listeners are done.
Two related issues that I've found, but haven't fixed in this patch:
1. The heartbeat interval was set to 1s by hard code. I kept it, but ideally
it's good to be configurable. Thoughts?
2. Samza will always report progress = 0 to YARN. Though it doesn't do harm to
the application execution, but users are going to be confused when monitoring
Samza jobs' progress
NMClient can be replaced by NMClientAsync as well. However, we need to think of
the logic when startContainer becomes nonblocking. Therefore, I prefer to solve
it separately. Thoughts?
> Use YARN's *Async API
> ---------------------
>
> Key: SAMZA-58
> URL: https://issues.apache.org/jira/browse/SAMZA-58
> Project: Samza
> Issue Type: Bug
> Components: yarn
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Zhijie Shen
> Attachments: SAMZA-58.1.patch
>
>
> YARN 2.2.0 has a nice Async API for clients and AMs. This API didn't exist
> when we did the initial YARN integration for Samza. We should upgrade Samza
> to use these new APIs.
> The API is loosely based off Samza's own AM code, so we can probably strip
> out a lot of it (YarnAppMaster, mainly), and switch everything over to the
> call-back based API.
> For details, see:
> https://issues.apache.org/jira/browse/YARN-417
> This new API is used in DistributedShell now, so we can use that for testing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
