[
https://issues.apache.org/jira/browse/SAMZA-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959524#comment-13959524
]
Chris Riccomini commented on SAMZA-58:
--------------------------------------
bq. MapReduce job computes the progress as following: 1. dividing a job into
several phases and each phase has a weight; 2. in Map/Reduce phase, the
sub-progress = # completed tasks / # total tasks. I'm not sure whether it is a
good idea to let progress = completedTasks / taskCount in SamzaAppMasterState
I think the trick is that Samza jobs tend to be infinitely running, so the
containers never complete. We could report progress the way you define, though.
It's just that most jobs would report a progress of 0 forever.
Also, at first glance, this patch looks good. Have you tried running it through
hello-samza? We don't really have integration tests right now, but it'd be good
to try it on a YARN grid first. :) If you can't, I can give it a shot, but it
might take a bit to get some time to do it.
> Use YARN's AMRMClientAsync client library
> -----------------------------------------
>
> Key: SAMZA-58
> URL: https://issues.apache.org/jira/browse/SAMZA-58
> Project: Samza
> Issue Type: Bug
> Components: yarn
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Zhijie Shen
> Attachments: SAMZA-58.1.patch
>
>
> YARN 2.2.0 has a nice Async API for clients and AMs. This API didn't exist
> when we did the initial YARN integration for Samza. We should upgrade Samza
> to use these new APIs.
> The API is loosely based off Samza's own AM code, so we can probably strip
> out a lot of it (YarnAppMaster, mainly), and switch everything over to the
> call-back based API.
> For details, see:
> https://issues.apache.org/jira/browse/YARN-417
> This new API is used in DistributedShell now, so we can use that for testing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
