[
https://issues.apache.org/jira/browse/MESOS-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577659#comment-14577659
]
Jie Yu commented on MESOS-2818:
-------------------------------
Here is a new proposal. I personally think that's better than all existing
proposals. Here is the summary.
1) We kill the ResourceMonitor and move its logic (currently, only 'usages()')
into the slave
{code}
Future<ResourceUsage> Slave::usage() { ... }
message ResourceUsage {
message Executor {
optional ExecutorInfo executor_info = 1;
repeated Resource allocated = 2;
repeated ResourceStatistics statistics = 3;
}
repeated Resource total = 1; // Slave's total resources.
repeated Executor executors = 2; // Per-executor allocated/usage information.
}
{code}
2) Slave collect per-executor allocated information when the resource estimator
calls 'usage()':
{code}
Future<ResourceUsage> Slave::usage()
{
ResourceUsage usage;
list<Future<ResourceStatistics> futures;
foreachvalue (const Framework* framework, frameworks) {
foreachvalue (const Executor* executor, framework->executors) {
ResourceUsage::Executor* entry = usage.add_executors()'
entry->set_allocated(...);
futures.push_back(containerizer->usage(executor->containerId));
}
}
return await(futures)
.then(defer(self(), &Self::_usage, usage, lambda::_1));
}
{code}
3) In this way, it's guaranteed that if an executor has 'statistics', it will
have 'allocated' info available. Vice not versa. But in that case, we can just
make conservative estimation (i.e., assuming all resources allocated are
consumed).
> Pass 'allocated' resources for each executor to the resource estimator.
> -----------------------------------------------------------------------
>
> Key: MESOS-2818
> URL: https://issues.apache.org/jira/browse/MESOS-2818
> Project: Mesos
> Issue Type: Task
> Reporter: Jie Yu
> Assignee: Jie Yu
>
> Resource estimator obviously need this information to calculate, say the
> usage slack. Now the question is how. There are two approaches:
> 1) Pass in the allocated resources for each executor through the
> 'oversubscribable()' interface.
> 2) Let containerizer return total resources allocated for each container when
> 'usages()' are invoked.
> I would suggest to take route (1) for several reasons:
> 1) Eventually, we'll need to pass in slave's total resources to the resource
> estimator (so that RE can calculate allocation slack). There is no way that
> we can get that from containerizer. The slave's total resources keep changing
> due to dynamic reservation. So we cannot pass in the slave total resources
> during initialization.
> 2) The current implementation of usages() might skip some containers if it
> fails to get statistics for that container (not an error). This will cause
> in-complete information to the RE.
> 3) We may want to calculate 'unallocated = total - allocated' so that we can
> send allocation slack as well. Getting 'total' and 'allocated' from two
> different components might result in inconsistent value. Remember that
> 'total' keeps changing due to dynamic reservation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
