[
https://issues.apache.org/jira/browse/MESOS-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553007#comment-14553007
]
Jie Yu commented on MESOS-2735:
-------------------------------
Thanks for all the comments!
tl;dr: I am still not fully convinced by the pull model, but I think we should
move on and not spending too much time arguing on this. So I'll do the change
as [~benjaminhindman] suggested (use 'Future<Resources> oversubscribable()' in
ResourceEstimator). [~nnielsen], does this sound good to you?
The following are the reasons why I am still not convinced. Just write them
down for future references:
1) Regarding the DoS concern. I think the pull model is vulnerable to that as
well. In some sense, it's more vulnerable to that IMO. Why? Imagine that at
Twitter, we want to write a constant resource estimator (i.e., returns a fix
amount of cpus to the slave when the slave asks about how many resources can be
oversubscribed). Now, one natural implementation will be like the following:
{code}
Future<Resources> oversubscribable()
{
return Resources::parse("cpus:2").get();
}
{code}
The estimator writer's thought process could be like: whenever the slave asks
me about how many resources can be oversubscribed, I'll tell it it's 2 cpus.
Well, this _natural_ implementation will cause a DoS problem for the slave
because its queue will be flooded, and it'll flood the logging file as well. I
would argue that a push model makes the programmer less likely to make such
mistake. Why? The programmer needs to write a tight loop (either synchronous or
asynchronous) to flood the slave, and we usually think twice when writing a
tight loop, don't we?
2) I liked the composability and and functional arguments Ben made in the
ticket. I think the push model mimics the actor style message passing model we
also constantly used in our code base. Just imagine replacing the lambda with
PID<Slave>, and invoking the callback becomes calling dispatch on the pid. I
guess the message passing model is more suitable for entities across process
boundaries (e.g., master <-> slave, framework <-> master). Otherwise, we prefer
the funcitonal style in general. [~benjaminhindman], is that what you're trying
to convey? I think I agree with you on this. Just wondering if in the future,
we ever want to move resources estimator to a separate unix process?
> Change the interaction between the slave and the resource estimator from
> polling to pushing
> --------------------------------------------------------------------------------------------
>
> Key: MESOS-2735
> URL: https://issues.apache.org/jira/browse/MESOS-2735
> Project: Mesos
> Issue Type: Bug
> Reporter: Jie Yu
> Assignee: Jie Yu
> Labels: twitter
>
> This will make the semantics more clear. The resource estimator can control
> the speed of sending resources estimation to the slave.
> To avoid cyclic dependency, slave will register a callback with the resource
> estimator and the resource estimator will simply invoke that callback when
> there's a new estimation ready. The callback will be a defer to the slave's
> main event queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
