[
https://issues.apache.org/jira/browse/MESOS-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567860#comment-14567860
]
Benjamin Mahler commented on MESOS-2735:
----------------------------------------
{quote}
Finally, one of the advantages of the pull model is that it's easier to reason
about because we don't have "anonymous" lambdas that cause execution in some
other random place in the code (i.e., you can easily see in the slave where the
future that gets returned from `ResourceEstimator::estimate()` gets handled).
In addition, the ResourceEstimator remains "functional" in the sense that it
just has to return some value (or a future) from it's functions versus invoking
some callback that causes something to get run some other place (and in fact,
may also block, so isn't it safer for the ResourceEstimator to invoke the
callback in it's own `async`?).
The invocation of the `ResourceEstimator::estimate()` followed by the `.then`
is a nice pattern that let's us compose with other things as well, which is
harder to do with the lambda style callbacks and why we've avoided it where
we've been able (in fact, I'm curious which place in the code are you imitating
here?).
{quote}
The anonymous lambdas are because we do not have an abstraction to represent an
asychronous stream of results:
{code}
// Returns a stream of the unused resources that can be oversubscribed.
Stream<Resources> unused();
{code}
This allows the caller to hold the stream and do composition on the next()
Future result, so we achieve push but we keep the ability to do composition.
{{http::Pipe}} is a string-specialized form of this and {{process::Queue<T>}}
is an infinite-only version of {{process::Stream<T>}}. An important additional
property of {{process::Stream<T>}} is that it should provide "asynchronous
back-pressure" (e.g. unix pipes provide synchronous back-pressure by blocking
writes, however we need an asynchronous mechanism to back-pressure the writes),
to control the DoS concerns you guys have mentioned.
Take the allocator for example, it seems non-intuitive if the master had to
continually call a method to obtain the next allocation from
{{Future<Allocation> allocation()}}. At least, it requires more understanding
than what it expressed in the return type. Whereas, if we have a
{{Stream<Allocation> initialize()}} or {{Stream<Allocation> allocations()}},
we're clearly capturing the semantics within the return type. Back-pressure is
the key to ensure that the master's speed of consumption limits the rate at
which the allocator makes allocations. Today, {{process::Queue<T>}} is an
equivalent replacement since allocations occur as an infinite stream and there
is no back-pressuring.
> Change the interaction between the slave and the resource estimator from
> polling to pushing
> --------------------------------------------------------------------------------------------
>
> Key: MESOS-2735
> URL: https://issues.apache.org/jira/browse/MESOS-2735
> Project: Mesos
> Issue Type: Bug
> Reporter: Jie Yu
> Assignee: Jie Yu
> Labels: twitter
>
> This will make the semantics more clear. The resource estimator can control
> the speed of sending resources estimation to the slave.
> To avoid cyclic dependency, slave will register a callback with the resource
> estimator and the resource estimator will simply invoke that callback when
> there's a new estimation ready. The callback will be a defer to the slave's
> main event queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
