[jira] [Commented] (MESOS-8302) Improve master failover performance.
[
https://issues.apache.org/jira/browse/MESOS-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282768#comment-16282768
]
Benjamin Mahler commented on MESOS-8302:
{noformat}
commit 4fe3bdb8ed5e8d4ddc894ff7fd5cbcd3183526be
Author: Benjamin Mahler
Date: Thu Dec 7 12:03:58 2017 -0800
Eliminated some copying of tasks / executors in agent re-registration.
Review: https://reviews.apache.org/r/64428
{noformat}
{noformat}
commit 54e03f3ceae87c69e5a01a585a23ec87f2dd8206
Author: Benjamin Mahler
Date: Thu Dec 7 14:29:03 2017 -0800
Fixed skipping of completed frameworks in the master failover benchmark.
This benchmark was previously accidentally skipping the completed
frameworks, due to an incorrect loop.
Review: https://reviews.apache.org/r/64429
{noformat}
> Improve master failover performance.
>
>
> Key: MESOS-8302
> URL: https://issues.apache.org/jira/browse/MESOS-8302
> Project: Mesos
> Issue Type: Improvement
> Components: master
>Reporter: Benjamin Mahler
>Assignee: Dmitry Zhuk
>
> This is somewhat more like an epic, but will track the different improvements
> here for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-8302) Improve master failover performance.
[
https://issues.apache.org/jira/browse/MESOS-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280840#comment-16280840
]
Michael Park commented on MESOS-8302:
-
{noformat}
commit 2e83009ef4d10e1cbfb3a55c8adefe37b916f06a
Author: Dmitry Zhuk
Date: Wed Dec 6 10:45:14 2017 -0800
Reduced copying of agent data during reregistration.
{noformat}
{noformat}
commit 8b4228049292cab6a8c4a3680de3c1aa5e72a9ff
Author: Dmitry Zhuk
Date: Wed Dec 6 09:03:49 2017 -0800
Changed agent reregistration to work with messages directly.
`reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
of using protobuf arena, and allows passing message through dispatch
chain without making any copies. Conversion of repeated message fields
to `std::vector`s is performed only when needed.
Review: https://reviews.apache.org/r/63914/
{noformat}
> Improve master failover performance.
>
>
> Key: MESOS-8302
> URL: https://issues.apache.org/jira/browse/MESOS-8302
> Project: Mesos
> Issue Type: Improvement
> Components: master
>Reporter: Benjamin Mahler
>Assignee: Dmitry Zhuk
>
> This is somewhat more like an epic, but will track the different improvements
> here for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-8302) Improve master failover performance.
[
https://issues.apache.org/jira/browse/MESOS-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279353#comment-16279353
]
Michael Park commented on MESOS-8302:
-
{noformat}
commit 6839897c5464fce6b8cbd253d959a7e2efd72987
Author: Dmitry Zhuk
Date: Tue Dec 5 11:21:27 2017 -0800
Made `Event` move-only in libprocess.
Review: https://reviews.apache.org/r/64347/
{noformat}
{noformat}
commit c9e6a03c02e9f8dc040b937ccd5ae89e5530fd7e
Author: Dmitry Zhuk
Date: Tue Dec 5 11:21:11 2017 -0800
Used `std::move` for `Event`s consumption in the master.
Review: https://reviews.apache.org/r/63641/
{noformat}
> Improve master failover performance.
>
>
> Key: MESOS-8302
> URL: https://issues.apache.org/jira/browse/MESOS-8302
> Project: Mesos
> Issue Type: Improvement
> Components: master
>Reporter: Benjamin Mahler
>Assignee: Dmitry Zhuk
>
> This is somewhat more like an epic, but will track the different improvements
> here for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-8302) Improve master failover performance.
[
https://issues.apache.org/jira/browse/MESOS-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279350#comment-16279350
]
Michael Park commented on MESOS-8302:
-
{noformat}
commit c9462f4927cfffb1f3a90827467ded730c0f40b9
Author: Dmitry Zhuk dz...@twopensource.com
Date: Tue Dec 5 10:11:28 2017 -0800
Migrated to use the `EventConsumer` interface.
Review: https://reviews.apache.org/r/63632/
{noformat}
{noformat}
commit 6b91f62769a1f8c525162fc716b1d5c231c77811
Author: Dmitry Zhuk dz...@twopensource.com
Date: Tue Dec 5 10:11:23 2017 -0800
Separated `Event` visitation and consumption.
This introduces the `EventConsumer` interface, which add support of
`Event`s with move-only data. This allows consumers to move data out of
`Event`, rather than copying it. This is required to implement move-only
objects support in `defer` to guarantee that deferred function object is
invoked only once, allowing deferred parameters to be moved into call.
Review: https://reviews.apache.org/r/63631/
{noformat}
{noformat}
commit 1074a9c3fd7aa9d2e4b484f86dbe657271abecc0
Author: Dmitry Zhuk dz...@twopensource.com
Date: Tue Dec 5 07:23:28 2017 -0800
Fixed the signature of `CallableOnce::operator()`.
This changes `operator()` signature to match the one defined in
`CallableOnce` template parameter. Previously used form incorrectly
specifies that `operator()` can be invoked with arbitrary number and
types of parameters, which can break other templates using SFINAE to
check if function object can be invoked with specific parameters.
Review: https://reviews.apache.org/r/64337/
{noformat}
> Improve master failover performance.
>
>
> Key: MESOS-8302
> URL: https://issues.apache.org/jira/browse/MESOS-8302
> Project: Mesos
> Issue Type: Improvement
> Components: master
>Reporter: Benjamin Mahler
>Assignee: Dmitry Zhuk
>
> This is somewhat more like an epic, but will track the different improvements
> here for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-8302) Improve master failover performance.
[
https://issues.apache.org/jira/browse/MESOS-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279341#comment-16279341
]
Benjamin Mahler commented on MESOS-8302:
{noformat}
commit f8e4f11e796b2b0d9bd101a7eb6f106e72cbaee1
Author: Dmitry Zhuk
Date: Mon Aug 7 11:01:04 2017 -0700
Reduced copying in `defer`, `dispatch` and `Future`.
This reduces number of copies made for each parameter in
a piece of code like this:
```
future.then(defer(pid, &SomeProcess::someMethod, param1, param2));
```
For the objects that do not support move semantics
(e.g., protobuf messages), number of copies is reduced from 8-10 to 6.
If move semantics is supported, then number of copies is reduced from
6-7 to 1 if parameter is passed with `std::move`, or 2 otherwise.
Review: https://reviews.apache.org/r/60003/
{noformat}
{noformat}
commit 834053d976e2db18c16e1612b3b723fe1c8ca1ac
Author: Dmitry Zhuk
Date: Thu Oct 12 14:45:26 2017 -0700
Used protobuf arenas for creating messages in ProtobufProcess.
When passing const protobuf messages and fields, we can allocate
the protobuf message within an arena. Arenas dramatically reduce
the number of malloc's involved. The use of arenas also improves
the cache locality of the protobuf memory.
Review: https://reviews.apache.org/r/62901/
{noformat}
{noformat}
commit f569b841cbd8d8b07eb906a711aeb2d395096af6
Author: Dmitry Zhuk
Date: Fri Oct 13 12:06:31 2017 -0700
Simplified RepeatedPtrField to vector conversion.
It's also possible that `vector` can take advantage of
`RepeatedPtrField` iterators implementing `RandomAccessIterator`
concept and avoid buffer resizes, when constructed from range.
Review: https://reviews.apache.org/r/62973/
{noformat}
> Improve master failover performance.
>
>
> Key: MESOS-8302
> URL: https://issues.apache.org/jira/browse/MESOS-8302
> Project: Mesos
> Issue Type: Improvement
> Components: master
>Reporter: Benjamin Mahler
>Assignee: Dmitry Zhuk
>
> This is somewhat more like an epic, but will track the different improvements
> here for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-8302) Improve master failover performance.
[
https://issues.apache.org/jira/browse/MESOS-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279335#comment-16279335
]
Benjamin Mahler commented on MESOS-8302:
{noformat}
commit 8ea9245e5c6d1f8b84219eb5d8b36f3cec1c6a7a
Author: Dmitry Zhuk
Date: Wed Nov 22 11:59:17 2017 -0700
Optimized resources logging in master.
When master logs agent or task resources, it uses `operator <<` for raw
protobuf data, which outputs resources in JSON format, and is rather
slow. However resources are known to be valid and refined when logged by
master, so it's faster to use `operator <<` after protobuf is converted
to `Resources`.
Review: https://reviews.apache.org/r/63959/
{noformat}
{noformat}
commit 24550e7b863fee371877ea9d1d9153e0f5054155
Author: Dmitry Zhuk
Date: Wed Nov 29 18:47:13 2017 -0800
Improved master failover performance by avoiding resource conversions.
RepeatedPtrField can be implicitly converted to Resources,
leading to hidden multiple resources conversions on performance-critical
paths in master. For example, operator += relies on implicit
conversion, when invoked with RepeatedPtrField argument.
Using protobuf also implies data validation and sanitization, e.g. when
converting to Resources, as protobuf generally comes from untrusted
sources. By doing conversion only once, and then reusing the result,
we save on these checks as well, as operations on Resources are
generally faster as they can trust data in Resources.
Review: https://reviews.apache.org/r/64028/
{noformat}
{noformat}
commit b7ad2c0d4e7308a70049a6a04f19e3709df0e539
Author: Dmitry Zhuk
Date: Tue Nov 21 10:11:46 2017 -0800
Preallocated buffer for resources conversion.
When converting collections of protobuf `Resource` to `Resources`,
`std::vector` could be resized several times. This patch ensures that
there is enough capacity to fit all resources in `vector` and avoid
resizes.
Review: https://reviews.apache.org/r/63960/
{noformat}
> Improve master failover performance.
>
>
> Key: MESOS-8302
> URL: https://issues.apache.org/jira/browse/MESOS-8302
> Project: Mesos
> Issue Type: Improvement
> Components: master
>Reporter: Benjamin Mahler
>Assignee: Dmitry Zhuk
>
> This is somewhat more like an epic, but will track the different improvements
> here for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
