Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-12-06 Thread Dmitry Zhuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/
---

(Updated Dec. 6, 2017, 6:31 p.m.)


Review request for mesos, Benjamin Mahler and Michael Park.


Changes
---

Rebased.


Bugs: MESOS-6972
https://issues.apache.org/jira/browse/MESOS-6972


Repository: mesos


Description
---

Tasks can be moved into master's internal data structures from message
to save some cycles on copying the data.


Diffs (updated)
-

  src/master/master.hpp d42acae5a7d31dccf7073b57c3f2ddcfadea9e55 
  src/master/master.cpp 16cdde73b90b2e1514c6196850b8df75a3b6ff28 


Diff: https://reviews.apache.org/r/63915/diff/4/

Changes: https://reviews.apache.org/r/63915/diff/3-4/


Testing
---

make check

Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
--enable-lock-free-event-queue --enable-last-in-first-out-fixed-size-semaphore`
`./mesos-tests.sh --benchmark 
--gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
```
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
Starting reregistration for all agents
Reregistered 2000 agents with a total of 10 running tasks and 10 
completed tasks in 16.202206916secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
 (30065 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
Starting reregistration for all agents
Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
tasks in 30.509804836secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
 (57145 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
Starting reregistration for all agents
Reregistered 2 agents with a total of 10 running tasks and 0 completed 
tasks in 22.581999748secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
 (39629 ms)
```

On this chain of patches:
```
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
Starting reregistration for all agents
Reregistered 2000 agents with a total of 10 running tasks and 10 
completed tasks in 8.456615936secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
 (22659 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
Starting reregistration for all agents
Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
tasks in 15.09102354secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
 (43828 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
Starting reregistration for all agents
Reregistered 2 agents with a total of 10 running tasks and 0 completed 
tasks in 16.122729767secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
 (33182 ms)
```


Thanks,

Dmitry Zhuk

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-12-06 Thread Michael Park 


> On Dec. 4, 2017, 11:37 a.m., Michael Park wrote:
> > src/master/master.hpp
> > Line 128 (original), 128 (patched)
> > 
> >
> > Maybe we can consider making this `vector&&` as discussed in 
> > https://reviews.apache.org/r/63914/
> 
> Dmitry Zhuk wrote:
> This will make some difficulties for `registerSlave` which also uses this 
> constructor. Shall we migrate `registerSlave` to using message similar to 
> `reregisterSlave`?

That sounds ideal, but I don't think it should necessarily be part of this 
work. We'll follow-up on it.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/#review192746
---


On Nov. 21, 2017, 9:53 a.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63915/
> ---
> 
> (Updated Nov. 21, 2017, 9:53 a.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Tasks can be moved into master's internal data structures from message
> to save some cycles on copying the data.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
> 
> 
> Diff: https://reviews.apache.org/r/63915/diff/3/
> 
> 
> Testing
> ---
> 
> make check
> 
> Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
> --enable-lock-free-event-queue 
> --enable-last-in-first-out-fixed-size-semaphore`
> `./mesos-tests.sh --benchmark 
> --gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
> On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 16.202206916secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (30065 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 30.509804836secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (57145 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 22.581999748secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (39629 ms)
> ```
> 
> On this chain of patches:
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 8.456615936secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (22659 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 15.09102354secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (43828 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 16.122729767secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (33182 ms)
> ```
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-12-05 Thread Dmitry Zhuk 


> On Dec. 4, 2017, 7:37 p.m., Michael Park wrote:
> > src/master/master.hpp
> > Line 128 (original), 128 (patched)
> > 
> >
> > Maybe we can consider making this `vector&&` as discussed in 
> > https://reviews.apache.org/r/63914/

This will make some difficulties for `registerSlave` which also uses this 
constructor. Shall we migrate `registerSlave` to using message similar to 
`reregisterSlave`?


- Dmitry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/#review192746
---


On Nov. 21, 2017, 5:53 p.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63915/
> ---
> 
> (Updated Nov. 21, 2017, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Tasks can be moved into master's internal data structures from message
> to save some cycles on copying the data.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
> 
> 
> Diff: https://reviews.apache.org/r/63915/diff/3/
> 
> 
> Testing
> ---
> 
> make check
> 
> Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
> --enable-lock-free-event-queue 
> --enable-last-in-first-out-fixed-size-semaphore`
> `./mesos-tests.sh --benchmark 
> --gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
> On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 16.202206916secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (30065 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 30.509804836secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (57145 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 22.581999748secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (39629 ms)
> ```
> 
> On this chain of patches:
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 8.456615936secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (22659 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 15.09102354secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (43828 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 16.122729767secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (33182 ms)
> ```
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-12-04 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/#review192815
---



FAIL: Failed to apply the dependent review: 63630.

Failed command: `python.exe .\support\apply-reviews.py -n -r 63630`

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/63915

Relevant logs:

- 
[apply-review-63630-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/63915/logs/apply-review-63630-stdout.log):

```
error: patch failed: 3rdparty/stout/include/stout/lambda.hpp:285
error: 3rdparty/stout/include/stout/lambda.hpp: patch does not apply
```

- Mesos Reviewbot Windows


On Nov. 21, 2017, 5:53 p.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63915/
> ---
> 
> (Updated Nov. 21, 2017, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Tasks can be moved into master's internal data structures from message
> to save some cycles on copying the data.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
> 
> 
> Diff: https://reviews.apache.org/r/63915/diff/3/
> 
> 
> Testing
> ---
> 
> make check
> 
> Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
> --enable-lock-free-event-queue 
> --enable-last-in-first-out-fixed-size-semaphore`
> `./mesos-tests.sh --benchmark 
> --gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
> On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 16.202206916secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (30065 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 30.509804836secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (57145 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 22.581999748secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (39629 ms)
> ```
> 
> On this chain of patches:
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 8.456615936secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (22659 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 15.09102354secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (43828 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 16.122729767secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (33182 ms)
> ```
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-12-04 Thread Benjamin Mahler 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/#review192763
---


Ship it!




Very happy to see this!

Looks good, modulo mpark's suggestion.

- Benjamin Mahler


On Nov. 21, 2017, 5:53 p.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63915/
> ---
> 
> (Updated Nov. 21, 2017, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Tasks can be moved into master's internal data structures from message
> to save some cycles on copying the data.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
> 
> 
> Diff: https://reviews.apache.org/r/63915/diff/3/
> 
> 
> Testing
> ---
> 
> make check
> 
> Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
> --enable-lock-free-event-queue 
> --enable-last-in-first-out-fixed-size-semaphore`
> `./mesos-tests.sh --benchmark 
> --gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
> On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 16.202206916secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (30065 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 30.509804836secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (57145 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 22.581999748secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (39629 ms)
> ```
> 
> On this chain of patches:
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 8.456615936secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (22659 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 15.09102354secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (43828 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 16.122729767secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (33182 ms)
> ```
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-12-04 Thread Michael Park 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/#review192746
---


Fix it, then Ship it!





src/master/master.hpp
Line 128 (original), 128 (patched)


Maybe we can consider making this `vector&&` as discussed in 
https://reviews.apache.org/r/63914/


- Michael Park


On Nov. 21, 2017, 9:53 a.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63915/
> ---
> 
> (Updated Nov. 21, 2017, 9:53 a.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Tasks can be moved into master's internal data structures from message
> to save some cycles on copying the data.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
> 
> 
> Diff: https://reviews.apache.org/r/63915/diff/3/
> 
> 
> Testing
> ---
> 
> make check
> 
> Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
> --enable-lock-free-event-queue 
> --enable-last-in-first-out-fixed-size-semaphore`
> `./mesos-tests.sh --benchmark 
> --gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
> On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 16.202206916secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (30065 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 30.509804836secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (57145 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 22.581999748secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (39629 ms)
> ```
> 
> On this chain of patches:
> ```
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 10 running tasks and 10 
> completed tasks in 8.456615936secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
>  (22659 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
> Starting reregistration for all agents
> Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
> tasks in 15.09102354secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
>  (43828 ms)
> [ RUN  ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
> Starting reregistration for all agents
> Reregistered 2 agents with a total of 10 running tasks and 0 
> completed tasks in 16.122729767secs
> [   OK ] 
> AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
>  (33182 ms)
> ```
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-11-21 Thread Dmitry Zhuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/
---

(Updated Nov. 21, 2017, 5:53 p.m.)


Review request for mesos, Benjamin Mahler and Michael Park.


Changes
---

Rebased. Included resource versions support.


Bugs: MESOS-6972
https://issues.apache.org/jira/browse/MESOS-6972


Repository: mesos


Description
---

Tasks can be moved into master's internal data structures from message
to save some cycles on copying the data.


Diffs (updated)
-

  src/master/master.hpp 2a2e830354db4a2191fb8321beb8174b80f7ba7d 
  src/master/master.cpp 7417b5d641fd4bb6d91cb0e6456c60201bbc8206 


Diff: https://reviews.apache.org/r/63915/diff/2/

Changes: https://reviews.apache.org/r/63915/diff/1-2/


Testing
---

make check

Ran benchmark with `--enable-optimize --enable-lock-free-run-queue 
--enable-lock-free-event-queue --enable-last-in-first-out-fixed-size-semaphore`
`./mesos-tests.sh --benchmark 
--gtest_filter=AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.*`
On 2608c0b8f62a9359d3d23e1724b6e91f316cfc76 (includes protobuf-3.5.0):
```
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
Starting reregistration for all agents
Reregistered 2000 agents with a total of 10 running tasks and 10 
completed tasks in 16.202206916secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
 (30065 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
Starting reregistration for all agents
Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
tasks in 30.509804836secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
 (57145 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
Starting reregistration for all agents
Reregistered 2 agents with a total of 10 running tasks and 0 completed 
tasks in 22.581999748secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
 (39629 ms)
```

On this chain of patches:
```
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
Starting reregistration for all agents
Reregistered 2000 agents with a total of 10 running tasks and 10 
completed tasks in 8.456615936secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/0
 (22659 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
Starting reregistration for all agents
Reregistered 2000 agents with a total of 20 running tasks and 0 completed 
tasks in 15.09102354secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/1
 (43828 ms)
[ RUN  ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
Starting reregistration for all agents
Reregistered 2 agents with a total of 10 running tasks and 0 completed 
tasks in 16.122729767secs
[   OK ] 
AgentFrameworkTaskCount/MasterFailover_BENCHMARK_Test.AgentReregistrationDelay/2
 (33182 ms)
```


Thanks,

Dmitry Zhuk

Re: Review Request 63915: Reduced tasks copying during agent reregistration.

2017-11-17 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63915/#review191369
---



FAIL: mesos-java failed to build.

Reviews applied: `['63628', '63629', '63630', '63631', '63632', '63633', 
'63634', '63635', '63636', '63637', '63638', '63641', '63761', '63913', 
'63914', '63915']`

Failed command: `cmake.exe --build . --target mesos-java`

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/63915

Relevant logs:

- 
[mesos-java-build-cmake-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/63915/logs/mesos-java-build-cmake-stdout.log):

```
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'process::network::internal::PollSocketImpl::accept::'
 to 'lambda::CallableOnce> (const T &)>' 
[C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder' to 'lambda::CallableOnce' [C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder (__cdecl *)(const 
std::shared_ptr &,const char 
*,::size_t),const std::shared_ptr 
&,const char *&,size_t &>' to 'lambda::CallableOnce 
(const T &)>' [C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder (__cdecl *)(const 
std::shared_ptr 
&,int_fd,off_t,::size_t),const 
std::shared_ptr &,int_fd &,off_t 
&,size_t &>' to 'lambda::CallableOnce (const T &)>' 
[C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder (__cdecl *)(const 
std::shared_ptr &,const char 
*,::size_t),std::shared_ptr,const 
char *&,size_t &>' to 'lambda::CallableOnce (const T 
&)>' [C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder (__cdecl *)(const 
std::shared_ptr 
&,int_fd,off_t,::size_t),std::shared_ptr,int_fd
 &,off_t &,size_t &>' to 'lambda::CallableOnce (const 
T &)>' [C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/dispatch.hpp(354): 
error C2665: 'lambda::CallableOnce::CallableOnce': none of the 3 overloads could convert all the argument 
types (compiling source file 
C:\DCOS\mesos\mesos\3rdparty\libprocess\src\reap.cpp) 
[C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder &,const Option 
&,process::Owned,::size_t,boost::shared_array,::size_t),const
 std::shared_ptr<_Ty> &,const Option &,process::Owned 
&,size_t &,boost::shared_array &,const std::_Ph<1> &>' to 
'lambda::CallableOnce' 
[C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder &,const Option 
&,process::Owned,::size_t,boost::shared_array,::size_t),std::shared_ptr<_Ty>,const
 Option &,process::Owned &,size_t 
&,boost::shared_array &,const std::_Ph<1> &>' to 
'lambda::CallableOnce' 
[C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder &,process::Owned,::size_t,::size_t),const 
std::shared_ptr<_Ty> &,process::Owned &,size_t &,const std::_Ph<1> 
&>' to 'lambda::CallableOnce' 
[C:\DCOS\mesos\3rdparty\libprocess\src\process.vcxproj]
  C:\DCOS\mesos\mesos\3rdparty\libprocess\include\process/future.hpp(412): 
error C2440: '': cannot convert from 
'std::_Binder