Re: Review Request 63914: Changed agent reregistration to work with message directly.

[Dmitry Zhuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/
---

(Updated Dec. 6, 2017, 4:24 p.m.)


Review request for mesos, Benjamin Mahler and Michael Park.


Changes
---

Rebased (non-trivial due to changes in https://reviews.apache.org/r/64011/ ).


Bugs: MESOS-6972
https://issues.apache.org/jira/browse/MESOS-6972


Repository: mesos


Description
---

`reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
of using protobuf arena, and allows passing message through dispatch
chain without making any copies.
Conversion of repeated message fields to `std::vector`s is performed
only when needed.


Diffs (updated)
-

  src/master/master.hpp d42acae5a7d31dccf7073b57c3f2ddcfadea9e55 
  src/master/master.cpp 16cdde73b90b2e1514c6196850b8df75a3b6ff28 
  src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
  src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
  src/tests/master_validation_tests.cpp 
6398f16a3e1c0bbf0b188a5993b1d4ad9e105f41 


Diff: https://reviews.apache.org/r/63914/diff/5/

Changes: https://reviews.apache.org/r/63914/diff/4-5/


Testing
---

make check


Thanks,

Dmitry Zhuk

Re: Review Request 63914: Changed agent reregistration to work with message directly.

[Dmitry Zhuk]

> On Dec. 4, 2017, 9:18 p.m., Benjamin Mahler wrote:
> > src/master/master.cpp
> > Lines 6492-6499 (patched)
> > 
> >
> > Hm.. I'm curious if you have any thoughts on avoiding copying it out of 
> > the message. We can look at that more closely in a different patch though :)

Simplest solution is to add
```
template 
std::vector convert(google::protobuf::RepeatedPtrField&& items)
```
and use it like this:
```
vector tasks = 
google::protobuf::convert(std::move(*message.mutable_tasks()));
```
but the problem is that it becomes difficult to track which parts of the 
message have been moved from. And I suppose `clang-tidy` will not be able to 
detect "use after move" in such case. So I haven't published this patch.
I was thinking about decomposing a message with moves and invoking continuation 
method with all individual message fields as parameters to make sure message 
goes out of scope, but I haven't tried this yet.

In many cases it's possible to avoid using `convert`, and use 
`RepeatedPtrField` instead. I did this for `tasks`, but the problem is still 
the same - difficult to track moved message parts.


- Dmitry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/#review192747
---


On Nov. 21, 2017, 5:53 p.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63914/
> ---
> 
> (Updated Nov. 21, 2017, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> `reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
> of using protobuf arena, and allows passing message through dispatch
> chain without making any copies.
> Conversion of repeated message fields to `std::vector`s is performed
> only when needed.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
>   src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
>   src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
>   src/tests/master_validation_tests.cpp 
> 0e1c8b490ebe10bc50b8b6b530fa85128b967584 
> 
> 
> Diff: https://reviews.apache.org/r/63914/diff/4/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63914: Changed agent reregistration to work with message directly.

[Dmitry Zhuk]

> On Dec. 4, 2017, 5:57 p.m., Michael Park wrote:
> > src/master/master.cpp
> > Line 6433 (original), 6394 (patched)
> > 
> >
> > I think these `message` to `message_` changes should either take the 
> > type of the mesage (e.g., `shutdown`), or simply a `s/message/outgoing/`. 
> > Having unrelated `message` and `message_` in the same scope will likely 
> > cause confusion.

Maybe it's better change `ReregisterSlaveMessage` name to `request` and keep 
`message` in other places?
I tried using names like `checkpointResourcesMessage` as bmahler suggested 
below, but lines became too long and wrapping was required. Names like 
`shutdown` should work, but this will be inconsistent with other code in this 
file.


- Dmitry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/#review192721
---


On Nov. 21, 2017, 5:53 p.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63914/
> ---
> 
> (Updated Nov. 21, 2017, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> `reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
> of using protobuf arena, and allows passing message through dispatch
> chain without making any copies.
> Conversion of repeated message fields to `std::vector`s is performed
> only when needed.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
>   src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
>   src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
>   src/tests/master_validation_tests.cpp 
> 0e1c8b490ebe10bc50b8b6b530fa85128b967584 
> 
> 
> Diff: https://reviews.apache.org/r/63914/diff/4/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63914: Changed agent reregistration to work with message directly.

[Benjamin Mahler]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/#review192747
---


Ship it!




Exciting to see this stuff come together! Looks good, thanks Dmitry!

I left some minor comments, but feel free to leave the patch as is, myself 
and/or mpark could make some adjustments prior to committing if that's easier 
for you.


src/master/master.cpp
Lines 6492-6499 (patched)


Hm.. I'm curious if you have any thoughts on avoiding copying it out of the 
message. We can look at that more closely in a different patch though :)



src/master/master.cpp
Line 6589 (original), 6560 (patched)


Per mpark's suggestion, not your fault, but something like 
`checkpointResourcesMessage` here would be more readable.



src/master/master.cpp
Lines 6672-6682 (patched)


Ditto here, curious if you had any thoughts on eliminating these copies, 
but we can look at that later :)



src/tests/master_validation_tests.cpp
Line 4213 (original), 4213 (patched)


Not yours, but this should probably be ReregisterSlaveValidationTest.


- Benjamin Mahler


On Nov. 21, 2017, 5:53 p.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63914/
> ---
> 
> (Updated Nov. 21, 2017, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> `reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
> of using protobuf arena, and allows passing message through dispatch
> chain without making any copies.
> Conversion of repeated message fields to `std::vector`s is performed
> only when needed.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
>   src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
>   src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
>   src/tests/master_validation_tests.cpp 
> 0e1c8b490ebe10bc50b8b6b530fa85128b967584 
> 
> 
> Diff: https://reviews.apache.org/r/63914/diff/4/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63914: Changed agent reregistration to work with message directly.

[Michael Park]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/#review192721
---


Fix it, then Ship it!





src/master/master.cpp
Line 6433 (original), 6394 (patched)


I think these `message` to `message_` changes should either take the type 
of the mesage (e.g., `shutdown`), or simply a `s/message/outgoing/`. Having 
unrelated `message` and `message_` in the same scope will likely cause 
confusion.



src/master/master.cpp
Line 6823 (original), 6779 (patched)


I think the intention is that we move this, right? I think the `Slave` ctor 
was updated on the next patch to take `vector`, rather than `vector&&`. Maybe 
we should make it `vector&&` such that we don't run into this?

Same with `agentCapabilities` and other related data


- Michael Park


On Nov. 21, 2017, 9:53 a.m., Dmitry Zhuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63914/
> ---
> 
> (Updated Nov. 21, 2017, 9:53 a.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Michael Park.
> 
> 
> Bugs: MESOS-6972
> https://issues.apache.org/jira/browse/MESOS-6972
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> `reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
> of using protobuf arena, and allows passing message through dispatch
> chain without making any copies.
> Conversion of repeated message fields to `std::vector`s is performed
> only when needed.
> 
> 
> Diffs
> -
> 
>   src/master/master.hpp 0f8a2ac72c3484f911853c2994fc71a488d66d96 
>   src/master/master.cpp fadc78b2ca5d46b8cc12a794b428753aa79ac095 
>   src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
>   src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
>   src/tests/master_validation_tests.cpp 
> 0e1c8b490ebe10bc50b8b6b530fa85128b967584 
> 
> 
> Diff: https://reviews.apache.org/r/63914/diff/4/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Dmitry Zhuk
> 
>

Re: Review Request 63914: Changed agent reregistration to work with message directly.

[Dmitry Zhuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/
---

(Updated Nov. 21, 2017, 5:53 p.m.)


Review request for mesos, Benjamin Mahler and Michael Park.


Changes
---

Rebased. Included resources versions support.


Bugs: MESOS-6972
https://issues.apache.org/jira/browse/MESOS-6972


Repository: mesos


Description
---

`reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
of using protobuf arena, and allows passing message through dispatch
chain without making any copies.
Conversion of repeated message fields to `std::vector`s is performed
only when needed.


Diffs (updated)
-

  src/master/master.hpp 2a2e830354db4a2191fb8321beb8174b80f7ba7d 
  src/master/master.cpp 7417b5d641fd4bb6d91cb0e6456c60201bbc8206 
  src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
  src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
  src/tests/master_validation_tests.cpp 
0e1c8b490ebe10bc50b8b6b530fa85128b967584 


Diff: https://reviews.apache.org/r/63914/diff/2/

Changes: https://reviews.apache.org/r/63914/diff/1-2/


Testing
---

make check


Thanks,

Dmitry Zhuk

Review Request 63914: Changed agent reregistration to work with message directly.

[Dmitry Zhuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63914/
---

Review request for mesos, Benjamin Mahler and Michael Park.


Bugs: MESOS-6972
https://issues.apache.org/jira/browse/MESOS-6972


Repository: mesos


Description
---

`reregisterSlave` now accepts `ReregisterSlaveMessage&&`, which opts-out
of using protobuf arena, and allows passing message through dispatch
chain without making any copies.
Conversion of repeated message fields to `std::vector`s is performed
only when needed.


Diffs
-

  src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3 
  src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f 
  src/master/validation.hpp ac54062ea09f97ad96bd17deb106ea89a57f394a 
  src/master/validation.cpp 8b5848bfd8c069f34a92a9a68597955c6e0c2ee2 
  src/tests/master_validation_tests.cpp 
0e1c8b490ebe10bc50b8b6b530fa85128b967584 


Diff: https://reviews.apache.org/r/63914/diff/1/


Testing
---

make check


Thanks,

Dmitry Zhuk