Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Benjamin Bannier]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/#review165846
---




src/master/master.cpp (lines 6943 - 6944)


It seems like if we recover resources here, there is nothing preventing the 
allocator from offering the exact same resources again immediately. In that 
case allocation progress could be blocked indefinitely as the allocator 
prepares an offer which the master immediately filters.

Another possibility might be to directly filter frameworks in the allocator 
loop like we already do for GPUs, 
https://github.com/apache/mesos/blob/7012ee28a5a91f4e6427bc134a99feb9536cb4e0/src/master/allocator/mesos/hierarchical.cpp#L1525-L1531.
 For that we'd need to store agent capabilities in 
`HierarchicalAllocatorProcess::Slave` to perform the filtering.

The downside of that approach is of course that all of this handling would 
be specifically patched into the hierarchical allocator making writing proper 
custom allocators even harder, but I cannot see another way to perform 
filtering in a deadlock-safe way. Do you have an idea?


- Benjamin Bannier


On Feb. 15, 2017, 7:44 a.m., Jay Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56530/
> ---
> 
> (Updated Feb. 15, 2017, 7:44 a.m.)
> 
> 
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
> Michael Park.
> 
> 
> Bugs: MESOS-6940
> https://issues.apache.org/jira/browse/MESOS-6940
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> In a mixed cluster, when an old agent (not MULTI_ROLE capable)
> registers with new master (MULTI_ROLE capable), it cannot correctly
> handle tasks from a MULTI_ROLE framework. Therefore, we should
> disallow offers from these agents being sent to MULTI_ROLE frameworks.
> 
> 
> Diffs
> -
> 
>   src/master/master.cpp dff320acd1a13995d51cc0c303edea0cdfaa336d 
>   src/tests/master_tests.cpp c8ef9d6bd9f4f8d631ac87d5e5f8dbe679a70fd1 
> 
> Diff: https://reviews.apache.org/r/56530/diff/
> 
> 
> Testing
> ---
> 
> ./bin/mesos-tests.sh 
> --gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
> --gtest_break_on_failure --gtest_repeat=-1
> 
> 
> Thanks,
> 
> Jay Guo
> 
>

Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Jay Guo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/
---

(Updated Feb. 15, 2017, 2:44 p.m.)


Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
Michael Park.


Bugs: MESOS-6940
https://issues.apache.org/jira/browse/MESOS-6940


Repository: mesos


Description
---

In a mixed cluster, when an old agent (not MULTI_ROLE capable)
registers with new master (MULTI_ROLE capable), it cannot correctly
handle tasks from a MULTI_ROLE framework. Therefore, we should
disallow offers from these agents being sent to MULTI_ROLE frameworks.


Diffs (updated)
-

  src/master/master.cpp dff320acd1a13995d51cc0c303edea0cdfaa336d 
  src/tests/master_tests.cpp c8ef9d6bd9f4f8d631ac87d5e5f8dbe679a70fd1 

Diff: https://reviews.apache.org/r/56530/diff/


Testing
---

./bin/mesos-tests.sh 
--gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
--gtest_break_on_failure --gtest_repeat=-1


Thanks,

Jay Guo

Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Mesos Reviewbot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/#review165485
---



Patch looks great!

Reviews applied: [56530]

Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; 
./support/docker-build.sh

- Mesos Reviewbot


On Feb. 14, 2017, 3:48 a.m., Jay Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56530/
> ---
> 
> (Updated Feb. 14, 2017, 3:48 a.m.)
> 
> 
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
> Michael Park.
> 
> 
> Bugs: MESOS-6940
> https://issues.apache.org/jira/browse/MESOS-6940
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> In a mixed cluster, when an old agent (not MULTI_ROLE capable)
> registers with new master (MULTI_ROLE capable), it cannot correctly
> handle tasks from a MULTI_ROLE framework. Therefore, we should
> disallow offers from these agents being sent to MULTI_ROLE frameworks.
> 
> 
> Diffs
> -
> 
>   src/master/master.cpp dff320acd1a13995d51cc0c303edea0cdfaa336d 
>   src/tests/master_tests.cpp c8ef9d6bd9f4f8d631ac87d5e5f8dbe679a70fd1 
> 
> Diff: https://reviews.apache.org/r/56530/diff/
> 
> 
> Testing
> ---
> 
> ./bin/mesos-tests.sh 
> --gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
> --gtest_break_on_failure --gtest_repeat=-1
> 
> 
> Thanks,
> 
> Jay Guo
> 
>

Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Jay Guo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/
---

(Updated Feb. 14, 2017, 11:48 a.m.)


Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
Michael Park.


Changes
---

rebase


Bugs: MESOS-6940
https://issues.apache.org/jira/browse/MESOS-6940


Repository: mesos


Description
---

In a mixed cluster, when an old agent (not MULTI_ROLE capable)
registers with new master (MULTI_ROLE capable), it cannot correctly
handle tasks from a MULTI_ROLE framework. Therefore, we should
disallow offers from these agents being sent to MULTI_ROLE frameworks.


Diffs (updated)
-

  src/master/master.cpp dff320acd1a13995d51cc0c303edea0cdfaa336d 
  src/tests/master_tests.cpp c8ef9d6bd9f4f8d631ac87d5e5f8dbe679a70fd1 

Diff: https://reviews.apache.org/r/56530/diff/


Testing
---

./bin/mesos-tests.sh 
--gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
--gtest_break_on_failure --gtest_repeat=-1


Thanks,

Jay Guo

Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Jay Guo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/
---

(Updated Feb. 14, 2017, 11:43 a.m.)


Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
Michael Park.


Bugs: MESOS-6940
https://issues.apache.org/jira/browse/MESOS-6940


Repository: mesos


Description
---

In a mixed cluster, when an old agent (not MULTI_ROLE capable)
registers with new master (MULTI_ROLE capable), it cannot correctly
handle tasks from a MULTI_ROLE framework. Therefore, we should
disallow offers from these agents being sent to MULTI_ROLE frameworks.


Diffs (updated)
-

  include/mesos/master/master.proto 472348ba20dfd746c713bbeb6a429919943e5935 
  include/mesos/v1/master/master.proto 9d2f38e0b12f5f2c8a233db8f32bb48918cf6df0 
  src/common/protobuf_utils.hpp 4f69483d9ab66b7b5f2213300e696c67a335 
  src/common/protobuf_utils.cpp 442184920521f372c535426f49155cb19980bf6d 
  src/master/http.cpp caa1fabb7a709d993ff711cc7803dd05000fa987 
  src/master/master.cpp dff320acd1a13995d51cc0c303edea0cdfaa336d 
  src/tests/containerizer/io_switchboard_tests.cpp 
9031815f5711cf940315ab3d8538aa099b356847 
  src/tests/master_tests.cpp c8ef9d6bd9f4f8d631ac87d5e5f8dbe679a70fd1 

Diff: https://reviews.apache.org/r/56530/diff/


Testing (updated)
---

./bin/mesos-tests.sh 
--gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
--gtest_break_on_failure --gtest_repeat=-1


Thanks,

Jay Guo

Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Mesos Reviewbot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/#review165070
---



Patch looks great!

Reviews applied: [56530]

Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; 
./support/docker-build.sh

- Mesos Reviewbot


On Feb. 10, 2017, 6:10 a.m., Jay Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56530/
> ---
> 
> (Updated Feb. 10, 2017, 6:10 a.m.)
> 
> 
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
> Michael Park.
> 
> 
> Bugs: MESOS-6940
> https://issues.apache.org/jira/browse/MESOS-6940
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> In a mixed cluster, when an old agent (not MULTI_ROLE capable)
> registers with new master (MULTI_ROLE capable), it cannot correctly
> handle tasks from a MULTI_ROLE framework. Therefore, we should
> disallow offers from these agents being sent to MULTI_ROLE frameworks.
> 
> 
> Diffs
> -
> 
>   src/master/master.cpp 620919ecfe85367b5c1281afc5216cc20e5e2e3c 
>   src/tests/master_tests.cpp 3b4123b49ee32c902a5d2a01fcc7026da21fdd18 
> 
> Diff: https://reviews.apache.org/r/56530/diff/
> 
> 
> Testing
> ---
> 
> WIP
> 
> ./bin/mesos-tests.sh 
> --gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
> --gtest_break_on_failure --gtest_repeat=-1
> 
> 
> Thanks,
> 
> Jay Guo
> 
>

Re: Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Jay Guo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/#review165063
---




src/tests/master_tests.cpp (lines 6346 - 6347)


Need to figure out how to reliably intercept *all* `RegisterSlaveMessage` 
to avoid flakiness. Slave may retry registration with a very small backoff 
delay, which leads to a second (potentially more) `RegisterSlaveMessage` leaked 
to master without being spoofed.


- Jay Guo


On Feb. 10, 2017, 2:10 p.m., Jay Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56530/
> ---
> 
> (Updated Feb. 10, 2017, 2:10 p.m.)
> 
> 
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
> Michael Park.
> 
> 
> Bugs: MESOS-6940
> https://issues.apache.org/jira/browse/MESOS-6940
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> In a mixed cluster, when an old agent (not MULTI_ROLE capable)
> registers with new master (MULTI_ROLE capable), it cannot correctly
> handle tasks from a MULTI_ROLE framework. Therefore, we should
> disallow offers from these agents being sent to MULTI_ROLE frameworks.
> 
> 
> Diffs
> -
> 
>   src/master/master.cpp 620919ecfe85367b5c1281afc5216cc20e5e2e3c 
>   src/tests/master_tests.cpp 3b4123b49ee32c902a5d2a01fcc7026da21fdd18 
> 
> Diff: https://reviews.apache.org/r/56530/diff/
> 
> 
> Testing
> ---
> 
> WIP
> 
> ./bin/mesos-tests.sh 
> --gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
> --gtest_break_on_failure --gtest_repeat=-1
> 
> 
> Thanks,
> 
> Jay Guo
> 
>

Review Request 56530: Prevent offers for old agents being sent to MULTI_ROLE frameworks.

[Jay Guo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56530/
---

Review request for mesos, Benjamin Bannier, Benjamin Mahler, Guangya Liu, and 
Michael Park.


Bugs: MESOS-6940
https://issues.apache.org/jira/browse/MESOS-6940


Repository: mesos


Description
---

In a mixed cluster, when an old agent (not MULTI_ROLE capable)
registers with new master (MULTI_ROLE capable), it cannot correctly
handle tasks from a MULTI_ROLE framework. Therefore, we should
disallow offers from these agents being sent to MULTI_ROLE frameworks.


Diffs
-

  src/master/master.cpp 620919ecfe85367b5c1281afc5216cc20e5e2e3c 
  src/tests/master_tests.cpp 3b4123b49ee32c902a5d2a01fcc7026da21fdd18 

Diff: https://reviews.apache.org/r/56530/diff/


Testing
---

WIP

./bin/mesos-tests.sh 
--gtest_filter="MasterTest.MultiRoleFrameworkDoesNotReceiveOfferFromOldAgent" 
--gtest_break_on_failure --gtest_repeat=-1


Thanks,

Jay Guo