Re: Review Request 40935: Fixed flakiness in MasterMaintenanceTest.InverseOffersFilters.

2015-12-04 Thread Joris Van Remoortere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40935/#review109001
---

Ship it!


Addressed Joseph's comments, verified with Joseph & Neil before committing.

- Joris Van Remoortere


On Dec. 4, 2015, 12:17 a.m., Neil Conway wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40935/
> ---
> 
> (Updated Dec. 4, 2015, 12:17 a.m.)
> 
> 
> Review request for mesos, Joris Van Remoortere and Joseph Wu.
> 
> 
> Bugs: MESOS-4059
> https://issues.apache.org/jira/browse/MESOS-4059
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> There were two problems:
> 
> (1) After launching two tasks, we assumed that we would see TASK_RUNNING 
> updates
> for the tasks in the same order they were launched. This is not 
> guaranteed,
> so adjust the test to handle TASK_RUNNING updates in the order they are
> received.
> 
> (2) The test used this pattern:
> 
> Mesos m;
> Call c;
> 
> m.send(c);
> Clock::settle();
> // Trigger a new batch allocation that reflects the call
> Clock::advance();
> 
> However, this is actually unsafe (see MESOS-3760): the send() call might 
> not
> have reached the master by the time `Clock::settle()` happens. This was
> fixed by blocking using `FUTURE_DISPATCH` on the downstream logic in the
> allocator that is invoked to handle the delivered event.
> 
> 
> Diffs
> -
> 
>   src/tests/master_maintenance_tests.cpp 
> 00900561a1b8dd03a7a2f3d60a036b4beb920aa1 
> 
> Diff: https://reviews.apache.org/r/40935/diff/
> 
> 
> Testing
> ---
> 
> ./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
> --gtest_repeat=2000 # on OSX
> ./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
> --gtest_repeat=100  # on Ubuntu Wily (slow VM)
> 
> 
> Thanks,
> 
> Neil Conway
> 
>

Re: Review Request 40935: Fixed flakiness in MasterMaintenanceTest.InverseOffersFilters.

2015-12-03 Thread Mesos ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40935/#review108918
---


Bad patch!

Reviews applied: [40935]

Failed command: export OS=ubuntu:14.04;export CONFIGURATION="--verbose";export 
COMPILER=gcc; ./support/docker_build.sh

Error:
 + : ubuntu:14.04
+ : gcc
+ : --verbose
+++ dirname ./support/docker_build.sh
++ cd ./support/..
++ pwd
+ MESOS_DIRECTORY=/home/jenkins/jenkins-slave/workspace/mesos-reviewbot
+ cd /home/jenkins/jenkins-slave/workspace/mesos-reviewbot
+ DOCKERFILE=Dockerfile
+ rm -f Dockerfile
+ case $OS in
+ append_dockerfile 'FROM ubuntu:14.04'
+ echo FROM ubuntu:14.04
+ append_dockerfile 'RUN rm -rf /var/lib/apt/lists/*'
+ echo RUN rm -rf 
/var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-amd64_Packages 
/var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-i386_Packages 
/var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release 
/var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release.gpg 
/var/lib/apt/lists/lock 
/var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-amd64_Packages
 
/var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-i386_Packages
 /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release 
/var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release.gpg 
/var/lib/apt/lists/partial 
/var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-amd64_Packages
 
/var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-i386_Packages
 
/var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_i18n_Translation-en
 /va
 
r/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_source_Sources
 
/var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_Release
 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_InRelease 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_i18n_Translation-en
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_source_Sources
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_i18n_Translation-en
 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiver
 se_source_Sources 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_i18n_Translation-en
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_source_Sources
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_i18n_Translation-en
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_source_Sources
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_InRelease
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-amd64_Packages
 /var/lib/apt/lists/u
 s.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-i386_Packages 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_i18n_Translation-en
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_source_Sources
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-amd64_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-i386_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_i18n_Translation-en
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_source_Sources
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-amd64_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-i386_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_i18n_Translation-en
 /var/lib/apt/lists
 /us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_source_Sources

Review Request 40935: Fixed flakiness in MasterMaintenanceTest.InverseOffersFilters.

2015-12-03 Thread Neil Conway 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40935/
---

Review request for mesos, Joris Van Remoortere and Joseph Wu.


Bugs: MESOS-4059
https://issues.apache.org/jira/browse/MESOS-4059


Repository: mesos


Description
---

There were two problems:

(1) After launching two tasks, we assumed that we would see TASK_RUNNING updates
for the tasks in the same order they were launched. This is not guaranteed,
so adjust the test to handle TASK_RUNNING updates in the order they are
received.

(2) The test used this pattern:

Mesos m;
Call c;

m.send(c);
Clock::settle();
// Trigger a new batch allocation that reflects the call
Clock::advance();

However, this is actually unsafe (see MESOS-3760): the send() call might not
have reached the master by the time `Clock::settle()` happens. This was
fixed by blocking using `FUTURE_DISPATCH` on the downstream logic in the
allocator that is invoked to handle the delivered event.


Diffs
-

  src/tests/master_maintenance_tests.cpp 
00900561a1b8dd03a7a2f3d60a036b4beb920aa1 

Diff: https://reviews.apache.org/r/40935/diff/


Testing
---

./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
--gtest_repeat=2000 # on OSX
./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
--gtest_repeat=100  # on Ubuntu Wily (slow VM)


Thanks,

Neil Conway

Re: Review Request 40935: Fixed flakiness in MasterMaintenanceTest.InverseOffersFilters.

2015-12-03 Thread Neil Conway 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40935/
---

(Updated Dec. 4, 2015, 12:17 a.m.)


Review request for mesos, Joris Van Remoortere and Joseph Wu.


Changes
---

Fix another instance of the `send(); Clock::settle();` pattern in the same test.


Bugs: MESOS-4059
https://issues.apache.org/jira/browse/MESOS-4059


Repository: mesos


Description
---

There were two problems:

(1) After launching two tasks, we assumed that we would see TASK_RUNNING updates
for the tasks in the same order they were launched. This is not guaranteed,
so adjust the test to handle TASK_RUNNING updates in the order they are
received.

(2) The test used this pattern:

Mesos m;
Call c;

m.send(c);
Clock::settle();
// Trigger a new batch allocation that reflects the call
Clock::advance();

However, this is actually unsafe (see MESOS-3760): the send() call might not
have reached the master by the time `Clock::settle()` happens. This was
fixed by blocking using `FUTURE_DISPATCH` on the downstream logic in the
allocator that is invoked to handle the delivered event.


Diffs (updated)
-

  src/tests/master_maintenance_tests.cpp 
00900561a1b8dd03a7a2f3d60a036b4beb920aa1 

Diff: https://reviews.apache.org/r/40935/diff/


Testing
---

./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
--gtest_repeat=2000 # on OSX
./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
--gtest_repeat=100  # on Ubuntu Wily (slow VM)


Thanks,

Neil Conway

Re: Review Request 40935: Fixed flakiness in MasterMaintenanceTest.InverseOffersFilters.

2015-12-03 Thread Neil Conway 


> On Dec. 4, 2015, 12:24 a.m., Mesos ReviewBot wrote:
> > Bad patch!
> > 
> > Reviews applied: [40935]
> > 
> > Failed command: export OS=ubuntu:14.04;export 
> > CONFIGURATION="--verbose";export COMPILER=gcc; ./support/docker_build.sh
> > 
> > Error:
> >  + : ubuntu:14.04
> > + : gcc
> > + : --verbose
> > +++ dirname ./support/docker_build.sh
> > ++ cd ./support/..
> > ++ pwd
> > + MESOS_DIRECTORY=/home/jenkins/jenkins-slave/workspace/mesos-reviewbot
> > + cd /home/jenkins/jenkins-slave/workspace/mesos-reviewbot
> > + DOCKERFILE=Dockerfile
> > + rm -f Dockerfile
> > + case $OS in
> > + append_dockerfile 'FROM ubuntu:14.04'
> > + echo FROM ubuntu:14.04
> > + append_dockerfile 'RUN rm -rf /var/lib/apt/lists/*'
> > + echo RUN rm -rf 
> > /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-amd64_Packages
> >  
> > /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-i386_Packages
> >  /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release 
> > /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release.gpg 
> > /var/lib/apt/lists/lock 
> > /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-amd64_Packages
> >  
> > /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-i386_Packages
> >  /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release 
> > /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release.gpg
> >  /var/lib/apt/lists/partial 
> > /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-amd64_Packages
> >  
> > /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-i386_Packages
> >  
> > /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_i18n_Translation-en
  
/var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_source_Sources
 
/var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_Release
 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_InRelease 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_i18n_Translation-en
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_source_Sources
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_i18n_Translation-en
 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_mult
 iverse_source_Sources 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_i18n_Translation-en
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_source_Sources
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-i386_Packages
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_i18n_Translation-en
 
/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_source_Sources
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_InRelease
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-amd64_Packages
 /var/lib/apt/lis
 
ts/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-i386_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_i18n_Translation-en
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_source_Sources
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-amd64_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-i386_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_i18n_Translation-en
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_source_Sources
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-amd64_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-i386_Packages
 
/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_i18n_Translation-en
 /var/lib/apt/l
 
ists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_source_Sources

Re: Review Request 40935: Fixed flakiness in MasterMaintenanceTest.InverseOffersFilters.

2015-12-03 Thread Joseph Wu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40935/#review108911
---

Ship it!


Tested on Centos7.

There still appears to be some flakiness (in the first inverse offer filter) 
that requires further investigation.  But these changes make the test 
significantly more stable.
I vote for committing this patch, but keeping the JIRA open until we nail down 
the flakiness for good.


src/tests/master_maintenance_tests.cpp (lines 1485 - 1488)


It might be valuable to expect an update for each task.

Maybe check:
`updateStatus.task_id() == taskInfo1.task_id() || updateStatus.task_id() == 
taskInfo2.task_id()`
And between the two acknowledges, you could do:
`updateStatus.task_id() != event.get().update().status().taskid()`


Note: As far as I know, no other HTTP scheduler library test launches two 
tasks simultaneously.


- Joseph Wu


On Dec. 3, 2015, 4:17 p.m., Neil Conway wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40935/
> ---
> 
> (Updated Dec. 3, 2015, 4:17 p.m.)
> 
> 
> Review request for mesos, Joris Van Remoortere and Joseph Wu.
> 
> 
> Bugs: MESOS-4059
> https://issues.apache.org/jira/browse/MESOS-4059
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> There were two problems:
> 
> (1) After launching two tasks, we assumed that we would see TASK_RUNNING 
> updates
> for the tasks in the same order they were launched. This is not 
> guaranteed,
> so adjust the test to handle TASK_RUNNING updates in the order they are
> received.
> 
> (2) The test used this pattern:
> 
> Mesos m;
> Call c;
> 
> m.send(c);
> Clock::settle();
> // Trigger a new batch allocation that reflects the call
> Clock::advance();
> 
> However, this is actually unsafe (see MESOS-3760): the send() call might 
> not
> have reached the master by the time `Clock::settle()` happens. This was
> fixed by blocking using `FUTURE_DISPATCH` on the downstream logic in the
> allocator that is invoked to handle the delivered event.
> 
> 
> Diffs
> -
> 
>   src/tests/master_maintenance_tests.cpp 
> 00900561a1b8dd03a7a2f3d60a036b4beb920aa1 
> 
> Diff: https://reviews.apache.org/r/40935/diff/
> 
> 
> Testing
> ---
> 
> ./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
> --gtest_repeat=2000 # on OSX
> ./src/mesos-tests --gtest_filter="MasterMaintenanceTest.InverseOffersFilters" 
> --gtest_repeat=100  # on Ubuntu Wily (slow VM)
> 
> 
> Thanks,
> 
> Neil Conway
> 
>