Re: Review Request 68970: Fixed JSON object type error when using `support/apply-reviews.py`.

2018-10-10 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68970/#review209400
---



PASS: Mesos patch 68970 was successfully built and tested.

Reviews applied: `['68970']`

All the build artifacts available at: 
http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2444/mesos-review-68970

- Mesos Reviewbot Windows


On Oct. 10, 2018, 5:21 a.m., Armand Grillet wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68970/
> ---
> 
> (Updated Oct. 10, 2018, 5:21 a.m.)
> 
> 
> Review request for mesos, Till Toenshoff and Vinod Kone.
> 
> 
> Bugs: MESOS-9253
> https://issues.apache.org/jira/browse/MESOS-9253
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> The error `TypeError: the JSON object must be str, not 'bytes'` was
> visible when using `support/apply-reviews.py` in our CI. This issue
> seems to only happen in old versions of Python 3 that are not supported
> but the fix works with any version of Python 3 and allows us to not
> change the Python interpreter used by reviewbot.
> 
> 
> Diffs
> -
> 
>   support/apply-reviews.py 92ad85945fb97e6ae3e21b3b9ab53c885158417e 
> 
> 
> Diff: https://reviews.apache.org/r/68970/diff/1/
> 
> 
> Testing
> ---
> 
> Added the line `print(type(json_str))` in `def url_to_json(url):` to verify 
> that the type was now a `str` and not a `http.client.HTTPResponse`.
> 
> 
> Thanks,
> 
> Armand Grillet
> 
>

Re: Review Request 68970: Fixed JSON object type error when using `support/apply-reviews.py`.

2018-10-10 Thread Till Toenshoff via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68970/#review209406
---


Ship it!




Ship It!

- Till Toenshoff


On Oct. 10, 2018, 5:21 a.m., Armand Grillet wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68970/
> ---
> 
> (Updated Oct. 10, 2018, 5:21 a.m.)
> 
> 
> Review request for mesos, Till Toenshoff and Vinod Kone.
> 
> 
> Bugs: MESOS-9253
> https://issues.apache.org/jira/browse/MESOS-9253
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> The error `TypeError: the JSON object must be str, not 'bytes'` was
> visible when using `support/apply-reviews.py` in our CI. This issue
> seems to only happen in old versions of Python 3 that are not supported
> but the fix works with any version of Python 3 and allows us to not
> change the Python interpreter used by reviewbot.
> 
> 
> Diffs
> -
> 
>   support/apply-reviews.py 92ad85945fb97e6ae3e21b3b9ab53c885158417e 
> 
> 
> Diff: https://reviews.apache.org/r/68970/diff/1/
> 
> 
> Testing
> ---
> 
> Added the line `print(type(json_str))` in `def url_to_json(url):` to verify 
> that the type was now a `str` and not a `http.client.HTTPResponse`.
> 
> 
> Thanks,
> 
> Armand Grillet
> 
>

Re: Review Request 68971: Added pylint init-hook to add Mesos library to `PYTHONPATH`.

2018-10-10 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68971/#review209403
---



FAIL: Some of the unit tests failed. Please check the relevant logs.

Reviews applied: `['68971']`

Failed command: `Start-MesosCITesting`

All the build artifacts available at: 
http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2445/mesos-review-68971

Relevant logs:

- 
[mesos-tests.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2445/mesos-review-68971/logs/mesos-tests.log):

```
I1010 12:15:27.519279 28376 executor.cpp:918] Sending SIGTERM to process tree 
at pid 28work 9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3- (latest state: 
TASK_KILLED, status update state: TASK_KILLED)
I1010 12:15:27.517290 33352 slave.cpp:6640] Shutting down executor 
'aab7c622-0137-4216-987a-fa30153e3a82' of framework 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3- at executor(1)@192.10.1.5:61645
I1010 12:15:27.519279 33352 slave.cpp:909] Agent terminating
W1010 12:15:27.519279 33352 slave.cpp:3917] Ignoring shutdown framework 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3- because it is terminating
I1010 12:15:27.519279 30124 master.cpp:11030] Removing task 
aab7c622-0137-4216-987a-fa30153e3a82 with resources cpus(allocated: *):4; 
mem(allocated: *):2048; disk(allocated: *):1024; ports(allocated: 
*):[31000-32000] of framework 9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3- on 
agent 9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-S0 at slave(461)@192.10.1.5:59867 
(windows-02.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net)
I1010 12:15:27.521296 21816 master.cpp:1251] Agent 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-S0 at slave(461)@192.10.1.5:59867 
(windows-02.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net) disconnected
I1010 12:15:27.521296 21816 master.cpp:3267] Disconnecting agent 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-S0 at slave(461)@192.10.1.5:59867 
(windows-02.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net)
I1010 12:15:27.522272 21816 master.cpp:3286] Deactivating agent 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-S0 at slave(461)@192.10.1.5:59867 
(windows-02.aa0q4n2kgcyefckmv0xukjvy4f.xx.internal.cloudapp.net)
I1010 12:15:27.522272 32756 hierarchical.cpp:359] Removed framework 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-
I1010 12:15:27.522272 32756 hierarchical.cpp:803] Agent 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-S0 deactivated
I1010 12:15:27.523303 21816 containerizer.cpp:2455] Destroying container 
763fffdf-18cc-40e8-a004-f0237832f4d2 in RUNNING state
I1010 12:15:27.523303 21816 containerizer.cpp:3122] Transitioning the state of 
container 763fffdf-18cc-40e8-a004-f0237832f4d2 from RUNNING to DESTROYING
I1010 12:15:27.524273 21816 l[   OK ] 
IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (588 ms)
[--] 1 test from IsolationFlag/MemoryIsolatorTest (606 ms total)

[--] Global test environment tear-down
[==] 1051 tests from 103 test cases ran. (486030 ms total)
[  PASSED  ] 1050 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DockerFetcherPluginTest.INTERNET_CURL_FetchBlob

 1 FAILED TEST
  YOU HAVE 232 DISABLED TESTS

auncher.cpp:166] Asked to destroy container 763fffdf-18cc-40e8-a004-f0237832f4d2
I1010 12:15:27.532269 30568 containerizer.cpp:2961] Container 
763fffdf-18cc-40e8-a004-f0237832f4d2 has exited
I1010 12:15:27.561275 28116 master.cpp:1093] Master terminating
I1010 12:15:27.562273 32964 hierarchical.cpp:645] Removed agent 
9ff01ed5-975a-49dd-a0e3-0c609b1ca8a3-S0
I1010 12:15:28.006341 33012 process.cpp:926] Stopped the socket accept loop
```

- Mesos Reviewbot Windows


On Oct. 10, 2018, 11:11 a.m., Armand Grillet wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68971/
> ---
> 
> (Updated Oct. 10, 2018, 11:11 a.m.)
> 
> 
> Review request for mesos and Kevin Klues.
> 
> 
> Bugs: MESOS-6551
> https://issues.apache.org/jira/browse/MESOS-6551
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Without this change, adding `from mesos.exceptions import
> MesosException` to a file in the new CLI returns `E0401:
> Unable to import 'mesos.exceptions' (import-error)`.
> 
> This is because the Mesos library is a local module that we need to add
> to the `PYTHONPATH`. This is already done when using the virtual
> environment of the new CLI and we now also do it when linting.
> 
> 
> Diffs
> -
> 
>   support/pylint.config af25dd90cb2d467c688ea4b060dc4640040a068b 
> 
> 
> Diff: https://reviews.apache.org/r/68971/diff/1/
> 
> 
> Testing
> ---
> 
> Updated the agent plugin to do `from mesos.exceptions import MesosException` 
> and use `MesosException`. Commited to see the git hook running and saw the 
> error `E0401: Unable to import

Review Request 68976: Added `post()` function to new CLI for interacting with the HTTP API.

2018-10-10 Thread Armand Grillet 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68976/
---

Review request for mesos and Kevin Klues.


Bugs: MESOS-6551
https://issues.apache.org/jira/browse/MESOS-6551


Repository: mesos


Description
---

Added `post()` function to new CLI for interacting with the HTTP API.


Diffs
-

  src/python/lib/mesos/http.py cd1587797db7d75c6b839851f0f3e5671269307c 


Diff: https://reviews.apache.org/r/68976/diff/1/


Testing
---


Thanks,

Armand Grillet

Review Request 68977: Added Record-IO encoder and decoder to new CLI.

2018-10-10 Thread Armand Grillet 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68977/
---

Review request for mesos and Kevin Klues.


Bugs: MESOS-6551
https://issues.apache.org/jira/browse/MESOS-6551


Repository: mesos


Description
---

Added Record-IO encoder and decoder to new CLI.


Diffs
-

  src/python/cli_new/lib/cli/recordio.py PRE-CREATION 


Diff: https://reviews.apache.org/r/68977/diff/1/


Testing
---


Thanks,

Armand Grillet

Review Request 68978: Added TaskIO object to new CLI for `task exec` and `task attach`.

2018-10-10 Thread Armand Grillet 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68978/
---

Review request for mesos and Kevin Klues.


Bugs: MESOS-6551
https://issues.apache.org/jira/browse/MESOS-6551


Repository: mesos


Description
---

Added TaskIO object to new CLI for `task exec` and `task attach`.


Diffs
-

  src/python/cli_new/lib/cli/mesos.py 7cf84bcf1d327bc9c63934e371692cef989ad3aa 


Diff: https://reviews.apache.org/r/68978/diff/1/


Testing
---


Thanks,

Armand Grillet

Re: Review Request 68978: Added TaskIO object to new CLI for `task exec` and `task attach`.

2018-10-10 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68978/#review209414
---



PASS: Mesos patch 68978 was successfully built and tested.

Reviews applied: `['68971', '68976', '68977', '68978']`

All the build artifacts available at: 
http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2446/mesos-review-68978

- Mesos Reviewbot Windows


On Oct. 10, 2018, 2:45 p.m., Armand Grillet wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68978/
> ---
> 
> (Updated Oct. 10, 2018, 2:45 p.m.)
> 
> 
> Review request for mesos and Kevin Klues.
> 
> 
> Bugs: MESOS-6551
> https://issues.apache.org/jira/browse/MESOS-6551
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Added TaskIO object to new CLI for `task exec` and `task attach`.
> 
> 
> Diffs
> -
> 
>   src/python/cli_new/lib/cli/mesos.py 
> 7cf84bcf1d327bc9c63934e371692cef989ad3aa 
> 
> 
> Diff: https://reviews.apache.org/r/68978/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Armand Grillet
> 
>

Re: Review Request 68706: Added master failover reregistration progress metrics.

2018-10-10 Thread Xudong Ni via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68706/
---

(Updated Oct. 10, 2018, 5:22 p.m.)


Review request for mesos, Benjamin Mahler, James Peach, and Jiang Yan Xu.


Bugs: MESOS-9178
https://issues.apache.org/jira/browse/MESOS-9178


Repository: mesos


Description
---

During the master failover, the time that the master elected is
considered as the start of failover. In the progress of
reregistration, the percentile represents the time when such
percentile of agents finished registration again; The percentile of
these data as in this metrics can represent overall reregistration
progress; In case of degradation towards to the end of reregistration,
the high percentile can reflect it; In the case there are unreachable
agents in the failover, if certain percentile recovery couldn't be
reached, the intiail value of that percentile will not be updated.


Diffs
-

  docs/monitoring.md 00c6ea94bcb73746aef740236632ede123f5b534 
  src/master/master.hpp ea7e9242b62fe6c2cc0e717f9a9f2f0c1cc0a390 
  src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 
  src/master/metrics.hpp e1da18e6ba2737f729e1e30653020538150ae898 
  src/master/metrics.cpp 56a7eef2d279ad3248092d37d19013d3ac110757 


Diff: https://reviews.apache.org/r/68706/diff/3/


Testing (updated)
---

Tested in mmaster with 6 reregistration agents:
"master/slave_reregistrations": 6,

In the middle of reregistration process:
"master/slaves_100_percent_reregistered_secs": 0,
"master/slaves_25_percent_reregistered_secs": 2.244662016,
"master/slaves_50_percent_reregistered_secs": 3.599491072,
"master/slaves_75_percent_reregistered_secs": 9.53919616,
"master/slaves_90_percent_reregistered_secs": 0,
"master/slaves_99_percent_reregistered_secs": 0,

When all registrations finished:
"master/slaves_100_percent_reregistered_secs": 29.697210112,
"master/slaves_25_percent_reregistered_secs": 2.244662016,
"master/slaves_50_percent_reregistered_secs": 3.599491072,
"master/slaves_75_percent_reregistered_secs": 9.53919616,
"master/slaves_90_percent_reregistered_secs": 29.697210112,
"master/slaves_99_percent_reregistered_secs": 29.697210112,

With 3606 agents, the last 1% take significant time
"master/slave_reregistrations": 3606,
"master/slave_shutdowns_canceled": 0,
"master/slave_shutdowns_completed": 0,
"master/slave_shutdowns_scheduled": 0,
"master/slave_unreachable_canceled": 0,
"master/slave_unreachable_completed": 0,
"master/slave_unreachable_scheduled": 0,
"master/slaves_100_percent_reregistered_secs": 58.585202944,
"master/slaves_25_percent_reregistered_secs": 9.966434048,
"master/slaves_50_percent_reregistered_secs": 20.259571968,
"master/slaves_75_percent_reregistered_secs": 30.598885888,
"master/slaves_90_percent_reregistered_secs": 36.396082944,
"master/slaves_99_percent_reregistered_secs": 39.811022848,


Thanks,

Xudong Ni

Review Request 68971: Added pylint init-hook to add Mesos library to `PYTHONPATH`.

2018-10-10 Thread Armand Grillet 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68971/
---

Review request for mesos and Kevin Klues.


Bugs: MESOS-6551
https://issues.apache.org/jira/browse/MESOS-6551


Repository: mesos


Description
---

Without this change, adding `from mesos.exceptions import
MesosException` to a file in the new CLI returns `E0401:
Unable to import 'mesos.exceptions' (import-error)`.

This is because the Mesos library is a local module that we need to add
to the `PYTHONPATH`. This is already done when using the virtual
environment of the new CLI and we now also do it when linting.


Diffs
-

  support/pylint.config af25dd90cb2d467c688ea4b060dc4640040a068b 


Diff: https://reviews.apache.org/r/68971/diff/1/


Testing
---

Updated the agent plugin to do `from mesos.exceptions import MesosException` 
and use `MesosException`. Commited to see the git hook running and saw the 
error `E0401: Unable to import 'mesos.exceptions' (import-error).`. Updated 
`support/pylint.config`, run the git hook again and saw that the error was gone.


Thanks,

Armand Grillet

Re: Review Request 68955: Add per framework metrics in member function.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68955/
---

(Updated Oct. 10, 2018, 9:55 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

This is a preliminary step for control registering
per framework metrics.


Diffs
-

  src/master/metrics.hpp e1da18e6ba2737f729e1e30653020538150ae898 
  src/master/metrics.cpp 56a7eef2d279ad3248092d37d19013d3ac110757 


Diff: https://reviews.apache.org/r/68955/diff/1/


Testing
---


Thanks,

Jacob Janco

Re: Review Request 68954: Fix tests related to allocator options refactor.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68954/
---

(Updated Oct. 10, 2018, 9:55 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

Fix tests related to allocator options refactor.


Diffs
-

  src/tests/allocator.hpp 15a5396986ea6939f25b8d23b87a91c27338fc7b 
  src/tests/api_tests.cpp c681b9b0b83f7472312083730a5433e3d6f0efc0 
  src/tests/hierarchical_allocator_benchmarks.cpp 
bf9167b63747f7b8a402d950947028436307082a 
  src/tests/hierarchical_allocator_tests.cpp 
27fbd9cf0c4442e7675362a806d35bad141ffb6d 
  src/tests/master_allocator_tests.cpp 88288ae8528900b017941db9b07f5f83649fa096 
  src/tests/master_quota_tests.cpp a89574afa8652f86404e8b71ae002c3bc7f83c34 
  src/tests/reservation_tests.cpp d6931220139d91620c886591d7916079b8541982 
  src/tests/resource_offers_tests.cpp 24800c2aa291431e4865e4104da62054b14e5eca 
  src/tests/slave_recovery_tests.cpp 5842ccffaf8c409aaa9c84720ba6c7b07ba6dc7c 


Diff: https://reviews.apache.org/r/68954/diff/1/


Testing
---


Thanks,

Jacob Janco

Re: Review Request 68956: Add flag to toggle per framework metrics.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68956/
---

(Updated Oct. 10, 2018, 9:55 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

In clusters with high numbers of frameworks, it is
necessary to control the registration of per
framework metrics.


Diffs
-

  include/mesos/allocator/allocator.hpp 
61b2b843a10fd73bf876a86e9e18b2d78b70f9b1 
  src/master/allocator/mesos/hierarchical.hpp 
e09a615979014909e7b88d44e6617fd1476415c5 
  src/master/allocator/mesos/hierarchical.cpp 
6f389ed46683f7c642e87b673dc15307cc2b5cd6 
  src/master/allocator/mesos/metrics.hpp 
34cc16b3a4e0622d8ab1c39b093e58e9a37702c0 
  src/master/allocator/mesos/metrics.cpp 
73e68eb4ef53b56f2a764b0504e92d4688eb183c 
  src/master/flags.hpp 4a260155b32fada4ba7a6ae6de7aaecaa25839ff 
  src/master/flags.cpp 6ad53ed44d82e25c05548135b8f152d45ebf6629 
  src/master/framework.cpp 7cfe9f49cb407655984bdee16f7567576d553711 
  src/master/master.hpp ea7e9242b62fe6c2cc0e717f9a9f2f0c1cc0a390 
  src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 
  src/master/metrics.hpp e1da18e6ba2737f729e1e30653020538150ae898 
  src/master/metrics.cpp 56a7eef2d279ad3248092d37d19013d3ac110757 


Diff: https://reviews.apache.org/r/68956/diff/1/


Testing
---


Thanks,

Jacob Janco

Re: Review Request 68955: Add per framework metrics in member function.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68955/
---

(Updated Oct. 10, 2018, 10:19 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

This is a preliminary step for control registering
per framework metrics.


Diffs (updated)
-

  src/master/metrics.hpp e1da18e6ba2737f729e1e30653020538150ae898 
  src/master/metrics.cpp 56a7eef2d279ad3248092d37d19013d3ac110757 


Diff: https://reviews.apache.org/r/68955/diff/2/

Changes: https://reviews.apache.org/r/68955/diff/1-2/


Testing (updated)
---

make check on OSX


Thanks,

Jacob Janco

Re: Review Request 68957: Add documentation for per-framework metrics flag.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68957/
---

(Updated Oct. 10, 2018, 10:19 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

Add documentation for per-framework metrics flag.


Diffs (updated)
-

  docs/configuration/master.md f290e377be9d7424899c82e099be98fc88688de1 


Diff: https://reviews.apache.org/r/68957/diff/2/

Changes: https://reviews.apache.org/r/68957/diff/1-2/


Testing (updated)
---

make check on OSX


Thanks,

Jacob Janco

Re: Review Request 68953: Refactor allocator configuration into class.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68953/
---

(Updated Oct. 10, 2018, 10:19 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

Refactor allocator configuration into class.


Diffs (updated)
-

  include/mesos/allocator/allocator.hpp 
61b2b843a10fd73bf876a86e9e18b2d78b70f9b1 
  src/master/allocator/mesos/allocator.hpp 
a4d7f2be3e8ff71cc2c45cb8ed808b9dbb821aaf 
  src/master/allocator/mesos/hierarchical.hpp 
e09a615979014909e7b88d44e6617fd1476415c5 
  src/master/allocator/mesos/hierarchical.cpp 
6f389ed46683f7c642e87b673dc15307cc2b5cd6 
  src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 


Diff: https://reviews.apache.org/r/68953/diff/2/

Changes: https://reviews.apache.org/r/68953/diff/1-2/


Testing (updated)
---

make check on OSX


Thanks,

Jacob Janco

Re: Review Request 68956: Add flag to toggle per framework metrics.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68956/
---

(Updated Oct. 10, 2018, 10:19 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

In clusters with high numbers of frameworks, it is
necessary to control the registration of per
framework metrics.


Diffs (updated)
-

  include/mesos/allocator/allocator.hpp 
61b2b843a10fd73bf876a86e9e18b2d78b70f9b1 
  src/master/allocator/mesos/hierarchical.hpp 
e09a615979014909e7b88d44e6617fd1476415c5 
  src/master/allocator/mesos/hierarchical.cpp 
6f389ed46683f7c642e87b673dc15307cc2b5cd6 
  src/master/allocator/mesos/metrics.hpp 
34cc16b3a4e0622d8ab1c39b093e58e9a37702c0 
  src/master/allocator/mesos/metrics.cpp 
73e68eb4ef53b56f2a764b0504e92d4688eb183c 
  src/master/flags.hpp 4a260155b32fada4ba7a6ae6de7aaecaa25839ff 
  src/master/flags.cpp 6ad53ed44d82e25c05548135b8f152d45ebf6629 
  src/master/framework.cpp 7cfe9f49cb407655984bdee16f7567576d553711 
  src/master/master.hpp ea7e9242b62fe6c2cc0e717f9a9f2f0c1cc0a390 
  src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 
  src/master/metrics.hpp e1da18e6ba2737f729e1e30653020538150ae898 
  src/master/metrics.cpp 56a7eef2d279ad3248092d37d19013d3ac110757 


Diff: https://reviews.apache.org/r/68956/diff/2/

Changes: https://reviews.apache.org/r/68956/diff/1-2/


Testing (updated)
---

make check on OSX

test-framework in local cluster with flag on/off


Thanks,

Jacob Janco

Re: Review Request 68954: Fix tests related to allocator options refactor.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68954/
---

(Updated Oct. 10, 2018, 10:19 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

Fix tests related to allocator options refactor.


Diffs (updated)
-

  src/tests/allocator.hpp 15a5396986ea6939f25b8d23b87a91c27338fc7b 
  src/tests/api_tests.cpp c681b9b0b83f7472312083730a5433e3d6f0efc0 
  src/tests/hierarchical_allocator_benchmarks.cpp 
bf9167b63747f7b8a402d950947028436307082a 
  src/tests/hierarchical_allocator_tests.cpp 
27fbd9cf0c4442e7675362a806d35bad141ffb6d 
  src/tests/master_allocator_tests.cpp 88288ae8528900b017941db9b07f5f83649fa096 
  src/tests/master_quota_tests.cpp a89574afa8652f86404e8b71ae002c3bc7f83c34 
  src/tests/reservation_tests.cpp d6931220139d91620c886591d7916079b8541982 
  src/tests/resource_offers_tests.cpp 24800c2aa291431e4865e4104da62054b14e5eca 
  src/tests/slave_recovery_tests.cpp 5842ccffaf8c409aaa9c84720ba6c7b07ba6dc7c 


Diff: https://reviews.apache.org/r/68954/diff/2/

Changes: https://reviews.apache.org/r/68954/diff/1-2/


Testing (updated)
---

make check on OSX


Thanks,

Jacob Janco

Re: Review Request 68953: Refactor allocator configuration into class.

2018-10-10 Thread James Peach 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68953/#review209421
---




include/mesos/allocator/allocator.hpp
Lines 47 (patched)


This shouldn't be indented.



include/mesos/allocator/allocator.hpp
Lines 57 (patched)


I think that you can simplify this to:
```C
struct Options
{
  Duration allocationInterval = Seconds();
  ...
};
```



src/master/allocator/mesos/allocator.hpp
Lines 29 (patched)


Don't `use` names in headers, because it has side-effects on the files that 
include them.



src/master/allocator/mesos/hierarchical.hpp
Lines 46 (patched)


Don't `use` names in headers, because it has side-effects on the files that 
include them.



src/master/master.cpp
Lines 758 (patched)


Add a newline here.


- James Peach


On Oct. 10, 2018, 10:19 p.m., Jacob Janco wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68953/
> ---
> 
> (Updated Oct. 10, 2018, 10:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.
> 
> 
> Bugs: MESOS-9301
> https://issues.apache.org/jira/browse/MESOS-9301
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Refactor allocator configuration into class.
> 
> 
> Diffs
> -
> 
>   include/mesos/allocator/allocator.hpp 
> 61b2b843a10fd73bf876a86e9e18b2d78b70f9b1 
>   src/master/allocator/mesos/allocator.hpp 
> a4d7f2be3e8ff71cc2c45cb8ed808b9dbb821aaf 
>   src/master/allocator/mesos/hierarchical.hpp 
> e09a615979014909e7b88d44e6617fd1476415c5 
>   src/master/allocator/mesos/hierarchical.cpp 
> 6f389ed46683f7c642e87b673dc15307cc2b5cd6 
>   src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 
> 
> 
> Diff: https://reviews.apache.org/r/68953/diff/2/
> 
> 
> Testing
> ---
> 
> make check on OSX
> 
> 
> Thanks,
> 
> Jacob Janco
> 
>

Re: Review Request 68957: Add documentation for per-framework metrics flag.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68957/
---

(Updated Oct. 10, 2018, 9:55 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

Add documentation for per-framework metrics flag.


Diffs
-

  docs/configuration/master.md f290e377be9d7424899c82e099be98fc88688de1 


Diff: https://reviews.apache.org/r/68957/diff/1/


Testing
---


Thanks,

Jacob Janco

Re: Review Request 68953: Refactor allocator configuration into class.

2018-10-10 Thread Jacob Janco 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68953/
---

(Updated Oct. 10, 2018, 9:55 p.m.)


Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.


Bugs: MESOS-9301
https://issues.apache.org/jira/browse/MESOS-9301


Repository: mesos


Description
---

Refactor allocator configuration into class.


Diffs
-

  include/mesos/allocator/allocator.hpp 
61b2b843a10fd73bf876a86e9e18b2d78b70f9b1 
  src/master/allocator/mesos/allocator.hpp 
a4d7f2be3e8ff71cc2c45cb8ed808b9dbb821aaf 
  src/master/allocator/mesos/hierarchical.hpp 
e09a615979014909e7b88d44e6617fd1476415c5 
  src/master/allocator/mesos/hierarchical.cpp 
6f389ed46683f7c642e87b673dc15307cc2b5cd6 
  src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 


Diff: https://reviews.apache.org/r/68953/diff/1/


Testing
---


Thanks,

Jacob Janco

Review Request 68984: Changed a benign warning log message in slave.cpp to info.

2018-10-10 Thread Meng Zhu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68984/
---

Review request for mesos.


Repository: mesos


Description
---

Currently, `UpdateFrameworkMessage` is broadcasted by the master
to all agents regardless of whether the framework actually exists
on the agent (see: https://bit.ly/2OiPB4F). So ignoring info
update for framework due to missing framework on the agent is not
unexpected. A warning message would false alarm the user. This
patch changes the log to info to reduce noises.


Diffs
-

  src/slave/slave.cpp 9d84dcb52e5b7e507fc375f184c5f77be08d70a2 


Diff: https://reviews.apache.org/r/68984/diff/1/


Testing
---


Thanks,

Meng Zhu

Re: Review Request 68957: Add documentation for per-framework metrics flag.

2018-10-10 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68957/#review209423
---



PASS: Mesos patch 68957 was successfully built and tested.

Reviews applied: `['68953', '68954', '68955', '68956', '68957']`

All the build artifacts available at: 
http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2447/mesos-review-68957

- Mesos Reviewbot Windows


On Oct. 10, 2018, 10:19 p.m., Jacob Janco wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68957/
> ---
> 
> (Updated Oct. 10, 2018, 10:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, Greg Mann, and James Peach.
> 
> 
> Bugs: MESOS-9301
> https://issues.apache.org/jira/browse/MESOS-9301
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Add documentation for per-framework metrics flag.
> 
> 
> Diffs
> -
> 
>   docs/configuration/master.md f290e377be9d7424899c82e099be98fc88688de1 
> 
> 
> Diff: https://reviews.apache.org/r/68957/diff/2/
> 
> 
> Testing
> ---
> 
> make check on OSX
> 
> 
> Thanks,
> 
> Jacob Janco
> 
>

[GitHub] asfgit closed pull request #311: BugFix backporting 1.4.x

2018-10-10 Thread GitBox 
asfgit closed pull request #311: BugFix backporting 1.4.x
URL: https://github.com/apache/mesos/pull/311
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/3rdparty/libprocess/include/process/subprocess.hpp 
b/3rdparty/libprocess/include/process/subprocess.hpp
index 6a1262340c..135bf243c7 100644
--- a/3rdparty/libprocess/include/process/subprocess.hpp
+++ b/3rdparty/libprocess/include/process/subprocess.hpp
@@ -125,7 +125,8 @@ class Subprocess
 const Option&)>>& clone,
 const std::vector& parent_hooks,
-const std::vector& child_hooks);
+const std::vector& child_hooks,
+const std::vector& whitelist_fds);
 
 IO(const lambda::function()>& _input,
const lambda::function()>& _output)
@@ -305,7 +306,8 @@ class Subprocess
   const Option&)>>& clone,
   const std::vector& parent_hooks,
-  const std::vector& child_hooks);
+  const std::vector& child_hooks,
+  const std::vector& whitelist_fds);
 
   struct Data
   {
@@ -377,7 +379,8 @@ Try subprocess(
 const Option&)>>& clone = None(),
 const std::vector& parent_hooks = {},
-const std::vector& child_hooks = {});
+const std::vector& child_hooks = {},
+const std::vector& whitelist_fds = {});
 
 
 /**
@@ -413,7 +416,8 @@ inline Try subprocess(
 const Option&)>>& clone = None(),
 const std::vector& parent_hooks = {},
-const std::vector& child_hooks = {})
+const std::vector& child_hooks = {},
+const std::vector& whitelist_fds = {})
 {
   std::vector argv = {os::Shell::arg0, os::Shell::arg1, command};
 
@@ -427,7 +431,8 @@ inline Try subprocess(
   environment,
   clone,
   parent_hooks,
-  child_hooks);
+  child_hooks,
+  whitelist_fds);
 }
 
 } // namespace process {
diff --git a/3rdparty/libprocess/src/subprocess.cpp 
b/3rdparty/libprocess/src/subprocess.cpp
index 785e2e1083..fffb640769 100644
--- a/3rdparty/libprocess/src/subprocess.cpp
+++ b/3rdparty/libprocess/src/subprocess.cpp
@@ -324,7 +324,8 @@ Try subprocess(
 const Option&)>>& _clone,
 const vector& parent_hooks,
-const vector& child_hooks)
+const vector& child_hooks,
+const vector& whitelist_fds)
 {
   // TODO(hausdorff): We should error out on Windows here if we are passing
   // parameters that aren't used.
@@ -423,7 +424,8 @@ Try subprocess(
   parent_hooks,
   stdinfds,
   stdoutfds,
-  stderrfds);
+  stderrfds,
+  whitelist_fds);
 
 if (process_data.isError()) {
   process::internal::close(stdinfds, stdoutfds, stderrfds);
diff --git a/3rdparty/libprocess/src/subprocess_posix.cpp 
b/3rdparty/libprocess/src/subprocess_posix.cpp
index 01e3272fcc..caba704833 100644
--- a/3rdparty/libprocess/src/subprocess_posix.cpp
+++ b/3rdparty/libprocess/src/subprocess_posix.cpp
@@ -28,10 +28,12 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
 
+using std::array;
 using std::map;
 using std::string;
 using std::vector;
@@ -46,25 +48,25 @@ Subprocess::IO Subprocess::PIPE()
 {
   return Subprocess::IO(
   []() -> Try {
-int pipefd[2];
-if (::pipe(pipefd) == -1) {
-  return ErrnoError("Failed to create pipe");
+Try> pipefd = os::pipe();
+if (pipefd.isError()) {
+  return Error(pipefd.error());
 }
 
 InputFileDescriptors fds;
-fds.read = pipefd[0];
-fds.write = pipefd[1];
+fds.read = pipefd->at(0);
+fds.write = pipefd->at(1);
 return fds;
   },
   []() -> Try {
-int pipefd[2];
-if (::pipe(pipefd) == -1) {
-  return ErrnoError("Failed to create pipe");
+Try> pipefd = os::pipe();
+if (pipefd.isError()) {
+  return Error(pipefd.error());
 }
 
 OutputFileDescriptors fds;
-fds.read = pipefd[0];
-fds.write = pipefd[1];
+fds.read = pipefd->at(0);
+fds.write = pipefd->at(1);
 return fds;
   });
 }
diff --git a/3rdparty/libprocess/src/subprocess_windows.hpp 
b/3rdparty/libprocess/src/subprocess_windows.hpp
index 0183bb451f..fab57335ed 100644
--- a/3rdparty/libprocess/src/subprocess_windows.hpp
+++ b/3rdparty/libprocess/src/subprocess_windows.hpp
@@ -51,7 +51,8 @@ inline Try<::internal::windows::ProcessData> 
createChildProcess(
 const std::vector& parent_hooks,
 const InputFileDescriptors stdinfds,
 const OutputFileDescriptors stdoutfds,
-const OutputFileDescriptors stderrfds)
+const OutputFileDescriptors stderrfds,
+const std::vector& whitelist_fds = {})
 {
   Try<::internal::windows::ProcessData> process_data =
 ::internal::windows::create_process(
diff --git a/3rdparty/stout/include/stout/os/posix/pipe.hpp

Re: Review Request 68984: Changed a benign warning log message in slave.cpp to info.

2018-10-10 Thread Meng Zhu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68984/
---

(Updated Oct. 10, 2018, 4:54 p.m.)


Review request for mesos.


Changes
---

Updated indentation.


Repository: mesos


Description
---

Currently, `UpdateFrameworkMessage` is broadcasted by the master
to all agents regardless of whether the framework actually exists
on the agent (see: https://bit.ly/2OiPB4F). So ignoring info
update for framework due to missing framework on the agent is not
unexpected. A warning message would false alarm the user. This
patch changes the log to info to reduce noises.


Diffs (updated)
-

  src/slave/slave.cpp 9d84dcb52e5b7e507fc375f184c5f77be08d70a2 


Diff: https://reviews.apache.org/r/68984/diff/2/

Changes: https://reviews.apache.org/r/68984/diff/1-2/


Testing
---


Thanks,

Meng Zhu

[GitHub] asfgit closed pull request #312: BugFix backporting 1.5.x

2018-10-10 Thread GitBox 
asfgit closed pull request #312: BugFix backporting 1.5.x
URL: https://github.com/apache/mesos/pull/312
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/3rdparty/libprocess/include/process/subprocess.hpp 
b/3rdparty/libprocess/include/process/subprocess.hpp
index 6a1262340c..135bf243c7 100644
--- a/3rdparty/libprocess/include/process/subprocess.hpp
+++ b/3rdparty/libprocess/include/process/subprocess.hpp
@@ -125,7 +125,8 @@ class Subprocess
 const Option&)>>& clone,
 const std::vector& parent_hooks,
-const std::vector& child_hooks);
+const std::vector& child_hooks,
+const std::vector& whitelist_fds);
 
 IO(const lambda::function()>& _input,
const lambda::function()>& _output)
@@ -305,7 +306,8 @@ class Subprocess
   const Option&)>>& clone,
   const std::vector& parent_hooks,
-  const std::vector& child_hooks);
+  const std::vector& child_hooks,
+  const std::vector& whitelist_fds);
 
   struct Data
   {
@@ -377,7 +379,8 @@ Try subprocess(
 const Option&)>>& clone = None(),
 const std::vector& parent_hooks = {},
-const std::vector& child_hooks = {});
+const std::vector& child_hooks = {},
+const std::vector& whitelist_fds = {});
 
 
 /**
@@ -413,7 +416,8 @@ inline Try subprocess(
 const Option&)>>& clone = None(),
 const std::vector& parent_hooks = {},
-const std::vector& child_hooks = {})
+const std::vector& child_hooks = {},
+const std::vector& whitelist_fds = {})
 {
   std::vector argv = {os::Shell::arg0, os::Shell::arg1, command};
 
@@ -427,7 +431,8 @@ inline Try subprocess(
   environment,
   clone,
   parent_hooks,
-  child_hooks);
+  child_hooks,
+  whitelist_fds);
 }
 
 } // namespace process {
diff --git a/3rdparty/libprocess/src/subprocess.cpp 
b/3rdparty/libprocess/src/subprocess.cpp
index 785e2e1083..fffb640769 100644
--- a/3rdparty/libprocess/src/subprocess.cpp
+++ b/3rdparty/libprocess/src/subprocess.cpp
@@ -324,7 +324,8 @@ Try subprocess(
 const Option&)>>& _clone,
 const vector& parent_hooks,
-const vector& child_hooks)
+const vector& child_hooks,
+const vector& whitelist_fds)
 {
   // TODO(hausdorff): We should error out on Windows here if we are passing
   // parameters that aren't used.
@@ -423,7 +424,8 @@ Try subprocess(
   parent_hooks,
   stdinfds,
   stdoutfds,
-  stderrfds);
+  stderrfds,
+  whitelist_fds);
 
 if (process_data.isError()) {
   process::internal::close(stdinfds, stdoutfds, stderrfds);
diff --git a/3rdparty/libprocess/src/subprocess_posix.cpp 
b/3rdparty/libprocess/src/subprocess_posix.cpp
index 01e3272fcc..caba704833 100644
--- a/3rdparty/libprocess/src/subprocess_posix.cpp
+++ b/3rdparty/libprocess/src/subprocess_posix.cpp
@@ -28,10 +28,12 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
 
+using std::array;
 using std::map;
 using std::string;
 using std::vector;
@@ -46,25 +48,25 @@ Subprocess::IO Subprocess::PIPE()
 {
   return Subprocess::IO(
   []() -> Try {
-int pipefd[2];
-if (::pipe(pipefd) == -1) {
-  return ErrnoError("Failed to create pipe");
+Try> pipefd = os::pipe();
+if (pipefd.isError()) {
+  return Error(pipefd.error());
 }
 
 InputFileDescriptors fds;
-fds.read = pipefd[0];
-fds.write = pipefd[1];
+fds.read = pipefd->at(0);
+fds.write = pipefd->at(1);
 return fds;
   },
   []() -> Try {
-int pipefd[2];
-if (::pipe(pipefd) == -1) {
-  return ErrnoError("Failed to create pipe");
+Try> pipefd = os::pipe();
+if (pipefd.isError()) {
+  return Error(pipefd.error());
 }
 
 OutputFileDescriptors fds;
-fds.read = pipefd[0];
-fds.write = pipefd[1];
+fds.read = pipefd->at(0);
+fds.write = pipefd->at(1);
 return fds;
   });
 }
diff --git a/3rdparty/libprocess/src/subprocess_windows.hpp 
b/3rdparty/libprocess/src/subprocess_windows.hpp
index 0183bb451f..fab57335ed 100644
--- a/3rdparty/libprocess/src/subprocess_windows.hpp
+++ b/3rdparty/libprocess/src/subprocess_windows.hpp
@@ -51,7 +51,8 @@ inline Try<::internal::windows::ProcessData> 
createChildProcess(
 const std::vector& parent_hooks,
 const InputFileDescriptors stdinfds,
 const OutputFileDescriptors stdoutfds,
-const OutputFileDescriptors stderrfds)
+const OutputFileDescriptors stderrfds,
+const std::vector& whitelist_fds = {})
 {
   Try<::internal::windows::ProcessData> process_data =
 ::internal::windows::create_process(
diff --git a/3rdparty/stout/include/stout/os/posix/pipe.hpp

[GitHub] asfgit closed pull request #313: BugFix backporting 1.6.x

2018-10-10 Thread GitBox 
asfgit closed pull request #313: BugFix backporting 1.6.x
URL: https://github.com/apache/mesos/pull/313
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/3rdparty/libprocess/include/process/subprocess.hpp 
b/3rdparty/libprocess/include/process/subprocess.hpp
index 6a1262340c..135bf243c7 100644
--- a/3rdparty/libprocess/include/process/subprocess.hpp
+++ b/3rdparty/libprocess/include/process/subprocess.hpp
@@ -125,7 +125,8 @@ class Subprocess
 const Option&)>>& clone,
 const std::vector& parent_hooks,
-const std::vector& child_hooks);
+const std::vector& child_hooks,
+const std::vector& whitelist_fds);
 
 IO(const lambda::function()>& _input,
const lambda::function()>& _output)
@@ -305,7 +306,8 @@ class Subprocess
   const Option&)>>& clone,
   const std::vector& parent_hooks,
-  const std::vector& child_hooks);
+  const std::vector& child_hooks,
+  const std::vector& whitelist_fds);
 
   struct Data
   {
@@ -377,7 +379,8 @@ Try subprocess(
 const Option&)>>& clone = None(),
 const std::vector& parent_hooks = {},
-const std::vector& child_hooks = {});
+const std::vector& child_hooks = {},
+const std::vector& whitelist_fds = {});
 
 
 /**
@@ -413,7 +416,8 @@ inline Try subprocess(
 const Option&)>>& clone = None(),
 const std::vector& parent_hooks = {},
-const std::vector& child_hooks = {})
+const std::vector& child_hooks = {},
+const std::vector& whitelist_fds = {})
 {
   std::vector argv = {os::Shell::arg0, os::Shell::arg1, command};
 
@@ -427,7 +431,8 @@ inline Try subprocess(
   environment,
   clone,
   parent_hooks,
-  child_hooks);
+  child_hooks,
+  whitelist_fds);
 }
 
 } // namespace process {
diff --git a/3rdparty/libprocess/src/subprocess.cpp 
b/3rdparty/libprocess/src/subprocess.cpp
index d7a7253632..0b2c02a965 100644
--- a/3rdparty/libprocess/src/subprocess.cpp
+++ b/3rdparty/libprocess/src/subprocess.cpp
@@ -331,7 +331,8 @@ Try subprocess(
 const Option&)>>& _clone,
 const vector& parent_hooks,
-const vector& child_hooks)
+const vector& child_hooks,
+const vector& whitelist_fds)
 {
   // TODO(hausdorff): We should error out on Windows here if we are passing
   // parameters that aren't used.
@@ -430,7 +431,8 @@ Try subprocess(
   parent_hooks,
   stdinfds,
   stdoutfds,
-  stderrfds);
+  stderrfds,
+  whitelist_fds);
 
 if (process_data.isError()) {
   // NOTE: `createChildProcess` either succeeds entirely or returns an
diff --git a/3rdparty/libprocess/src/subprocess_posix.cpp 
b/3rdparty/libprocess/src/subprocess_posix.cpp
index 01e3272fcc..caba704833 100644
--- a/3rdparty/libprocess/src/subprocess_posix.cpp
+++ b/3rdparty/libprocess/src/subprocess_posix.cpp
@@ -28,10 +28,12 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
 
+using std::array;
 using std::map;
 using std::string;
 using std::vector;
@@ -46,25 +48,25 @@ Subprocess::IO Subprocess::PIPE()
 {
   return Subprocess::IO(
   []() -> Try {
-int pipefd[2];
-if (::pipe(pipefd) == -1) {
-  return ErrnoError("Failed to create pipe");
+Try> pipefd = os::pipe();
+if (pipefd.isError()) {
+  return Error(pipefd.error());
 }
 
 InputFileDescriptors fds;
-fds.read = pipefd[0];
-fds.write = pipefd[1];
+fds.read = pipefd->at(0);
+fds.write = pipefd->at(1);
 return fds;
   },
   []() -> Try {
-int pipefd[2];
-if (::pipe(pipefd) == -1) {
-  return ErrnoError("Failed to create pipe");
+Try> pipefd = os::pipe();
+if (pipefd.isError()) {
+  return Error(pipefd.error());
 }
 
 OutputFileDescriptors fds;
-fds.read = pipefd[0];
-fds.write = pipefd[1];
+fds.read = pipefd->at(0);
+fds.write = pipefd->at(1);
 return fds;
   });
 }
diff --git a/3rdparty/libprocess/src/subprocess_windows.hpp 
b/3rdparty/libprocess/src/subprocess_windows.hpp
index c7ed0ad18f..5afd34cd4c 100644
--- a/3rdparty/libprocess/src/subprocess_windows.hpp
+++ b/3rdparty/libprocess/src/subprocess_windows.hpp
@@ -51,7 +51,8 @@ inline Try<::internal::windows::ProcessData> 
createChildProcess(
 const std::vector& parent_hooks,
 const InputFileDescriptors& stdinfds,
 const OutputFileDescriptors& stdoutfds,
-const OutputFileDescriptors& stderrfds)
+const OutputFileDescriptors& stderrfds,
+const std::vector& whitelist_fds = {})
 {
   const std::array fds{
 stdinfds.read, stdoutfds.write, stderrfds.write};
diff --git a/3rdparty/stout/include/stout/os/posix/pipe.hpp

Review Request 68986: Created cgroup recursively when calling prepare on containers.

2018-10-10 Thread fei long 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68986/
---

Review request for mesos and Jie Yu.


Bugs: MESOS-9305
https://issues.apache.org/jira/browse/MESOS-9305


Repository: mesos


Description
---

Created cgroup recursively when calling prepare on containers.


Diffs
-

  src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp 
fbb1b43a6147b4d1d1ef95e2ae1dc6b22261602a 


Diff: https://reviews.apache.org/r/68986/diff/1/


Testing
---


Thanks,

fei long

Review Request 68987: Fixed protobuf map equality check in the URI disk profile adaptor.

2018-10-10 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68987/
---

Review request for mesos and Jie Yu.


Bugs: MESOS-9308
https://issues.apache.org/jira/browse/MESOS-9308


Repository: mesos


Description
---

Fixed protobuf map equality check in the URI disk profile adaptor.


Diffs
-

  src/resource_provider/storage/uri_disk_profile_adaptor.cpp 
d214af89f20f3e8184df83e89512e39e3daa1fc1 


Diff: https://reviews.apache.org/r/68987/diff/1/


Testing
---

make check


Thanks,

Chun-Hung Hsiao

Re: Review Request 68987: Fixed protobuf map equality check in the URI disk profile adaptor.

2018-10-10 Thread Gilbert Song 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68987/#review209434
---


Ship it!




Ship It!

- Gilbert Song


On Oct. 10, 2018, 10:18 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68987/
> ---
> 
> (Updated Oct. 10, 2018, 10:18 p.m.)
> 
> 
> Review request for mesos and Jie Yu.
> 
> 
> Bugs: MESOS-9308
> https://issues.apache.org/jira/browse/MESOS-9308
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Fixed protobuf map equality check in the URI disk profile adaptor.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/storage/uri_disk_profile_adaptor.cpp 
> d214af89f20f3e8184df83e89512e39e3daa1fc1 
> 
> 
> Diff: https://reviews.apache.org/r/68987/diff/1/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 68987: Fixed protobuf map equality check in the URI disk profile adaptor.

2018-10-10 Thread Jie Yu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68987/#review209433
---


Ship it!




Ship It!

- Jie Yu


On Oct. 11, 2018, 5:18 a.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68987/
> ---
> 
> (Updated Oct. 11, 2018, 5:18 a.m.)
> 
> 
> Review request for mesos and Jie Yu.
> 
> 
> Bugs: MESOS-9308
> https://issues.apache.org/jira/browse/MESOS-9308
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Fixed protobuf map equality check in the URI disk profile adaptor.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/storage/uri_disk_profile_adaptor.cpp 
> d214af89f20f3e8184df83e89512e39e3daa1fc1 
> 
> 
> Diff: https://reviews.apache.org/r/68987/diff/1/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 68987: Fixed protobuf map equality check in the URI disk profile adaptor.

2018-10-10 Thread Gilbert Song 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68987/#review209435
---



Do we want a simple unit test?

- Gilbert Song


On Oct. 10, 2018, 10:18 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68987/
> ---
> 
> (Updated Oct. 10, 2018, 10:18 p.m.)
> 
> 
> Review request for mesos and Jie Yu.
> 
> 
> Bugs: MESOS-9308
> https://issues.apache.org/jira/browse/MESOS-9308
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Fixed protobuf map equality check in the URI disk profile adaptor.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/storage/uri_disk_profile_adaptor.cpp 
> d214af89f20f3e8184df83e89512e39e3daa1fc1 
> 
> 
> Diff: https://reviews.apache.org/r/68987/diff/1/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 68986: Created cgroup recursively when calling prepare on containers.

2018-10-10 Thread Gilbert Song 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68986/#review209436
---


Ship it!




Seems like we aim to backport this improvement to previous release. Could we 
add a unit test to simulate the cgroup root being removed issue?

- Gilbert Song


On Oct. 10, 2018, 10:12 p.m., fei long wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68986/
> ---
> 
> (Updated Oct. 10, 2018, 10:12 p.m.)
> 
> 
> Review request for mesos and Jie Yu.
> 
> 
> Bugs: MESOS-9305
> https://issues.apache.org/jira/browse/MESOS-9305
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Created cgroup recursively when calling prepare on containers.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp 
> fbb1b43a6147b4d1d1ef95e2ae1dc6b22261602a 
> 
> 
> Diff: https://reviews.apache.org/r/68986/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> fei long
> 
>

Re: Review Request 68987: Fixed protobuf map equality check in the URI disk profile adaptor.

2018-10-10 Thread Chun-Hung Hsiao 


> On Oct. 11, 2018, 5:19 a.m., Gilbert Song wrote:
> > Do we want a simple unit test?

I'll follow up with one tomorrow.


- Chun-Hung


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68987/#review209435
---


On Oct. 11, 2018, 5:18 a.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68987/
> ---
> 
> (Updated Oct. 11, 2018, 5:18 a.m.)
> 
> 
> Review request for mesos and Jie Yu.
> 
> 
> Bugs: MESOS-9308
> https://issues.apache.org/jira/browse/MESOS-9308
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Fixed protobuf map equality check in the URI disk profile adaptor.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/storage/uri_disk_profile_adaptor.cpp 
> d214af89f20f3e8184df83e89512e39e3daa1fc1 
> 
> 
> Diff: https://reviews.apache.org/r/68987/diff/1/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 68984: Changed a benign warning log message in slave.cpp to info.

2018-10-10 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68984/#review209424
---



PASS: Mesos patch 68984 was successfully built and tested.

Reviews applied: `['68984']`

All the build artifacts available at: 
http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2448/mesos-review-68984

- Mesos Reviewbot Windows


On Oct. 10, 2018, 11:54 p.m., Meng Zhu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68984/
> ---
> 
> (Updated Oct. 10, 2018, 11:54 p.m.)
> 
> 
> Review request for mesos.
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Currently, `UpdateFrameworkMessage` is broadcasted by the master
> to all agents regardless of whether the framework actually exists
> on the agent (see: https://bit.ly/2OiPB4F). So ignoring info
> update for framework due to missing framework on the agent is not
> unexpected. A warning message would false alarm the user. This
> patch changes the log to info to reduce noises.
> 
> 
> Diffs
> -
> 
>   src/slave/slave.cpp 9d84dcb52e5b7e507fc375f184c5f77be08d70a2 
> 
> 
> Diff: https://reviews.apache.org/r/68984/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Meng Zhu
> 
>

Re: Review Request 68706: Added master failover reregistration progress metrics.

2018-10-10 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68706/#review209425
---



Patch looks great!

Reviews applied: [68706]

Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose --disable-libtool-wrappers' ENVIRONMENT='GLOG_v=1 
MESOS_VERBOSE=1'; ./support/docker-build.sh

- Mesos Reviewbot


On Oct. 10, 2018, 5:22 p.m., Xudong Ni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68706/
> ---
> 
> (Updated Oct. 10, 2018, 5:22 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, James Peach, and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-9178
> https://issues.apache.org/jira/browse/MESOS-9178
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> During the master failover, the time that the master elected is
> considered as the start of failover. In the progress of
> reregistration, the percentile represents the time when such
> percentile of agents finished registration again; The percentile of
> these data as in this metrics can represent overall reregistration
> progress; In case of degradation towards to the end of reregistration,
> the high percentile can reflect it; In the case there are unreachable
> agents in the failover, if certain percentile recovery couldn't be
> reached, the intiail value of that percentile will not be updated.
> 
> 
> Diffs
> -
> 
>   docs/monitoring.md 00c6ea94bcb73746aef740236632ede123f5b534 
>   src/master/master.hpp ea7e9242b62fe6c2cc0e717f9a9f2f0c1cc0a390 
>   src/master/master.cpp 06d769aeba16586a020729d454f4d00688b78c78 
>   src/master/metrics.hpp e1da18e6ba2737f729e1e30653020538150ae898 
>   src/master/metrics.cpp 56a7eef2d279ad3248092d37d19013d3ac110757 
> 
> 
> Diff: https://reviews.apache.org/r/68706/diff/3/
> 
> 
> Testing
> ---
> 
> Tested in mmaster with 6 reregistration agents:
> "master/slave_reregistrations": 6,
> 
> In the middle of reregistration process:
> "master/slaves_100_percent_reregistered_secs": 0,
> "master/slaves_25_percent_reregistered_secs": 2.244662016,
> "master/slaves_50_percent_reregistered_secs": 3.599491072,
> "master/slaves_75_percent_reregistered_secs": 9.53919616,
> "master/slaves_90_percent_reregistered_secs": 0,
> "master/slaves_99_percent_reregistered_secs": 0,
> 
> When all registrations finished:
> "master/slaves_100_percent_reregistered_secs": 29.697210112,
> "master/slaves_25_percent_reregistered_secs": 2.244662016,
> "master/slaves_50_percent_reregistered_secs": 3.599491072,
> "master/slaves_75_percent_reregistered_secs": 9.53919616,
> "master/slaves_90_percent_reregistered_secs": 29.697210112,
> "master/slaves_99_percent_reregistered_secs": 29.697210112,
> 
> With 3606 agents, the last 1% take significant time
> "master/slave_reregistrations": 3606,
> "master/slave_shutdowns_canceled": 0,
> "master/slave_shutdowns_completed": 0,
> "master/slave_shutdowns_scheduled": 0,
> "master/slave_unreachable_canceled": 0,
> "master/slave_unreachable_completed": 0,
> "master/slave_unreachable_scheduled": 0,
> "master/slaves_100_percent_reregistered_secs": 58.585202944,
> "master/slaves_25_percent_reregistered_secs": 9.966434048,
> "master/slaves_50_percent_reregistered_secs": 20.259571968,
> "master/slaves_75_percent_reregistered_secs": 30.598885888,
> "master/slaves_90_percent_reregistered_secs": 36.396082944,
> "master/slaves_99_percent_reregistered_secs": 39.811022848,
> 
> 
> Thanks,
> 
> Xudong Ni
> 
>