date:20141104

Hi Ankur,

I like it, too. However, I cannot refrain from relaying (not my choice of word 
here) to you the advice to break your relatively large patch down into smaller 
parts. My patch for MESOS-336 certainly was, as I know now. My plan is to get 
MESOS-1316 (which is now under review) and MESOS-1248 (which should be quick) 
committed and then rebase MESOS-336 in smaller pieces. If you have small 
pieces, too, I do think we can do this somewhat concurrently, as your 
refactoring seems to affect mostly how the transport part works. I am on the 
other hand mostly interested in the bookkeeping of what is in progress, what 
succeeded, what failed, what has been put where, and so on, which can be 
separated, if we are careful about it.

(BTW, I think we need both unit tests and integration tests. And we don’t have 
nearly enough of either.)

Bernd

 On Nov 3, 2014, at 6:18 PM, Adam Bordelon a...@mesosphere.io wrote:
 
 + Bernd, who has done some fetcher work, including additional testing, for 
 MESOS-1316, MESOS-1945, and MESOS-336
 
 On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon dha...@twopensource.com 
 mailto:dha...@twopensource.com wrote:
 Hi Ankur
 
 I think this is a great approach. It makes the code much simpler, extensible, 
 and more testable. Anyone that's heard me rant knows I am a big fan of unit 
 tests over integration tests, so this shouldn't surprise anyone :)
 
 If you haven't already, please read the documentation on contributing to 
 Mesos and the style guide to ensure all the naming is as expected, then you 
 can push the patch to reviewboard to get it reviewed and committed.
 
 On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi,
 
 I did some learning today! This is pretty much a very rough draft of the 
 tests/refactor of mesos-fetcher that I have come up with. Again, If there are 
 some obvious mistakes, please let me know. (this is my first pass after all).
 https://github.com/ankurcha/mesos/compare/prefer_2 
 https://github.com/ankurcha/mesos/compare/prefer_2
 
 My main intention is to break the logic of the fetcher info some very 
 discrete components that i can write tests against. I am still re-learning 
 cpp/mesos code styles etc so I may be a little slow to catch up but I would 
 really appreciate any comments and/or suggestions.
 
 -- Ankur
 @ankurcha
 
 On 2 Nov 2014, at 18:17, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 
 Hi,
 
 I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` is 
 pretty coarse grained and are more on the lines of a functional test. I was 
 going to add some tests but it seems like if I am to do that I would need to 
 add a test dependency on hadoop. 
 
 As an alternative, I propose adding a good set of unit tests around the 
 methods used by `src/launcher/fetcher.cpp` and `src/hdfs/hdfs.cpp`. This 
 should be able to catch a good portion of cases at the same time keeping the 
 dependencies and runtime of tests low. What do you guys thing about this?
 
 PS: I am pretty green in terms of gtest and the overall c++ testing 
 methodology. Can someone give me pointers to good examples of tests in the 
 codebase.
 
 -- Ankur
 
 On 1 Nov 2014, at 22:54, Adam Bordelon a...@mesosphere.io 
 mailto:a...@mesosphere.io wrote:
 
 Thank you Ankur. At first glance, it looks great. We'll do a more thorough 
 review of it very soon.
 I know Tim St. Clair had some ideas for fixing MESOS-1711 
 https://issues.apache.org/jira/browse/MESOS-1711; he may want to review 
 too.
 
 On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi Tim,
 
 I just created a review https://reviews.apache.org/r/27483/ 
 https://reviews.apache.org/r/27483/ It's my first stab at it and I will 
 try to add more tests as I figure out how to do the hadoop mocking and 
 stuff. Have a look and let me know what you think about it so far.
 
 -- Ankur
 
 On 1 Nov 2014, at 20:05, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 
 Yea, i saw that the minute i pressed send. I'll start the review board so 
 that people can have a look at the change.
 
 -- Ankur
 
 On 1 Nov 2014, at 20:01, Tim Chen t...@mesosphere.io 
 mailto:t...@mesosphere.io wrote:
 
 Hi Ankur,
 
 There is a fetcher_tests.cpp in src/tests.
 
 Tim
 
 On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi Tim,
 
 I am trying to find/write some test cases. I couldn't find a 
 fetcher_tests.{cpp|hpp} so once I have something, I'll post on review 
 board. I am new to gmock/gtest so bear with me while i get up to speed.
 
 -- Ankur
 
 On 1 Nov 2014, at 19:23, Timothy Chen t...@mesosphere.io 
 mailto:t...@mesosphere.io wrote:
 
 Hi Ankur,
 
 Can you post on reviewboard? We can discuss more about the code there.
 
 Tim
 
 Sent from my iPhone
 
 On Nov 1, 2014, at 6:29 PM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com

Re: Why rely on url scheme for fetching?

Typo: not - note

 On Nov 4, 2014, at 10:59 AM, Bernd Mathiske be...@mesosphere.io wrote:
 
 Hi Ankur,
 
 I like it, too. However, I cannot refrain from relaying (not my choice of 
 word here) to you the advice to break your relatively large patch down into 
 smaller parts. My patch for MESOS-336 certainly was, as I know now. My plan 
 is to get MESOS-1316 (which is now under review) and MESOS-1248 (which should 
 be quick) committed and then rebase MESOS-336 in smaller pieces. If you have 
 small pieces, too, I do think we can do this somewhat concurrently, as your 
 refactoring seems to affect mostly how the transport part works. I am on the 
 other hand mostly interested in the bookkeeping of what is in progress, what 
 succeeded, what failed, what has been put where, and so on, which can be 
 separated, if we are careful about it.
 
 (BTW, I think we need both unit tests and integration tests. And we don’t 
 have nearly enough of either.)
 
 Bernd
 
 On Nov 3, 2014, at 6:18 PM, Adam Bordelon a...@mesosphere.io 
 mailto:a...@mesosphere.io wrote:
 
 + Bernd, who has done some fetcher work, including additional testing, for 
 MESOS-1316, MESOS-1945, and MESOS-336
 
 On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon dha...@twopensource.com 
 mailto:dha...@twopensource.com wrote:
 Hi Ankur
 
 I think this is a great approach. It makes the code much simpler, 
 extensible, and more testable. Anyone that's heard me rant knows I am a big 
 fan of unit tests over integration tests, so this shouldn't surprise anyone 
 :)
 
 If you haven't already, please read the documentation on contributing to 
 Mesos and the style guide to ensure all the naming is as expected, then you 
 can push the patch to reviewboard to get it reviewed and committed.
 
 On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi,
 
 I did some learning today! This is pretty much a very rough draft of the 
 tests/refactor of mesos-fetcher that I have come up with. Again, If there 
 are some obvious mistakes, please let me know. (this is my first pass after 
 all).
 https://github.com/ankurcha/mesos/compare/prefer_2 
 https://github.com/ankurcha/mesos/compare/prefer_2
 
 My main intention is to break the logic of the fetcher info some very 
 discrete components that i can write tests against. I am still re-learning 
 cpp/mesos code styles etc so I may be a little slow to catch up but I would 
 really appreciate any comments and/or suggestions.
 
 -- Ankur
 @ankurcha
 
 On 2 Nov 2014, at 18:17, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 
 Hi,
 
 I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` is 
 pretty coarse grained and are more on the lines of a functional test. I was 
 going to add some tests but it seems like if I am to do that I would need 
 to add a test dependency on hadoop. 
 
 As an alternative, I propose adding a good set of unit tests around the 
 methods used by `src/launcher/fetcher.cpp` and `src/hdfs/hdfs.cpp`. This 
 should be able to catch a good portion of cases at the same time keeping 
 the dependencies and runtime of tests low. What do you guys thing about 
 this?
 
 PS: I am pretty green in terms of gtest and the overall c++ testing 
 methodology. Can someone give me pointers to good examples of tests in the 
 codebase.
 
 -- Ankur
 
 On 1 Nov 2014, at 22:54, Adam Bordelon a...@mesosphere.io 
 mailto:a...@mesosphere.io wrote:
 
 Thank you Ankur. At first glance, it looks great. We'll do a more thorough 
 review of it very soon.
 I know Tim St. Clair had some ideas for fixing MESOS-1711 
 https://issues.apache.org/jira/browse/MESOS-1711; he may want to review 
 too.
 
 On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi Tim,
 
 I just created a review https://reviews.apache.org/r/27483/ 
 https://reviews.apache.org/r/27483/ It's my first stab at it and I will 
 try to add more tests as I figure out how to do the hadoop mocking and 
 stuff. Have a look and let me know what you think about it so far.
 
 -- Ankur
 
 On 1 Nov 2014, at 20:05, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 
 Yea, i saw that the minute i pressed send. I'll start the review board so 
 that people can have a look at the change.
 
 -- Ankur
 
 On 1 Nov 2014, at 20:01, Tim Chen t...@mesosphere.io 
 mailto:t...@mesosphere.io wrote:
 
 Hi Ankur,
 
 There is a fetcher_tests.cpp in src/tests.
 
 Tim
 
 On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi Tim,
 
 I am trying to find/write some test cases. I couldn't find a 
 fetcher_tests.{cpp|hpp} so once I have something, I'll post on review 
 board. I am new to gmock/gtest so bear with me while i get up to speed.
 
 -- Ankur
 
 On 1 Nov 2014, at 19:23, Timothy Chen t...@mesosphere.io 
 mailto:t...@mesosphere.io wrote:
 
 Hi Ankur,
 
 Can you post on reviewboard? We can discuss more

Re: Why rely on url scheme for fetching?

Hi ben,

Thanks for the follow up. I think thats a perfectly fine game plan. I think I 
can break down things into smaller, more isolated chunks. But from the looks of 
MESOS-1316 the invocation code and the testing code seems to change a lot (in a 
good way), so to avoid a bunch of wasted cycles, I was going to hold off a 
little till it is committed and then go on to refactoring the transport parts. 
The json config change (MESOS-1248) is completely disconnected to this so I am 
not too worried about that one.
The only place where some coordination may be required is the 
launcher/fetcher.cpp. It seems there are common changes happening between my  
change https://reviews.apache.org/r/27483/ 
https://reviews.apache.org/r/27483/ and your 
https://reviews.apache.org/r/21316 https://reviews.apache.org/r/21316 which 
would mean a little rework but it's totally within reasonable limits.
I think if you get the MESOS-1316 committed and I get MESOS-1711 ( 
https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483 ) in, 
both of us would get unblocked in a way that we can get a way that lets us 
concurrently work on MESOS-336 and refactoring fetcher into more testable parts.

Does this make sense to you or did I misunderstand some part?

-- Ankur

 On 4 Nov 2014, at 02:05, Bernd Mathiske be...@mesosphere.io wrote:
 
 Typo: not - note
 
 On Nov 4, 2014, at 10:59 AM, Bernd Mathiske be...@mesosphere.io wrote:
 
 Hi Ankur,
 
 I like it, too. However, I cannot refrain from relaying (not my choice of 
 word here) to you the advice to break your relatively large patch down into 
 smaller parts. My patch for MESOS-336 certainly was, as I know now. My plan 
 is to get MESOS-1316 (which is now under review) and MESOS-1248 (which 
 should be quick) committed and then rebase MESOS-336 in smaller pieces. If 
 you have small pieces, too, I do think we can do this somewhat concurrently, 
 as your refactoring seems to affect mostly how the transport part works. I 
 am on the other hand mostly interested in the bookkeeping of what is in 
 progress, what succeeded, what failed, what has been put where, and so on, 
 which can be separated, if we are careful about it.
 
 (BTW, I think we need both unit tests and integration tests. And we don’t 
 have nearly enough of either.)
 
 Bernd
 
 On Nov 3, 2014, at 6:18 PM, Adam Bordelon a...@mesosphere.io 
 mailto:a...@mesosphere.io wrote:
 
 + Bernd, who has done some fetcher work, including additional testing, for 
 MESOS-1316, MESOS-1945, and MESOS-336
 
 On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon dha...@twopensource.com 
 mailto:dha...@twopensource.com wrote:
 Hi Ankur
 
 I think this is a great approach. It makes the code much simpler, 
 extensible, and more testable. Anyone that's heard me rant knows I am a big 
 fan of unit tests over integration tests, so this shouldn't surprise anyone 
 :)
 
 If you haven't already, please read the documentation on contributing to 
 Mesos and the style guide to ensure all the naming is as expected, then you 
 can push the patch to reviewboard to get it reviewed and committed.
 
 On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi,
 
 I did some learning today! This is pretty much a very rough draft of the 
 tests/refactor of mesos-fetcher that I have come up with. Again, If there 
 are some obvious mistakes, please let me know. (this is my first pass after 
 all).
 https://github.com/ankurcha/mesos/compare/prefer_2 
 https://github.com/ankurcha/mesos/compare/prefer_2
 
 My main intention is to break the logic of the fetcher info some very 
 discrete components that i can write tests against. I am still re-learning 
 cpp/mesos code styles etc so I may be a little slow to catch up but I would 
 really appreciate any comments and/or suggestions.
 
 -- Ankur
 @ankurcha
 
 On 2 Nov 2014, at 18:17, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 
 Hi,
 
 I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` 
 is pretty coarse grained and are more on the lines of a functional test. I 
 was going to add some tests but it seems like if I am to do that I would 
 need to add a test dependency on hadoop. 
 
 As an alternative, I propose adding a good set of unit tests around the 
 methods used by `src/launcher/fetcher.cpp` and `src/hdfs/hdfs.cpp`. This 
 should be able to catch a good portion of cases at the same time keeping 
 the dependencies and runtime of tests low. What do you guys thing about 
 this?
 
 PS: I am pretty green in terms of gtest and the overall c++ testing 
 methodology. Can someone give me pointers to good examples of tests in the 
 codebase.
 
 -- Ankur
 
 On 1 Nov 2014, at 22:54, Adam Bordelon a...@mesosphere.io 
 mailto:a...@mesosphere.io wrote:
 
 Thank you Ankur. At first glance, it looks great. We'll do a more 
 thorough review of it very soon.
 I know Tim St. Clair had some ideas for fixing MESOS-1711

Review Request 27567: ~/patch/MESOS-2038.patch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/
---

Review request for mesos and Ben Mahler.


Bugs: MESOS-2038
https://issues.apache.org/jira/browse/MESOS-2038


Repository: mesos-git


Description
---

Removed a few lines of dead code that coverty discovered.


Diffs
-

  src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 

Diff: https://reviews.apache.org/r/27567/diff/


Testing
---

none.
expecting/waiting for review bot to report no problem.


Thanks,

Bernd Mathiske

Re: Review Request 23912: Fix MESOS-947: Slave should properly handle a killTask() that arrives between runTask() and _runTask()



 On Oct. 23, 2014, 12:54 p.m., Ben Mahler wrote:
  src/slave/slave.cpp, lines 1270-1273
  https://reviews.apache.org/r/23912/diff/13/?file=724332#file724332line1270
 
  Hey Bernd and Vinod, coverity just reported this, looks like it's now 
  dead code!
  
  Can you follow up?
 
 Bernd Mathiske wrote:
 Coverty is correct. I believe we can remove this code and the subsequent 
 line as well. According to how I understand it (given Vinod's reviews), the 
 framework must still exist as long as any pending task exists. It follows 
 that this code must have already been de facto dead before, because the patch 
 for MESOS-947 does not change when a framework gets instantiated. It just 
 reclaims it more eagerly.

Filed MESOS-238 and proposed patch https://reviews.apache.org/r/27567/


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23912/#review58074
---


On Oct. 17, 2014, 9:25 a.m., Bernd Mathiske wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23912/
 ---
 
 (Updated Oct. 17, 2014, 9:25 a.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-947
 https://issues.apache.org/jira/browse/MESOS-947
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Fixes MESOS-947 Slave should properly handle a killTask() that arrives 
 between runTask() and _runTask().
 
 Slave::killTask() did not check for task in question combination to be 
 pending (i.e. Slave::runTask had happened, but Slave::_runTask had not yet) 
 and then erroneously assumed that Slave::runTask() had not been executed. The 
 task was then marked LOST instead of KILLED. But Slave::runTask had 
 already scheduled Slave::_runTask to follow. Now the entry for being 
 pending is removed, and the task is marked KILLED, and _runTask gets 
 informed about this. It checks whether the task in question is currently 
 pending and if it is not, then it infers that the task has been killed and 
 does not erroneously try to complete launching it.
 
 
 Diffs
 -
 
   src/slave/slave.hpp 342b09fc084c20d98d096bb129830440179c092c 
   src/slave/slave.cpp 0e342ed35e3db3b68f9f32b6cf4ace23e4a4db38 
   src/tests/mesos.hpp 957e2233cc11c438fd80d3b6d1907a1223093104 
   src/tests/mesos.cpp 3dcb2acd5ad4ab5e3a7b4fe524ee077558112773 
   src/tests/slave_tests.cpp f585bdd20ae1af466f2c1b4d85331ac67451552f 
 
 Diff: https://reviews.apache.org/r/23912/diff/
 
 
 Testing
 ---
 
 Wrote a unit test that reliably created the situation described in the 
 ticket. Observed that TASK_LOST and the listed log output occurred. This 
 pointed directly to the lines in killTask() where the problem is rooted. Ran 
 the test after fixing, it succeeded. Checked the log. It looks like a clean 
 kill now :-)
 
 
 Thanks,
 
 Bernd Mathiske

Re: Review Request 27567: ~/patch/MESOS-2038.patch

2014-11-04 Thread Adam B


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/#review59760
---

Ship it!


Ship It!

- Adam B


On Nov. 4, 2014, 2:56 a.m., Bernd Mathiske wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27567/
 ---
 
 (Updated Nov. 4, 2014, 2:56 a.m.)
 
 
 Review request for mesos and Ben Mahler.
 
 
 Bugs: MESOS-2038
 https://issues.apache.org/jira/browse/MESOS-2038
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Removed a few lines of dead code that coverty discovered.
 
 
 Diffs
 -
 
   src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 
 
 Diff: https://reviews.apache.org/r/27567/diff/
 
 
 Testing
 ---
 
 none.
 expecting/waiting for review bot to report no problem.
 
 
 Thanks,
 
 Bernd Mathiske

Re: Why rely on url scheme for fetching?

Hi Ankur, 

I am Bernd, not Ben, but I’ll try to do my best :-)

Your plan looks good to me and your patch for MESOS-1711 seems uncomplicated 
enough to not cause me major problems no matter when it lands. 

Bernd

 On Nov 4, 2014, at 11:44 AM, Ankur Chauhan an...@malloc64.com wrote:
 
 Hi ben,
 
 Thanks for the follow up. I think thats a perfectly fine game plan. I think I 
 can break down things into smaller, more isolated chunks. But from the looks 
 of MESOS-1316 the invocation code and the testing code seems to change a lot 
 (in a good way), so to avoid a bunch of wasted cycles, I was going to hold 
 off a little till it is committed and then go on to refactoring the transport 
 parts. The json config change (MESOS-1248) is completely disconnected to this 
 so I am not too worried about that one.
 The only place where some coordination may be required is the 
 launcher/fetcher.cpp. It seems there are common changes happening between my  
 change https://reviews.apache.org/r/27483/ 
 https://reviews.apache.org/r/27483/https://reviews.apache.org/r/27483/ 
 https://reviews.apache.org/r/27483/ and your 
 https://reviews.apache.org/r/21316 
 https://reviews.apache.org/r/21316https://reviews.apache.org/r/21316 
 https://reviews.apache.org/r/21316 which would mean a little rework but 
 it's totally within reasonable limits.
 I think if you get the MESOS-1316 committed and I get MESOS-1711 ( 
 https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483 
 https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483 ) 
 in, both of us would get unblocked in a way that we can get a way that lets 
 us concurrently work on MESOS-336 and refactoring fetcher into more testable 
 parts.
 
 Does this make sense to you or did I misunderstand some part?
 
 -- Ankur
 
 On 4 Nov 2014, at 02:05, Bernd Mathiske be...@mesosphere.io 
 mailto:be...@mesosphere.io wrote:
 
 Typo: not - note
 
 On Nov 4, 2014, at 10:59 AM, Bernd Mathiske be...@mesosphere.io 
 mailto:be...@mesosphere.io wrote:
 
 Hi Ankur,
 
 I like it, too. However, I cannot refrain from relaying (not my choice of 
 word here) to you the advice to break your relatively large patch down into 
 smaller parts. My patch for MESOS-336 certainly was, as I know now. My plan 
 is to get MESOS-1316 (which is now under review) and MESOS-1248 (which 
 should be quick) committed and then rebase MESOS-336 in smaller pieces. If 
 you have small pieces, too, I do think we can do this somewhat 
 concurrently, as your refactoring seems to affect mostly how the transport 
 part works. I am on the other hand mostly interested in the bookkeeping of 
 what is in progress, what succeeded, what failed, what has been put where, 
 and so on, which can be separated, if we are careful about it.
 
 (BTW, I think we need both unit tests and integration tests. And we don’t 
 have nearly enough of either.)
 
 Bernd
 
 On Nov 3, 2014, at 6:18 PM, Adam Bordelon a...@mesosphere.io 
 mailto:a...@mesosphere.iomailto:a...@mesosphere.io 
 mailto:a...@mesosphere.io wrote:
 
 + Bernd, who has done some fetcher work, including additional testing, for 
 MESOS-1316, MESOS-1945, and MESOS-336
 
 On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon dha...@twopensource.com 
 mailto:dha...@twopensource.commailto:dha...@twopensource.com 
 mailto:dha...@twopensource.com wrote:
 Hi Ankur
 
 I think this is a great approach. It makes the code much simpler, 
 extensible, and more testable. Anyone that's heard me rant knows I am a 
 big fan of unit tests over integration tests, so this shouldn't surprise 
 anyone :)
 
 If you haven't already, please read the documentation on contributing to 
 Mesos and the style guide to ensure all the naming is as expected, then 
 you can push the patch to reviewboard to get it reviewed and committed.
 
 On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.commailto:an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 Hi,
 
 I did some learning today! This is pretty much a very rough draft of the 
 tests/refactor of mesos-fetcher that I have come up with. Again, If there 
 are some obvious mistakes, please let me know. (this is my first pass 
 after all).
 https://github.com/ankurcha/mesos/compare/prefer_2 
 https://github.com/ankurcha/mesos/compare/prefer_2https://github.com/ankurcha/mesos/compare/prefer_2
  https://github.com/ankurcha/mesos/compare/prefer_2
 
 My main intention is to break the logic of the fetcher info some very 
 discrete components that i can write tests against. I am still re-learning 
 cpp/mesos code styles etc so I may be a little slow to catch up but I 
 would really appreciate any comments and/or suggestions.
 
 -- Ankur
 @ankurcha
 
 On 2 Nov 2014, at 18:17, Ankur Chauhan an...@malloc64.com 
 mailto:an...@malloc64.commailto:an...@malloc64.com 
 mailto:an...@malloc64.com wrote:
 
 Hi,
 
 I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` 
 is pretty coarse grained and are more on

Re: Review Request 27040: Replace hard-coded reap interval with a constant

2014-11-04 Thread Alexander Rukletsov



 On Oct. 22, 2014, 5:26 p.m., Ben Mahler wrote:
  There is already a constant, MESOS-1935 is to ensure that code that 
  previously hardcoded Seconds(1) now uses MAX_REAP_INTERVAL from reap.hpp.
 
 Aditi Bhatnagar wrote:
 Sorry, but I didn't find the constants in 
 https://github.com/apache/mesos/blob/4693728e4f604a1cff6f01d5f2d8f0ec60eb2f96/3rdparty/libprocess/include/process/reap.hpp
  .

`MAX_REAP_INTERVAL()` acts as a constant.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27040/#review57826
---


On Oct. 22, 2014, 2:45 p.m., Aditi Bhatnagar wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27040/
 ---
 
 (Updated Oct. 22, 2014, 2:45 p.m.)
 
 
 Review request for mesos and Dominic Hamon.
 
 
 Bugs: MESOS-1935
 https://issues.apache.org/jira/browse/MESOS-1935
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Replace the hard-coded value for the maximal and minimum reap intervals with 
 the constants from reap.hpp
 
 
 Diffs
 -
 
   3rdparty/libprocess/include/process/reap.hpp 5e5051a 
   3rdparty/libprocess/include/process/reap.hpp~ PRE-CREATION 
   3rdparty/libprocess/src/reap.cpp afd956b 
   3rdparty/libprocess/src/reap.cpp~ PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/27040/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Aditi Bhatnagar

Re: Why rely on url scheme for fetching?

Hi,

Sorry about that bernd! :s/ben/bernd/g

-- ankur
Sent from my iPhone

On Nov 4, 2014, at 3:13 AM, Bernd Mathiske be...@mesosphere.io wrote:

Hi Ankur,

I am Bernd, not Ben, but I’ll try to do my best :-)

Your plan looks good to me and your patch for MESOS-1711 seems uncomplicated
enough to not cause me major problems no matter when it lands.

Bernd

On Nov 4, 2014, at 11:44 AM, Ankur Chauhan an...@malloc64.com wrote:

Hi ben,

Thanks for the follow up. I think thats a perfectly fine game plan. I think
I can break down things into smaller, more isolated chunks. But from the
looks of MESOS-1316 the invocation code and the testing code seems to change
a lot (in a good way), so to avoid a bunch of wasted cycles, I was going to
hold off a little till it is committed and then go on to refactoring the
transport parts. The json config change (MESOS-1248) is completely
disconnected to this so I am not too worried about that one.
The only place where some coordination may be required is the
launcher/fetcher.cpp. It seems there are common changes happening between my
change https://reviews.apache.org/r/27483/
https://reviews.apache.org/r/27483/https://reviews.apache.org/r/27483/
https://reviews.apache.org/r/27483/ and your
https://reviews.apache.org/r/21316
https://reviews.apache.org/r/21316https://reviews.apache.org/r/21316
https://reviews.apache.org/r/21316 which would mean a little rework but
it's totally within reasonable limits.
I think if you get the MESOS-1316 committed and I get MESOS-1711 (
https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483
https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483 )
in, both of us would get unblocked in a way that we can get a way that lets
us concurrently work on MESOS-336 and refactoring fetcher into more testable
parts.

Does this make sense to you or did I misunderstand some part?

-- Ankur

On 4 Nov 2014, at 02:05, Bernd Mathiske be...@mesosphere.io
mailto:be...@mesosphere.io wrote:

Typo: not - note

On Nov 4, 2014, at 10:59 AM, Bernd Mathiske be...@mesosphere.io
mailto:be...@mesosphere.io wrote:

Hi Ankur,

I like it, too. However, I cannot refrain from relaying (not my choice of
word here) to you the advice to break your relatively large patch down
into smaller parts. My patch for MESOS-336 certainly was, as I know now.
My plan is to get MESOS-1316 (which is now under review) and MESOS-1248
(which should be quick) committed and then rebase MESOS-336 in smaller
pieces. If you have small pieces, too, I do think we can do this somewhat
concurrently, as your refactoring seems to affect mostly how the transport
part works. I am on the other hand mostly interested in the bookkeeping of
what is in progress, what succeeded, what failed, what has been put where,
and so on, which can be separated, if we are careful about it.

(BTW, I think we need both unit tests and integration tests. And we don’t
have nearly enough of either.)

Bernd

On Nov 3, 2014, at 6:18 PM, Adam Bordelon a...@mesosphere.io
mailto:a...@mesosphere.iomailto:a...@mesosphere.io
mailto:a...@mesosphere.io wrote:

+ Bernd, who has done some fetcher work, including additional testing,
for MESOS-1316, MESOS-1945, and MESOS-336

On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon dha...@twopensource.com
mailto:dha...@twopensource.commailto:dha...@twopensource.com
mailto:dha...@twopensource.com wrote:
Hi Ankur

I think this is a great approach. It makes the code much simpler,
extensible, and more testable. Anyone that's heard me rant knows I am a
big fan of unit tests over integration tests, so this shouldn't surprise
anyone :)

If you haven't already, please read the documentation on contributing to
Mesos and the style guide to ensure all the naming is as expected, then
you can push the patch to reviewboard to get it reviewed and committed.

On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com
mailto:an...@malloc64.commailto:an...@malloc64.com
mailto:an...@malloc64.com wrote:
Hi,

I did some learning today! This is pretty much a very rough draft of the
tests/refactor of mesos-fetcher that I have come up with. Again, If there
are some obvious mistakes, please let me know. (this is my first pass
after all).
https://github.com/ankurcha/mesos/compare/prefer_2
https://github.com/ankurcha/mesos/compare/prefer_2https://github.com/ankurcha/mesos/compare/prefer_2
https://github.com/ankurcha/mesos/compare/prefer_2

My main intention is to break the logic of the fetcher info some very
discrete components that i can write tests against. I am still
re-learning cpp/mesos code styles etc so I may be a little slow to catch
up but I would really appreciate any comments and/or suggestions.

-- Ankur
@ankurcha

On 2 Nov 2014, at 18:17, Ankur Chauhan an...@malloc64.com
mailto:an...@malloc64.commailto:an...@malloc64.com

Re: Review Request 27567: ~/patch/MESOS-2038.patch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/#review59762
---


Patch looks great!

Reviews applied: [23912, 27567]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 10:56 a.m., Bernd Mathiske wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27567/
 ---
 
 (Updated Nov. 4, 2014, 10:56 a.m.)
 
 
 Review request for mesos and Ben Mahler.
 
 
 Bugs: MESOS-2038
 https://issues.apache.org/jira/browse/MESOS-2038
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Removed a few lines of dead code that coverty discovered.
 
 
 Diffs
 -
 
   src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 
 
 Diff: https://reviews.apache.org/r/27567/diff/
 
 
 Testing
 ---
 
 none.
 expecting/waiting for review bot to report no problem.
 
 
 Thanks,
 
 Bernd Mathiske

Re: Review Request 27567: MESOS-2038: removed dead code from _runTask()


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/
---

(Updated Nov. 4, 2014, 6:12 a.m.)


Review request for mesos and Ben Mahler.


Changes
---

fixed summary


Summary (updated)
-

MESOS-2038: removed dead code from _runTask()


Bugs: MESOS-2038
https://issues.apache.org/jira/browse/MESOS-2038


Repository: mesos-git


Description
---

Removed a few lines of dead code that coverty discovered.


Diffs
-

  src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 

Diff: https://reviews.apache.org/r/27567/diff/


Testing
---

none.
expecting/waiting for review bot to report no problem.


Thanks,

Bernd Mathiske

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

2014-11-04 Thread Timothy St. Clair



 On Nov. 3, 2014, 9:59 p.m., Timothy St. Clair wrote:
  src/launcher/fetcher.cpp, line 77
  https://reviews.apache.org/r/27483/diff/1/?file=746870#file746870line77
 
  we should probably bail here, if somehow the return is != 0  (isError() 
  || false)
 
 Ankur Chauhan wrote:
 My thinking here was: In case of a user that does not have hadoop_home 
 set and no hadoop in path, we would like to continue with other methods. If 
 we bail here for whatever reason (no hadoop, misconfigured hadoop etc), we 
 would end up breaking the fetcher for all those people. - This holds only if 
 we assume that hadoop/hdfs is **not** a hard dependency.

We should probably add a comment then to outline the implied assumption(s).


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59644
---


On Nov. 4, 2014, 1:04 a.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 1:04 a.m.)
 
 
 Review request for mesos and Timothy St. Clair.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Why rely on url scheme for fetching?

No problem. We non-English folks have complicated names :-)

On Nov 4, 2014, at 12:51 PM, Ankur Chauhan an...@malloc64.com wrote:

Hi,

Sorry about that bernd! :s/ben/bernd/g

-- ankur
Sent from my iPhone

On Nov 4, 2014, at 3:13 AM, Bernd Mathiske be...@mesosphere.io wrote:

Hi Ankur,

I am Bernd, not Ben, but I’ll try to do my best :-)

Your plan looks good to me and your patch for MESOS-1711 seems uncomplicated
enough to not cause me major problems no matter when it lands.

Bernd

On Nov 4, 2014, at 11:44 AM, Ankur Chauhan an...@malloc64.com wrote:

Hi ben,

Thanks for the follow up. I think thats a perfectly fine game plan. I think
I can break down things into smaller, more isolated chunks. But from the
looks of MESOS-1316 the invocation code and the testing code seems to
change a lot (in a good way), so to avoid a bunch of wasted cycles, I was
going to hold off a little till it is committed and then go on to
refactoring the transport parts. The json config change (MESOS-1248) is
completely disconnected to this so I am not too worried about that one.
The only place where some coordination may be required is the
launcher/fetcher.cpp. It seems there are common changes happening between
my change https://reviews.apache.org/r/27483/
https://reviews.apache.org/r/27483/https://reviews.apache.org/r/27483/
https://reviews.apache.org/r/27483/ and your
https://reviews.apache.org/r/21316
https://reviews.apache.org/r/21316https://reviews.apache.org/r/21316
https://reviews.apache.org/r/21316 which would mean a little rework but
it's totally within reasonable limits.
I think if you get the MESOS-1316 committed and I get MESOS-1711 (
https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483
https://reviews.apache.org/r/27483 https://reviews.apache.org/r/27483 )
in, both of us would get unblocked in a way that we can get a way that lets
us concurrently work on MESOS-336 and refactoring fetcher into more
testable parts.

Does this make sense to you or did I misunderstand some part?

-- Ankur

On 4 Nov 2014, at 02:05, Bernd Mathiske be...@mesosphere.io
mailto:be...@mesosphere.io wrote:

Typo: not - note

On Nov 4, 2014, at 10:59 AM, Bernd Mathiske be...@mesosphere.io
mailto:be...@mesosphere.io wrote:

Hi Ankur,

I like it, too. However, I cannot refrain from relaying (not my choice of
word here) to you the advice to break your relatively large patch down
into smaller parts. My patch for MESOS-336 certainly was, as I know now.
My plan is to get MESOS-1316 (which is now under review) and MESOS-1248
(which should be quick) committed and then rebase MESOS-336 in smaller
pieces. If you have small pieces, too, I do think we can do this somewhat
concurrently, as your refactoring seems to affect mostly how the
transport part works. I am on the other hand mostly interested in the
bookkeeping of what is in progress, what succeeded, what failed, what has
been put where, and so on, which can be separated, if we are careful
about it.

(BTW, I think we need both unit tests and integration tests. And we don’t
have nearly enough of either.)

Bernd

On Nov 3, 2014, at 6:18 PM, Adam Bordelon a...@mesosphere.io
mailto:a...@mesosphere.iomailto:a...@mesosphere.io
mailto:a...@mesosphere.io wrote:

+ Bernd, who has done some fetcher work, including additional testing,
for MESOS-1316, MESOS-1945, and MESOS-336

On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon dha...@twopensource.com
mailto:dha...@twopensource.commailto:dha...@twopensource.com
mailto:dha...@twopensource.com wrote:
Hi Ankur

On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com
mailto:an...@malloc64.commailto:an...@malloc64.com
mailto:an...@malloc64.com wrote:
Hi,

I did some learning today! This is pretty much a very rough draft of the
tests/refactor of mesos-fetcher that I have come up with. Again, If
there are some obvious mistakes, please let me know. (this is my first
pass after all).
https://github.com/ankurcha/mesos/compare/prefer_2
https://github.com/ankurcha/mesos/compare/prefer_2https://github.com/ankurcha/mesos/compare/prefer_2
https://github.com/ankurcha/mesos/compare/prefer_2

Review Request 27573: Update release guide to mention configuration.md.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27573/
---

Review request for mesos, Adam B and Ian Downes.


Repository: mesos-git


Description
---

To update the flags for master, slave and configure.


Diffs
-

  docs/release-guide.md 0255b4ae964a42e5cf57e33ea83d8e78785feb47 

Diff: https://reviews.apache.org/r/27573/diff/


Testing
---


Thanks,

Kapil Arya

Re: Review Request 27573: Update release guide to mention configuration.md.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27573/#review59772
---


Bad patch!

Reviews applied: [27573]

Failed command: ./support/apply-review.sh -n -r 27573

Error:
 2014-11-04 15:54:33 URL:https://reviews.apache.org/r/27573/diff/raw/ 
[1125/1125] - 27573.patch [1]
error: patch failed: docs/release-guide.md:25
error: docs/release-guide.md: patch does not apply
Failed to apply patch

- Mesos ReviewBot


On Nov. 4, 2014, 3:37 p.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27573/
 ---
 
 (Updated Nov. 4, 2014, 3:37 p.m.)
 
 
 Review request for mesos, Adam B and Ian Downes.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 To update the flags for master, slave and configure.
 
 
 Diffs
 -
 
   docs/release-guide.md 0255b4ae964a42e5cf57e33ea83d8e78785feb47 
 
 Diff: https://reviews.apache.org/r/27573/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27573: Update release guide to mention configuration.md.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27573/
---

(Updated Nov. 4, 2014, 11:07 a.m.)


Review request for mesos, Adam B and Ian Downes.


Changes
---

Rebased diff.


Repository: mesos-git


Description
---

To update the flags for master, slave and configure.


Diffs (updated)
-

  docs/release-guide.md d31c616199dac6bbcf8ffce4934a20d31063cb88 

Diff: https://reviews.apache.org/r/27573/diff/


Testing
---


Thanks,

Kapil Arya

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59775
---


Patch looks great!

Reviews applied: [27573, 27560, 27556]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 4:15 p.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 4, 2014, 4:15 p.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27573: Update release guide to mention configuration.md.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27573/
---

(Updated Nov. 4, 2014, 11:54 a.m.)


Review request for mesos, Adam B and Ian Downes.


Changes
---

Replaced single quotes with back quotes for configure.ac.


Repository: mesos-git


Description
---

To update the flags for master, slave and configure.


Diffs (updated)
-

  docs/release-guide.md d31c616199dac6bbcf8ffce4934a20d31063cb88 

Diff: https://reviews.apache.org/r/27573/diff/


Testing
---


Thanks,

Kapil Arya

Re: Review Request 27531: Update Master and Slave metrics to match task source and reason scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27531/
---

(Updated Nov. 4, 2014, 8:58 a.m.)


Review request for mesos, Tobias Weingartner and Vinod Kone.


Changes
---

rebased


Bugs: MESOS-1830
https://issues.apache.org/jira/browse/MESOS-1830


Repository: mesos-git


Description
---

Update metrics in Master to match the source and reason split for task statuses.


Diffs (updated)
-

  src/master/master.hpp 687f1789712dcd867b411badd85f4a12ae8f16d7 
  src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
  src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
  src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
  src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
  src/tests/slave_tests.cpp d2cbaf82391411bdc5d85327478d7ca8072048af 

Diff: https://reviews.apache.org/r/27531/diff/


Testing
---

make check
run master and check endpoint:

{
...
master/reconciliation_states/task_error: 0,
master/reconciliation_states/task_failed: 0,
master/reconciliation_states/task_finished: 0,
master/reconciliation_states/task_killed: 0,
master/reconciliation_states/task_lost: 0,
master/reconciliation_states/task_running: 0,
master/reconciliation_states/task_staging: 0,
master/reconciliation_states/task_starting: 0,
master/recovery_slave_removals: 0,
master/slave_registrations: 1,
master/slave_removals: 0,
master/slave_reregistrations: 0,
master/slaves_active: 1,
master/slaves_connected: 1,
master/slaves_disconnected: 0,
master/slaves_inactive: 0,
master/tasks_failed: 0,
master/tasks_finished: 0,
master/tasks_killed: 0,
master/tasks_lost: 0,
master/tasks_lost/reason_executor_terminated: 0,
master/tasks_lost/reason_executor_unregistered: 0,
master/tasks_lost/reason_framework_removed: 0,
master/tasks_lost/reason_gc_error: 0,
master/tasks_lost/reason_invalid_command: 0,
master/tasks_lost/reason_invalid_frameworkid: 0,
master/tasks_lost/reason_invalid_offers: 0,
master/tasks_lost/reason_master_disconnected: 0,
master/tasks_lost/reason_memory_limit: 0,
master/tasks_lost/reason_reconciliation: 0,
master/tasks_lost/reason_slave_disconnected: 0,
master/tasks_lost/reason_slave_removed: 0,
master/tasks_lost/reason_slave_restarted: 0,
master/tasks_lost/reason_slave_unknown: 0,
master/tasks_lost/reason_task_invalid: 0,
master/tasks_lost/reason_task_unauthorized: 0,
master/tasks_lost/reason_task_unknown: 0,
...
}

and slave:

{
...
slave/tasks_lost: 0,
slave/tasks_lost/reason_executor_terminated: 0,
slave/tasks_lost/reason_executor_unregistered: 0,
slave/tasks_lost/reason_framework_removed: 0,
slave/tasks_lost/reason_gc_error: 0,
slave/tasks_lost/reason_invalid_command: 0,
slave/tasks_lost/reason_invalid_frameworkid: 0,
slave/tasks_lost/reason_invalid_offers: 0,
slave/tasks_lost/reason_master_disconnected: 0,
slave/tasks_lost/reason_memory_limit: 0,
slave/tasks_lost/reason_reconciliation: 0,
slave/tasks_lost/reason_slave_disconnected: 0,
slave/tasks_lost/reason_slave_removed: 0,
slave/tasks_lost/reason_slave_restarted: 0,
slave/tasks_lost/reason_slave_unknown: 0,
slave/tasks_lost/reason_task_invalid: 0,
slave/tasks_lost/reason_task_unauthorized: 0,
slave/tasks_lost/reason_task_unknown: 0,
slave/tasks_running: 0,
slave/tasks_staging: 0,
slave/tasks_starting: 0,
...
}


Thanks,

Dominic Hamon

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/
---

(Updated Nov. 4, 2014, 8:58 a.m.)


Review request for mesos, Vinod Kone and Bill Farner.


Changes
---

rebased


Bugs: MESOS-1830 and MESOS-343
https://issues.apache.org/jira/browse/MESOS-1830
https://issues.apache.org/jira/browse/MESOS-343


Repository: mesos-git


Description
---

Added source and reason, enabled TASK_ERROR, and made the changes necessary 
throughout the codebase.


Diffs (updated)
-

  include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
  src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
  src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
  src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
  src/examples/java/TestFramework.java bc593d0abfacb00690b1492b2b82c970f4e4de6d 
  src/examples/low_level_scheduler_libprocess.cpp 
7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
  src/examples/low_level_scheduler_pthread.cpp 
6e233a10117a1c7aa669806b5b430e746e227ee5 
  src/examples/no_executor_framework.cpp 
f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
  src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
  src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
  src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
  src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
  src/tests/fault_tolerance_tests.cpp a18a41a3e34ff112e04e693447d757403e5013bd 
  src/tests/master_authorization_tests.cpp 
652e80d0d4567b225c6ffb326725ddfde06f7fd3 
  src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
  src/tests/resource_offers_tests.cpp fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 

Diff: https://reviews.apache.org/r/26382/diff/


Testing
---

make check


Thanks,

Dominic Hamon

Re: Review Request 27531: Update Master and Slave metrics to match task source and reason scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27531/#review59780
---


Patch looks great!

Reviews applied: [26817, 26382, 27531]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 4:58 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27531/
 ---
 
 (Updated Nov. 4, 2014, 4:58 p.m.)
 
 
 Review request for mesos, Tobias Weingartner and Vinod Kone.
 
 
 Bugs: MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-1830
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Update metrics in Master to match the source and reason split for task 
 statuses.
 
 
 Diffs
 -
 
   src/master/master.hpp 687f1789712dcd867b411badd85f4a12ae8f16d7 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/slave_tests.cpp d2cbaf82391411bdc5d85327478d7ca8072048af 
 
 Diff: https://reviews.apache.org/r/27531/diff/
 
 
 Testing
 ---
 
 make check
 run master and check endpoint:
 
 {
 ...
 master/reconciliation_states/task_error: 0,
 master/reconciliation_states/task_failed: 0,
 master/reconciliation_states/task_finished: 0,
 master/reconciliation_states/task_killed: 0,
 master/reconciliation_states/task_lost: 0,
 master/reconciliation_states/task_running: 0,
 master/reconciliation_states/task_staging: 0,
 master/reconciliation_states/task_starting: 0,
 master/recovery_slave_removals: 0,
 master/slave_registrations: 1,
 master/slave_removals: 0,
 master/slave_reregistrations: 0,
 master/slaves_active: 1,
 master/slaves_connected: 1,
 master/slaves_disconnected: 0,
 master/slaves_inactive: 0,
 master/tasks_failed: 0,
 master/tasks_finished: 0,
 master/tasks_killed: 0,
 master/tasks_lost: 0,
 master/tasks_lost/reason_executor_terminated: 0,
 master/tasks_lost/reason_executor_unregistered: 0,
 master/tasks_lost/reason_framework_removed: 0,
 master/tasks_lost/reason_gc_error: 0,
 master/tasks_lost/reason_invalid_command: 0,
 master/tasks_lost/reason_invalid_frameworkid: 0,
 master/tasks_lost/reason_invalid_offers: 0,
 master/tasks_lost/reason_master_disconnected: 0,
 master/tasks_lost/reason_memory_limit: 0,
 master/tasks_lost/reason_reconciliation: 0,
 master/tasks_lost/reason_slave_disconnected: 0,
 master/tasks_lost/reason_slave_removed: 0,
 master/tasks_lost/reason_slave_restarted: 0,
 master/tasks_lost/reason_slave_unknown: 0,
 master/tasks_lost/reason_task_invalid: 0,
 master/tasks_lost/reason_task_unauthorized: 0,
 master/tasks_lost/reason_task_unknown: 0,
 ...
 }
 
 and slave:
 
 {
 ...
 slave/tasks_lost: 0,
 slave/tasks_lost/reason_executor_terminated: 0,
 slave/tasks_lost/reason_executor_unregistered: 0,
 slave/tasks_lost/reason_framework_removed: 0,
 slave/tasks_lost/reason_gc_error: 0,
 slave/tasks_lost/reason_invalid_command: 0,
 slave/tasks_lost/reason_invalid_frameworkid: 0,
 slave/tasks_lost/reason_invalid_offers: 0,
 slave/tasks_lost/reason_master_disconnected: 0,
 slave/tasks_lost/reason_memory_limit: 0,
 slave/tasks_lost/reason_reconciliation: 0,
 slave/tasks_lost/reason_slave_disconnected: 0,
 slave/tasks_lost/reason_slave_removed: 0,
 slave/tasks_lost/reason_slave_restarted: 0,
 slave/tasks_lost/reason_slave_unknown: 0,
 slave/tasks_lost/reason_task_invalid: 0,
 slave/tasks_lost/reason_task_unauthorized: 0,
 slave/tasks_lost/reason_task_unknown: 0,
 slave/tasks_running: 0,
 slave/tasks_staging: 0,
 slave/tasks_starting: 0,
 ...
 }
 
 
 Thanks,
 
 Dominic Hamon

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/#review59716
---



include/mesos/mesos.proto
https://reviews.apache.org/r/26382/#comment101022

just looked at the code for command executor, and the only way the command 
executor exits with a pending task is if command executor itself is bad (bad 
launcher dir flag, missing shared library etc). it has nothing to with the user 
command being bad/invalid. 

given that, here are some options for what we could name the reason

REASON_FAILED_COMMAND_EXECUTOR
REASON_FAILED_COMMAND
REASON_EXECUTOR_TERMINATED # original one you had



src/slave/slave.cpp
https://reviews.apache.org/r/26382/#comment101021

I would suggest this:

if (termination.isReady()  termination.get().killed()) {
  taskState = TASK_FAILED;
  // TODO...
  reason = TaskStatus::REASON_MEMORY_LIMIT; 
} else if (executor-isCommandExecutor()) {
  taskState = TASK_FAILED;
  reason = TaskStatus::REASON_FAILED_COMMAND;
} else {
  taskState = TASK_LOST;
  reason = TaskStatus::REASON_EXECUTOR_TERMINATED;
}



src/slave/slave.cpp
https://reviews.apache.org/r/26382/#comment101020

Missed here?


- Vinod Kone


On Nov. 4, 2014, 4:58 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26382/
 ---
 
 (Updated Nov. 4, 2014, 4:58 p.m.)
 
 
 Review request for mesos, Vinod Kone and Bill Farner.
 
 
 Bugs: MESOS-1830 and MESOS-343
 https://issues.apache.org/jira/browse/MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-343
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Added source and reason, enabled TASK_ERROR, and made the changes necessary 
 throughout the codebase.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
   src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
   src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
   src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
   src/examples/java/TestFramework.java 
 bc593d0abfacb00690b1492b2b82c970f4e4de6d 
   src/examples/low_level_scheduler_libprocess.cpp 
 7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
   src/examples/low_level_scheduler_pthread.cpp 
 6e233a10117a1c7aa669806b5b430e746e227ee5 
   src/examples/no_executor_framework.cpp 
 f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
   src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/fault_tolerance_tests.cpp 
 a18a41a3e34ff112e04e693447d757403e5013bd 
   src/tests/master_authorization_tests.cpp 
 652e80d0d4567b225c6ffb326725ddfde06f7fd3 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/resource_offers_tests.cpp 
 fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 
 
 Diff: https://reviews.apache.org/r/26382/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Dominic Hamon

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.



 On Nov. 3, 2014, 9:59 p.m., Timothy St. Clair wrote:
  src/launcher/fetcher.cpp, line 77
  https://reviews.apache.org/r/27483/diff/1/?file=746870#file746870line77
 
  we should probably bail here, if somehow the return is != 0  (isError() 
  || false)
 
 Ankur Chauhan wrote:
 My thinking here was: In case of a user that does not have hadoop_home 
 set and no hadoop in path, we would like to continue with other methods. If 
 we bail here for whatever reason (no hadoop, misconfigured hadoop etc), we 
 would end up breaking the fetcher for all those people. - This holds only if 
 we assume that hadoop/hdfs is **not** a hard dependency.
 
 Timothy St. Clair wrote:
 We should probably add a comment then to outline the implied 
 assumption(s).

I have added a comment detailing this.


- Ankur


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59644
---


On Nov. 4, 2014, 1:04 a.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 1:04 a.m.)
 
 
 Review request for mesos and Timothy St. Clair.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/
---

(Updated Nov. 4, 2014, 6:25 p.m.)


Review request for mesos and Timothy St. Clair.


Changes
---

Add comments about hadoop client dependency.


Bugs: MESOS-1711
https://issues.apache.org/jira/browse/MESOS-1711


Repository: mesos-git


Description
---

Previously, the fetcher used a hardcoded list of schemes to determine what URIs 
could be fetched by hadoop (if available). This is now changed such that we 
first check if hadoop can fetch them for us and then we fallback to the os::net 
and then a local copy method (same as it used to be). This allows users to 
fetch artifacts from arbitrary filesystems as long as hadoop is correctly 
configured (in core-site.xml).


Diffs (updated)
-

  src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
  src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 

Diff: https://reviews.apache.org/r/27483/diff/


Testing
---

make check
sudo bin/mesos-tests.sh --verbose
support/mesos-style.py


Thanks,

Ankur Chauhan

Re: Review Request 27531: Update Master and Slave metrics to match task source and reason scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27531/#review59784
---



src/master/master.hpp
https://reviews.apache.org/r/27531/#comment101093

forgot to update this during reconciliation?

make sure to update the reconciliation tests to test the metric is non-zero.



src/master/master.hpp
https://reviews.apache.org/r/27531/#comment101086

s/reason/TASK_LOST reason/

also, add a TODO for reasons for TASK_KILLED and TASK_FAILED?

more importantly, have you thought about how would you extend this for 
TASK_KILLED and TASK_FAILED with the current design? if you use vectors we are 
back to the same problem of having most entries being zero.

how about using a hashmap intead?

hashmapTaskState, process::metrics::Counter  task_lost_reasons;

you could explicitly initialize it with task states that you would expect 
to exist for this reason. that way we don't have task states that are not 
expected for this reason.



src/master/master.cpp
https://reviews.apache.org/r/27531/#comment101084


s/++metrics.task_lost_reasons[status.reason()]/metrics.task_lost_reasons[status.reason()]++/
 

to be consistent.



src/master/master.cpp
https://reviews.apache.org/r/27531/#comment101085

s/task_state/state/



src/slave/slave.cpp
https://reviews.apache.org/r/27531/#comment101089

hmm.. this needs non local reasoning to understand. why not have 
terminateTask() take a TaskStatus instead of TaskState?



src/tests/master_tests.cpp
https://reviews.apache.org/r/27531/#comment101090

s/task_state/state/


- Vinod Kone


On Nov. 4, 2014, 4:58 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27531/
 ---
 
 (Updated Nov. 4, 2014, 4:58 p.m.)
 
 
 Review request for mesos, Tobias Weingartner and Vinod Kone.
 
 
 Bugs: MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-1830
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Update metrics in Master to match the source and reason split for task 
 statuses.
 
 
 Diffs
 -
 
   src/master/master.hpp 687f1789712dcd867b411badd85f4a12ae8f16d7 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/slave_tests.cpp d2cbaf82391411bdc5d85327478d7ca8072048af 
 
 Diff: https://reviews.apache.org/r/27531/diff/
 
 
 Testing
 ---
 
 make check
 run master and check endpoint:
 
 {
 ...
 master/reconciliation_states/task_error: 0,
 master/reconciliation_states/task_failed: 0,
 master/reconciliation_states/task_finished: 0,
 master/reconciliation_states/task_killed: 0,
 master/reconciliation_states/task_lost: 0,
 master/reconciliation_states/task_running: 0,
 master/reconciliation_states/task_staging: 0,
 master/reconciliation_states/task_starting: 0,
 master/recovery_slave_removals: 0,
 master/slave_registrations: 1,
 master/slave_removals: 0,
 master/slave_reregistrations: 0,
 master/slaves_active: 1,
 master/slaves_connected: 1,
 master/slaves_disconnected: 0,
 master/slaves_inactive: 0,
 master/tasks_failed: 0,
 master/tasks_finished: 0,
 master/tasks_killed: 0,
 master/tasks_lost: 0,
 master/tasks_lost/reason_executor_terminated: 0,
 master/tasks_lost/reason_executor_unregistered: 0,
 master/tasks_lost/reason_framework_removed: 0,
 master/tasks_lost/reason_gc_error: 0,
 master/tasks_lost/reason_invalid_command: 0,
 master/tasks_lost/reason_invalid_frameworkid: 0,
 master/tasks_lost/reason_invalid_offers: 0,
 master/tasks_lost/reason_master_disconnected: 0,
 master/tasks_lost/reason_memory_limit: 0,
 master/tasks_lost/reason_reconciliation: 0,
 master/tasks_lost/reason_slave_disconnected: 0,
 master/tasks_lost/reason_slave_removed: 0,
 master/tasks_lost/reason_slave_restarted: 0,
 master/tasks_lost/reason_slave_unknown: 0,
 master/tasks_lost/reason_task_invalid: 0,
 master/tasks_lost/reason_task_unauthorized: 0,
 master/tasks_lost/reason_task_unknown: 0,
 ...
 }
 
 and slave:
 
 {
 ...
 slave/tasks_lost: 0,
 slave/tasks_lost/reason_executor_terminated: 0,
 slave/tasks_lost/reason_executor_unregistered: 0,
 slave/tasks_lost/reason_framework_removed: 0,
 slave/tasks_lost/reason_gc_error: 0,
 slave/tasks_lost/reason_invalid_command: 0,
 slave/tasks_lost/reason_invalid_frameworkid: 0,
 slave/tasks_lost/reason_invalid_offers: 0,
 slave/tasks_lost/reason_master_disconnected: 0,
 slave/tasks_lost/reason_memory_limit: 0,
 slave/tasks_lost/reason_reconciliation: 0,
 slave/tasks_lost/reason_slave_disconnected: 0,

Re: Review Request 27558: Permit unprivileged executors to created child cgroups.



 On Nov. 4, 2014, 2:05 a.m., Vinod Kone wrote:
  src/tests/isolator_tests.cpp, line 1070
  https://reviews.apache.org/r/27558/diff/1/?file=747971#file747971line1070
 
  maybe a bigger sleep interval to account for slower machines?
 
 Ian Downes wrote:
 The parent is just reading /proc/PID/cgroup which is directly from the 
 kernel so I think 100 seconds is ample/excessive.

oops didn't realize it was seconds. nm.


- Vinod


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27558/#review59713
---


On Nov. 4, 2014, 1 a.m., Ian Downes wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27558/
 ---
 
 (Updated Nov. 4, 2014, 1 a.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-1941
 https://issues.apache.org/jira/browse/MESOS-1941
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 MesosContainerizer Isolators using cgroups will chown the cgroup to
 the executor user, if specified, enabling the executor to create child
 cgroups. Cgroup control files are not chowned so the executor cannot
 modify its own cgroups, only control files for child cgroups.
 
 
 Diffs
 -
 
   src/slave/containerizer/isolators/cgroups/cpushare.cpp 
 f9531e447bc2b39ee99d6fb0c8f911a70a3498a3 
   src/slave/containerizer/isolators/cgroups/mem.cpp 
 96bc506bb9dce7a455b443d9e24236babe91d890 
   src/slave/containerizer/isolators/cgroups/perf_event.cpp 
 7ed418a8e8c4a0a68715ffb13660e57afeaef69e 
   src/tests/isolator_tests.cpp 4d03f46cbe20c0e7aff15bab0a26d7738b55aab9 
 
 Diff: https://reviews.apache.org/r/27558/diff/
 
 
 Testing
 ---
 
 # New test added
 make check
 
 
 Thanks,
 
 Ian Downes

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/
---

(Updated Nov. 4, 2014, 10:41 a.m.)


Review request for mesos, Vinod Kone and Bill Farner.


Bugs: MESOS-1830 and MESOS-343
https://issues.apache.org/jira/browse/MESOS-1830
https://issues.apache.org/jira/browse/MESOS-343


Repository: mesos-git


Description
---

Added source and reason, enabled TASK_ERROR, and made the changes necessary 
throughout the codebase.


Diffs (updated)
-

  include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
  src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
  src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
  src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
  src/examples/java/TestFramework.java bc593d0abfacb00690b1492b2b82c970f4e4de6d 
  src/examples/low_level_scheduler_libprocess.cpp 
7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
  src/examples/low_level_scheduler_pthread.cpp 
6e233a10117a1c7aa669806b5b430e746e227ee5 
  src/examples/no_executor_framework.cpp 
f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
  src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
  src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
  src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
  src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
  src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
  src/tests/fault_tolerance_tests.cpp a18a41a3e34ff112e04e693447d757403e5013bd 
  src/tests/master_authorization_tests.cpp 
652e80d0d4567b225c6ffb326725ddfde06f7fd3 
  src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
  src/tests/resource_offers_tests.cpp fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 

Diff: https://reviews.apache.org/r/26382/diff/


Testing
---

make check


Thanks,

Dominic Hamon

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59798
---


Patch looks great!

Reviews applied: [27483]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 6:25 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 6:25 p.m.)
 
 
 Review request for mesos and Timothy St. Clair.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/#review59799
---



src/slave/slave.cpp
https://reviews.apache.org/r/26382/#comment101104

kill this?



src/slave/slave.cpp
https://reviews.apache.org/r/26382/#comment101108

hmm. sendExecutorTerminatedUpdate sounds like we are sending an update 
about executor terminated (e.g., ExitedExecutorMessage) which is not correct. 
can you come up with a better name? i can't think of any. alternatively don't 
pull this out into a function for now.


- Vinod Kone


On Nov. 4, 2014, 6:41 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26382/
 ---
 
 (Updated Nov. 4, 2014, 6:41 p.m.)
 
 
 Review request for mesos, Vinod Kone and Bill Farner.
 
 
 Bugs: MESOS-1830 and MESOS-343
 https://issues.apache.org/jira/browse/MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-343
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Added source and reason, enabled TASK_ERROR, and made the changes necessary 
 throughout the codebase.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
   src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
   src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
   src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
   src/examples/java/TestFramework.java 
 bc593d0abfacb00690b1492b2b82c970f4e4de6d 
   src/examples/low_level_scheduler_libprocess.cpp 
 7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
   src/examples/low_level_scheduler_pthread.cpp 
 6e233a10117a1c7aa669806b5b430e746e227ee5 
   src/examples/no_executor_framework.cpp 
 f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
   src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
   src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/fault_tolerance_tests.cpp 
 a18a41a3e34ff112e04e693447d757403e5013bd 
   src/tests/master_authorization_tests.cpp 
 652e80d0d4567b225c6ffb326725ddfde06f7fd3 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/resource_offers_tests.cpp 
 fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 
 
 Diff: https://reviews.apache.org/r/26382/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Dominic Hamon

Review Request 27585: Documentation for namespaces/pid isolator.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27585/
---

Review request for mesos and Vinod Kone.


Bugs: MESOS-2034
https://issues.apache.org/jira/browse/MESOS-2034


Repository: mesos-git


Description
---

Documentation for namespaces/pid isolator.


Diffs
-

  docs/mesos-containerizer.md PRE-CREATION 

Diff: https://reviews.apache.org/r/27585/diff/


Testing
---


Thanks,

Ian Downes

Review Request 27584: Documentation for filesystem/shared isolator.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27584/
---

Review request for mesos and Vinod Kone.


Bugs: MESOS-2033
https://issues.apache.org/jira/browse/MESOS-2033


Repository: mesos-git


Description
---

Documentation for filesystem/shared isolator.


Diffs
-

  docs/mesos-containerizer.md PRE-CREATION 

Diff: https://reviews.apache.org/r/27584/diff/


Testing
---


Thanks,

Ian Downes

Re: Review Request 27573: Update release guide to mention configuration.md.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27573/#review59804
---

Ship it!


Ship It!

- Ian Downes


On Nov. 4, 2014, 8:54 a.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27573/
 ---
 
 (Updated Nov. 4, 2014, 8:54 a.m.)
 
 
 Review request for mesos, Adam B and Ian Downes.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 To update the flags for master, slave and configure.
 
 
 Diffs
 -
 
   docs/release-guide.md d31c616199dac6bbcf8ffce4934a20d31063cb88 
 
 Diff: https://reviews.apache.org/r/27573/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

2014-11-04 Thread Timothy St. Clair


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59806
---

Ship it!


Ship It!

- Timothy St. Clair


On Nov. 4, 2014, 6:25 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 6:25 p.m.)
 
 
 Review request for mesos and Timothy St. Clair.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27585: Documentation for namespaces/pid isolator.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27585/#review59807
---

Ship it!


Ship It!

- Vinod Kone


On Nov. 4, 2014, 7:30 p.m., Ian Downes wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27585/
 ---
 
 (Updated Nov. 4, 2014, 7:30 p.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-2034
 https://issues.apache.org/jira/browse/MESOS-2034
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Documentation for namespaces/pid isolator.
 
 
 Diffs
 -
 
   docs/mesos-containerizer.md PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/27585/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Ian Downes

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59808
---


What's our policy on newlines between the options?

For the combined master / slave options there are newlines separating each but 
for the separate master and slave it's inconsistent. I see you've cleaned up 
some but there are a few remaining, e.g., after --credentials=VALUE.

I personally think a newline helps visibility but whatever we decide we should 
be consistent.

- Ian Downes


On Nov. 4, 2014, 8:15 a.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 4, 2014, 8:15 a.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27584: Documentation for filesystem/shared isolator.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27584/#review59809
---

Ship it!



docs/mesos-containerizer.md
https://reviews.apache.org/r/27584/#comment101113

s/POSIX systems/POSIX systems (e.g., OSX)/


- Vinod Kone


On Nov. 4, 2014, 7:30 p.m., Ian Downes wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27584/
 ---
 
 (Updated Nov. 4, 2014, 7:30 p.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-2033
 https://issues.apache.org/jira/browse/MESOS-2033
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Documentation for filesystem/shared isolator.
 
 
 Diffs
 -
 
   docs/mesos-containerizer.md PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/27584/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Ian Downes

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/
---

(Updated Nov. 4, 2014, 7:47 p.m.)


Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.


Changes
---

updating reviewers to reflect.


Bugs: MESOS-1711
https://issues.apache.org/jira/browse/MESOS-1711


Repository: mesos-git


Description
---

Previously, the fetcher used a hardcoded list of schemes to determine what URIs 
could be fetched by hadoop (if available). This is now changed such that we 
first check if hadoop can fetch them for us and then we fallback to the os::net 
and then a local copy method (same as it used to be). This allows users to 
fetch artifacts from arbitrary filesystems as long as hadoop is correctly 
configured (in core-site.xml).


Diffs
-

  src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
  src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 

Diff: https://reviews.apache.org/r/27483/diff/


Testing
---

make check
sudo bin/mesos-tests.sh --verbose
support/mesos-style.py


Thanks,

Ankur Chauhan

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.



 On Nov. 4, 2014, 2:42 p.m., Ian Downes wrote:
  What's our policy on newlines between the options?
  
  For the combined master / slave options there are newlines separating each 
  but for the separate master and slave it's inconsistent. I see you've 
  cleaned up some but there are a few remaining, e.g., after 
  --credentials=VALUE.
  
  I personally think a newline helps visibility but whatever we decide we 
  should be consistent.

The current one is pasted verbatim from the help messages.  I did see that 
there is some inconsistency in the help message output as well (new line after 
--credentials, --offer_timeout for master, and --containerizer_path, 
--default_container_image for slave).  I can create a separate review request 
to remove those newlines from the --help message.

Further, I also think that we should put newlines in the configuration.md file 
but not in the --help text.  Having said that, is there some place where we can 
put this policy?


- Kapil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59808
---


On Nov. 4, 2014, 11:15 a.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 4, 2014, 11:15 a.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Build failed in Jenkins: Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME #2246

2014-11-04 Thread Apache Jenkins Server

See 
https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/2246/changes

Changes:

[idownes] Pass user into Isolator::prepare.

[idownes] Permit unprivileged executors to created child cgroups.

--
[...truncated 70230 lines...]
I1104 20:02:14.921856  1008 sched.cpp:544] Scheduler::resourceOffers took 
68813ns
I1104 20:02:14.922345   979 master.cpp:693] Master terminating
I1104 20:02:14.922346  1000 sched.cpp:745] Stopping framework 
'20141104-200214-3142697795-34886-979-'
I1104 20:02:14.922390  1004 sched.cpp:745] Stopping framework 
'20141104-200214-3142697795-34886-979-0001'
I1104 20:02:14.923130   996 slave.cpp:2607] master@67.195.81.187:34886 exited
W1104 20:02:14.923166   996 slave.cpp:2610] Master disconnected! Waiting for a 
new master to be elected
I1104 20:02:14.928050   979 slave.cpp:484] Slave terminating
[   OK ] ResourceOffersTest.ResourcesGetReofferedWhenUnused (87 ms)
[ RUN  ] ResourceOffersTest.TaskUsesInvalidFrameworkID
Using temporary directory 
'/tmp/ResourceOffersTest_TaskUsesInvalidFrameworkID_iYa4FS'
I1104 20:02:14.934859   979 leveldb.cpp:176] Opened db in 2.493319ms
I1104 20:02:14.935820   979 leveldb.cpp:183] Compacted db in 924293ns
I1104 20:02:14.935858   979 leveldb.cpp:198] Created db iterator in 4878ns
I1104 20:02:14.935874   979 leveldb.cpp:204] Seeked to beginning of db in 880ns
I1104 20:02:14.935884   979 leveldb.cpp:273] Iterated through 0 keys in the db 
in 247ns
I1104 20:02:14.935916   979 replica.cpp:741] Replica recovered with log 
positions 0 - 0 with 1 holes and 0 unlearned
I1104 20:02:14.936547   995 recover.cpp:437] Starting replica recovery
I1104 20:02:14.936830   995 recover.cpp:463] Replica is in EMPTY status
I1104 20:02:14.938045  1006 replica.cpp:638] Replica in EMPTY status received a 
broadcasted recover request
I1104 20:02:14.938586  1008 recover.cpp:188] Received a recover response from a 
replica in EMPTY status
I1104 20:02:14.939023   996 recover.cpp:554] Updating replica status to STARTING
I1104 20:02:14.939787  1007 leveldb.cpp:306] Persisting metadata (8 bytes) to 
leveldb took 603583ns
I1104 20:02:14.939821  1007 replica.cpp:320] Persisted replica status to 
STARTING
I1104 20:02:14.940017  1007 recover.cpp:463] Replica is in STARTING status
I1104 20:02:14.940688  1008 master.cpp:318] Master 
20141104-200214-3142697795-34886-979 (pomona.apache.org) started on 
67.195.81.187:34886
I1104 20:02:14.940743  1008 master.cpp:364] Master only allowing authenticated 
frameworks to register
I1104 20:02:14.940767  1008 master.cpp:369] Master only allowing authenticated 
slaves to register
I1104 20:02:14.940801  1008 credentials.hpp:36] Loading credentials for 
authentication from 
'/tmp/ResourceOffersTest_TaskUsesInvalidFrameworkID_iYa4FS/credentials'
I1104 20:02:14.941066  1008 master.cpp:408] Authorization enabled
I1104 20:02:14.941437   998 replica.cpp:638] Replica in STARTING status 
received a broadcasted recover request
I1104 20:02:14.941586   998 master.cpp:126] No whitelist given. Advertising 
offers for all slaves
I1104 20:02:14.941694  1005 hierarchical_allocator_process.hpp:299] 
Initializing hierarchical allocator process with master : 
master@67.195.81.187:34886
I1104 20:02:14.941891  1000 recover.cpp:188] Received a recover response from a 
replica in STARTING status
I1104 20:02:14.942592   999 recover.cpp:554] Updating replica status to VOTING
I1104 20:02:14.942880  1008 master.cpp:1258] The newly elected leader is 
master@67.195.81.187:34886 with id 20141104-200214-3142697795-34886-979
I1104 20:02:14.942912  1008 master.cpp:1271] Elected as the leading master!
I1104 20:02:14.942926  1008 master.cpp:1089] Recovering from registrar
I1104 20:02:14.943236  1000 registrar.cpp:313] Recovering registrar
I1104 20:02:14.943564  1002 leveldb.cpp:306] Persisting metadata (8 bytes) to 
leveldb took 601985ns
I1104 20:02:14.943603  1002 replica.cpp:320] Persisted replica status to VOTING
I1104 20:02:14.943786  1008 recover.cpp:568] Successfully joined the Paxos group
I1104 20:02:14.944027  1008 recover.cpp:452] Recover process terminated
I1104 20:02:14.944527   997 log.cpp:656] Attempting to start the writer
I1104 20:02:14.945966  1003 replica.cpp:474] Replica received implicit promise 
request with proposal 1
I1104 20:02:14.94  1003 leveldb.cpp:306] Persisting metadata (8 bytes) to 
leveldb took 658974ns
I1104 20:02:14.946699  1003 replica.cpp:342] Persisted promised to 1
I1104 20:02:14.947763   995 coordinator.cpp:230] Coordinator attemping to fill 
missing position
I1104 20:02:14.949609   995 replica.cpp:375] Replica received explicit promise 
request for position 0 with proposal 2
I1104 20:02:14.950023   995 leveldb.cpp:343] Persisting action (8 bytes) to 
leveldb took 369984ns
I1104 20:02:14.950052   995 replica.cpp:676] Persisted action at 0
I1104 20:02:14.951216   999 replica.cpp:508] Replica received write request for 
position 0
I1104 20:02:14.951272   999 leveldb.cpp:438

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

2014-11-04 Thread Timothy Chen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59819
---

Ship it!


Ship It!


src/launcher/fetcher.cpp
https://reviews.apache.org/r/27483/#comment101116

We cap comments at 70 char width, but I think we can fix this up before 
merging it.


- Timothy Chen


On Nov. 4, 2014, 7:47 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 7:47 p.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

2014-11-04 Thread Timothy Chen



 On Nov. 4, 2014, 8:07 p.m., Timothy Chen wrote:
  Ship It!

Vinod you have any more comments?


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59819
---


On Nov. 4, 2014, 7:47 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 7:47 p.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27585: Documentation for namespaces/pid isolator.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27585/#review59826
---


Patch looks great!

Reviews applied: [27584, 27585]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 7:30 p.m., Ian Downes wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27585/
 ---
 
 (Updated Nov. 4, 2014, 7:30 p.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-2034
 https://issues.apache.org/jira/browse/MESOS-2034
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Documentation for namespaces/pid isolator.
 
 
 Diffs
 -
 
   docs/mesos-containerizer.md PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/27585/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Ian Downes

Re: Review Request 26382: Add source and reason to TaskStatus.



 On Nov. 4, 2014, 11:24 a.m., Vinod Kone wrote:
  src/slave/slave.cpp, line 2959
  https://reviews.apache.org/r/26382/diff/13-14/?file=748368#file748368line2959
 
  hmm. sendExecutorTerminatedUpdate sounds like we are sending an 
  update about executor terminated (e.g., ExitedExecutorMessage) which is not 
  correct. can you come up with a better name? i can't think of any. 
  alternatively don't pull this out into a function for now.

pulling it out into a function is important in this case (and something we 
should do more). I changed the name to something i think is better. feel free 
to bikeshed :)


- Dominic


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/#review59799
---


On Nov. 4, 2014, 12:15 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26382/
 ---
 
 (Updated Nov. 4, 2014, 12:15 p.m.)
 
 
 Review request for mesos, Vinod Kone and Bill Farner.
 
 
 Bugs: MESOS-1830 and MESOS-343
 https://issues.apache.org/jira/browse/MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-343
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Added source and reason, enabled TASK_ERROR, and made the changes necessary 
 throughout the codebase.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
   src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
   src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
   src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
   src/examples/java/TestFramework.java 
 bc593d0abfacb00690b1492b2b82c970f4e4de6d 
   src/examples/low_level_scheduler_libprocess.cpp 
 7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
   src/examples/low_level_scheduler_pthread.cpp 
 6e233a10117a1c7aa669806b5b430e746e227ee5 
   src/examples/no_executor_framework.cpp 
 f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
   src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
   src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/fault_tolerance_tests.cpp 
 a18a41a3e34ff112e04e693447d757403e5013bd 
   src/tests/master_authorization_tests.cpp 
 652e80d0d4567b225c6ffb326725ddfde06f7fd3 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/resource_offers_tests.cpp 
 fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 
 
 Diff: https://reviews.apache.org/r/26382/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Dominic Hamon

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/
---

(Updated Nov. 4, 2014, 12:15 p.m.)


Review request for mesos, Vinod Kone and Bill Farner.


Bugs: MESOS-1830 and MESOS-343
https://issues.apache.org/jira/browse/MESOS-1830
https://issues.apache.org/jira/browse/MESOS-343


Repository: mesos-git


Description
---

Added source and reason, enabled TASK_ERROR, and made the changes necessary 
throughout the codebase.


Diffs (updated)
-

  include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
  src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
  src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
  src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
  src/examples/java/TestFramework.java bc593d0abfacb00690b1492b2b82c970f4e4de6d 
  src/examples/low_level_scheduler_libprocess.cpp 
7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
  src/examples/low_level_scheduler_pthread.cpp 
6e233a10117a1c7aa669806b5b430e746e227ee5 
  src/examples/no_executor_framework.cpp 
f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
  src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
  src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
  src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
  src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
  src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
  src/tests/fault_tolerance_tests.cpp a18a41a3e34ff112e04e693447d757403e5013bd 
  src/tests/master_authorization_tests.cpp 
652e80d0d4567b225c6ffb326725ddfde06f7fd3 
  src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
  src/tests/resource_offers_tests.cpp fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 

Diff: https://reviews.apache.org/r/26382/diff/


Testing
---

make check


Thanks,

Dominic Hamon

Re: Review Request 26382: Add source and reason to TaskStatus.



 On Nov. 4, 2014, 9:46 a.m., Vinod Kone wrote:
  include/mesos/mesos.proto, line 640
  https://reviews.apache.org/r/26382/diff/11-12/?file=746438#file746438line640
 
  just looked at the code for command executor, and the only way the 
  command executor exits with a pending task is if command executor itself is 
  bad (bad launcher dir flag, missing shared library etc). it has nothing to 
  with the user command being bad/invalid. 
  
  given that, here are some options for what we could name the reason
  
  REASON_FAILED_COMMAND_EXECUTOR
  REASON_FAILED_COMMAND
  REASON_EXECUTOR_TERMINATED # original one you had

REASON_COMMAND_EXECUTOR_FAILED to make it clear it was the command executor.


 On Nov. 4, 2014, 9:46 a.m., Vinod Kone wrote:
  src/slave/slave.cpp, lines 2969-2974
  https://reviews.apache.org/r/26382/diff/11-12/?file=746449#file746449line2969
 
  Missed here?

yep, missed. copy paste code makes me sad. Pulling out into a private method.


- Dominic


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/#review59716
---


On Nov. 4, 2014, 12:15 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26382/
 ---
 
 (Updated Nov. 4, 2014, 12:15 p.m.)
 
 
 Review request for mesos, Vinod Kone and Bill Farner.
 
 
 Bugs: MESOS-1830 and MESOS-343
 https://issues.apache.org/jira/browse/MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-343
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Added source and reason, enabled TASK_ERROR, and made the changes necessary 
 throughout the codebase.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
   src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
   src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
   src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
   src/examples/java/TestFramework.java 
 bc593d0abfacb00690b1492b2b82c970f4e4de6d 
   src/examples/low_level_scheduler_libprocess.cpp 
 7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
   src/examples/low_level_scheduler_pthread.cpp 
 6e233a10117a1c7aa669806b5b430e746e227ee5 
   src/examples/no_executor_framework.cpp 
 f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
   src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
   src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/fault_tolerance_tests.cpp 
 a18a41a3e34ff112e04e693447d757403e5013bd 
   src/tests/master_authorization_tests.cpp 
 652e80d0d4567b225c6ffb326725ddfde06f7fd3 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/resource_offers_tests.cpp 
 fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 
 
 Diff: https://reviews.apache.org/r/26382/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Dominic Hamon

Re: Review Request 26736: Ensure docker calls stop if a docker_stop_timeout is provided Added the flag docker_stop_timeout that defaults to 0, if the timeout is 0 the a docker kill will be run, otherw

2014-11-04 Thread Ryan Thomas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26736/
---

(Updated Nov. 4, 2014, 9:28 p.m.)


Review request for mesos.


Bugs: MESOS-1925
https://issues.apache.org/jira/browse/MESOS-1925


Repository: mesos-git


Description
---

Ensure docker calls stop if a docker_stop_timeout is provided Added the flag 
docker_stop_timeout that defaults to 0, if the timeout is 0 the a docker kill 
will be run, otherwise a docker stop with the timeout in seconds is used. A 
negative value will result in a Failure.


Diffs
-

  src/docker/docker.hpp 2dc692c51d6f9d45d8d7c59f8678991fd80c6e89 
  src/docker/docker.cpp 0c0a1bfd3c1260ebdf342d0a6e0e04d7b6a111a4 
  src/slave/containerizer/docker.cpp a6689203adbdcb0ad12583389eaeb83329e4ef6b 
  src/slave/flags.hpp 319c002c66212ffb11759dbcc39f7661203a3e4e 
  src/tests/docker_containerizer_tests.cpp 
9d4ccc57f58d61c62aab5cdc79a129e987920bf6 
  src/tests/docker_tests.cpp ff06a01d7f20656c05bcf5afd3bc10059dc5378d 

Diff: https://reviews.apache.org/r/26736/diff/


Testing
---

make check
sudo bin/mesos-tests.sh --gtest_filter=DOCKER --verbose 
support/mesos-style.py


Thanks,

Ryan Thomas

Jenkins build is back to normal : Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME #2247

2014-11-04 Thread Apache Jenkins Server

See 
https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/2247/changes

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/#review59841
---

Ship it!



src/slave/slave.cpp
https://reviews.apache.org/r/26382/#comment101125

2 blank lines.


- Vinod Kone


On Nov. 4, 2014, 8:15 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26382/
 ---
 
 (Updated Nov. 4, 2014, 8:15 p.m.)
 
 
 Review request for mesos, Vinod Kone and Bill Farner.
 
 
 Bugs: MESOS-1830 and MESOS-343
 https://issues.apache.org/jira/browse/MESOS-1830
 https://issues.apache.org/jira/browse/MESOS-343
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Added source and reason, enabled TASK_ERROR, and made the changes necessary 
 throughout the codebase.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
   src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
   src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
   src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
   src/examples/java/TestFramework.java 
 bc593d0abfacb00690b1492b2b82c970f4e4de6d 
   src/examples/low_level_scheduler_libprocess.cpp 
 7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
   src/examples/low_level_scheduler_pthread.cpp 
 6e233a10117a1c7aa669806b5b430e746e227ee5 
   src/examples/no_executor_framework.cpp 
 f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
   src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
   src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
   src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
   src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
   src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
   src/tests/fault_tolerance_tests.cpp 
 a18a41a3e34ff112e04e693447d757403e5013bd 
   src/tests/master_authorization_tests.cpp 
 652e80d0d4567b225c6ffb326725ddfde06f7fd3 
   src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
   src/tests/resource_offers_tests.cpp 
 fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 
 
 Diff: https://reviews.apache.org/r/26382/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Dominic Hamon

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59842
---



src/launcher/fetcher.cpp
https://reviews.apache.org/r/27483/#comment101128

Why fall through if result.isError()?


- Vinod Kone


On Nov. 4, 2014, 7:47 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 7:47 p.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27555: Refactored the C++ Resources class to support persistent disk resources.

2014-11-04 Thread Jie Yu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27555/
---

(Updated Nov. 4, 2014, 10:01 p.m.)


Review request for mesos, Ben Mahler and Vinod Kone.


Changes
---

BenM's suggestion on the 'remove' semantics.


Bugs: MESOS-1974
https://issues.apache.org/jira/browse/MESOS-1974


Repository: mesos-git


Description
---

The purpose of the refactor is to support persistent disk resources.

Here are the main things I've done in this refactor:
1) Resource objects in Resources are stored in minimal format 
(validated/non-zero). That allows us to kill isAllocatable, allocatable, 
isZero, etc.
2) 'matches' needs to be split into two pieces: one for combining and one for 
removing, in order to support persitent disk resource. For example, one cannot 
combine two Resource object with DiskInfo (it's like two disks), however, you 
can do removal if they are identical.
3) Some of the interfaces are not intuitive (e.g., =, see details in the 
ticket). I removed them in favor of more explicit interfaces.
4) Unified all the validation code.
5) Adjusted the tests accordingly.


Diffs (updated)
-

  include/mesos/resources.hpp 0e37170 
  src/cli/execute.cpp ddaa20d 
  src/common/resources.cpp e9a0c85 
  src/examples/low_level_scheduler_libprocess.cpp 7ef5ea7 
  src/examples/low_level_scheduler_pthread.cpp 6e233a1 
  src/examples/no_executor_framework.cpp f98a073 
  src/examples/test_framework.cpp 187a611 
  src/master/drf_sorter.cpp 5464900 
  src/master/hierarchical_allocator_process.hpp 31dfb2c 
  src/master/http.cpp 3189933 
  src/master/master.cpp d914786 
  src/tests/allocator_tests.cpp 58e15aa 
  src/tests/gc_tests.cpp f7747e2 
  src/tests/master_tests.cpp 2e52574 
  src/tests/mesos.hpp c1d64a7 
  src/tests/resource_offers_tests.cpp fe66432 
  src/tests/resources_tests.cpp 3e50889 
  src/tests/slave_recovery_tests.cpp 98e059f 
  src/tests/sorter_tests.cpp 0516ab5 

Diff: https://reviews.apache.org/r/27555/diff/


Testing
---

make check


Thanks,

Jie Yu

Re: Unable to install Mesos on Ubuntu 14.04. Error during 'make'

2014-11-04 Thread Sweta Rani

Thanks for your help. I have installed mesos but  unable to start mesos
master by the command
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos.

The file exists as mesos-master.sh.in in the bin folder.

It reports:
Failed to create work directory '/var/lib/mesos': Permission denied

*Sweta Rani*

On Tue, Nov 4, 2014 at 2:50 PM, Adam Bordelon a...@mesosphere.io wrote:

 As an alternate workaround, if you don't need the Java support (you're not
 running a JVM framework like Spark/Hadoop/Marathon), then you could use
 `configure --disable-java` to skip all the maven/pom build steps.

 By disabling Java, you could at least check that the rest of Mesos builds
 and runs fine, which would still work with C++, Python, or Go frameworks,
 or even any Java frameworks using the pure Java bindings (jesos).

 On Mon, Nov 3, 2014 at 2:01 PM, Joris Van Remoortere jo...@mesosphere.io
 wrote:

  Did you verify that the settings.xml file was parsed using mvn -X?
 
  Specifically after the first few lines you will see something along the
  lines of:
  [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
  [DEBUG] Reading user settings from /home/abc/.m2/settings.xml
  [DEBUG] Using local repository at /home/abc/.m2/repository
 
  Make sure your settings file is in the right location.
 
  On Mon, Nov 3, 2014 at 11:51 AM, Sweta Rani swetarani3...@gmail.com
  wrote:
 
   I use campus proxy configuration for internet. But I am unable to run
  make
   in installing Mesos on my Ubuntu -14.04, as it ends up with ERROR(1).
  Upon
   searching, I created settings.xml file inside .m2 with the required
   configuration but failed again. Please help me out, as it is greatly
   hindering me from my work.
  
   On Sat Nov 01 2014 at 3:38:57 AM Joris Van Remoortere 
  jo...@mesosphere.io
   
   wrote:
  
I think this suggests you're fetching through a proxy:
*Proxy request sent, awaiting response... 200 OK*
   
When I wget this is my output:
 wget http://repo.maven.apache.org/maven2/org/apache/apache/11/
apache-11.pom
--2014-10-31

  
 
 http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom--2014-10-31
   
15:05:32--
   
 http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
Resolving repo.maven.apache.org (repo.maven.apache.org)...
  23.235.47.215
Connecting to repo.maven.apache.org
(repo.maven.apache.org)|23.235.47.215|:80...
connected.
*HTTP request sent, awaiting response... 200 OK*
   
I think you might need to set up your proxy along these lines:
http://maven.apache.org/guides/mini/guide-proxies.html ... but I am
  not
familiar with Maven.
   
Joris
   
   
On Fri, Oct 31, 2014 at 2:36 PM, Sweta Rani swetarani3...@gmail.com
 
wrote:
   
 Tried wget for all the given https, it was successful but still
 after
that
 make was not successful and make check says YOU HAVE 3 DISABLED
  TESTS

 proxima@Centauri:~/mesos/build$ wget

  http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
 --2014-11-01 02:55:51--

  http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
 Connecting to 10.3.100.207:8080... connected.
 Proxy request sent, awaiting response... 200 OK
 Length: 14811 (14K) [text/xml]
 Saving to: ‘apache-11.pom’

  *Joris Van Remoortere*

 On Sat Nov 01 2014 at 2:36:27 AM Joris Van Remoortere 
jo...@mesosphere.io
 
 wrote:

  That seems like you might have a connectivity / firewall issue.
  Can you try just doing a wget on that artifact?
 
  On Fri, Oct 31, 2014 at 1:56 PM, Sweta Rani 
  swetarani3...@gmail.com
   
  wrote:
 
   I tried it again but failed with following errors :
  
   make[1]: Entering directory `/home/proxima/mesos/build/src'
   Building mesos-0.21.0.jar ...
   [INFO] Scanning for projects...
   Downloading:
   http://repo.maven.apache.org/maven2/org/apache/apache/11/
apache-11.pom
   [ERROR] The build could not read 1 project - [Help 1]
   [ERROR]
   [ERROR]   The project org.apache.mesos:mesos:0.21.0
   (/home/proxima/mesos/build/src/java/mesos.pom) has 1 error
   [ERROR] Non-resolvable parent POM: Could not transfer
  artifact
   org.apache:apache:pom:11 from/to central (
   http://repo.maven.apache.org/maven2): Connection to
   http://repo.maven.apache.org refused and 'parent.relativePath'
points
 at
   wrong local POM @ line 18, column 11: Connection refused -
 [Help
   2]
   [ERROR]
   [ERROR] To see the full stack trace of the errors, re-run Maven
   with
 the
  -e
   switch.
   [ERROR] Re-run Maven using the -X switch to enable full debug
logging.
   [ERROR]
   [ERROR] For more information about the errors and possible
   solutions,
   please read the following articles:
   [ERROR] [Help 1]

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.



 On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote:
  src/launcher/fetcher.cpp, line 223
  https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223
 
  Why fall through if result.isError()?

The way I am thinking of the logic of fetchers is on the lines of we try all 
fetchers one by one till we succeed *even* if an unsuccessful fetcher can 
handle it. This means that if for any reason (maybe mesos-fetcher is running 
doesn't have permissions or whatever be the case) a fetcher fails we always go 
to the next one. I understand that a case can be made to do something on the 
lines of 

```
if(local.canHandle(uri)) {
  return local.fetch(uri);
} else if (os_net.canHandle(uri)) {
  return os_net.fetch(uri);
} else if ( hadoop.canHandle(uri)) {
  return hadoop.fetch(uri);
} else {
  return Error(Can't download uri: + uri);
}
```

But this way we are assuming that local fetcher has all possible permissions to 
handle local/file uris *and* os_net has all the configuration to handle all 
http(s) | ftp/ftp(s) uris and takes away the fallback. It may be the case that 
the hadoop fetcher (and hence hadoop client) has some credentials in its config 
files that aren't present in the urls themselves. An example of this is if ftp 
uris are provided without user or password but core-site.xml has the required 
user/password in it.

Does this answer your concern?


- Ankur


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59842
---


On Nov. 4, 2014, 7:47 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 7:47 p.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 26382: Add source and reason to TaskStatus.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26382/
---

(Updated Nov. 4, 2014, 2:19 p.m.)


Review request for mesos, Vinod Kone and Bill Farner.


Changes
---

changed bugs


Bugs: MESOS-1143 and MESOS-343
https://issues.apache.org/jira/browse/MESOS-1143
https://issues.apache.org/jira/browse/MESOS-343


Repository: mesos-git


Description
---

Added source and reason, enabled TASK_ERROR, and made the changes necessary 
throughout the codebase.


Diffs
-

  include/mesos/mesos.proto 397d542d93962e857d800cf0d47252b592393326 
  src/common/protobuf_utils.hpp 212d5124b9a4cc58e61719fa7f07a61cd166e834 
  src/common/protobuf_utils.cpp a9b65e328c4c62bff7fbf5633dda25d742d79019 
  src/examples/balloon_framework.cpp b05d5679fe2915142907af0b2dc00c6cd76eb9c1 
  src/examples/java/TestFramework.java bc593d0abfacb00690b1492b2b82c970f4e4de6d 
  src/examples/low_level_scheduler_libprocess.cpp 
7ef5ea78ade4ed856b97009fdfe31281f0a55c17 
  src/examples/low_level_scheduler_pthread.cpp 
6e233a10117a1c7aa669806b5b430e746e227ee5 
  src/examples/no_executor_framework.cpp 
f98a0735b9f287e7f1bf98af6c2e9a47ca6a77b2 
  src/examples/test_framework.cpp 187a611ebfe35cb13ee48aa5eca934cf55f34dea 
  src/master/master.cpp d914786a2517677ee7dd4a3130b4b191ef416c26 
  src/sched/sched.cpp d84465ca8f61a079fffbc013e3f9302be9f87487 
  src/slave/slave.hpp 6c183f827ec2521052f8b088d4583f13e661db2c 
  src/slave/slave.cpp b8935173bae9c124b8d08db590893334d7a45a23 
  src/tests/fault_tolerance_tests.cpp a18a41a3e34ff112e04e693447d757403e5013bd 
  src/tests/master_authorization_tests.cpp 
652e80d0d4567b225c6ffb326725ddfde06f7fd3 
  src/tests/master_tests.cpp 2e525749247626c05efb2f54a707599facb114b6 
  src/tests/resource_offers_tests.cpp fe66432b1bf75ee25feb73c4bb353e4d4e5b503f 

Diff: https://reviews.apache.org/r/26382/diff/


Testing
---

make check


Thanks,

Dominic Hamon

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/
---

(Updated Nov. 4, 2014, 10:20 p.m.)


Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.


Changes
---

Reformat comments to be 70 cols width


Bugs: MESOS-1711
https://issues.apache.org/jira/browse/MESOS-1711


Repository: mesos-git


Description
---

Previously, the fetcher used a hardcoded list of schemes to determine what URIs 
could be fetched by hadoop (if available). This is now changed such that we 
first check if hadoop can fetch them for us and then we fallback to the os::net 
and then a local copy method (same as it used to be). This allows users to 
fetch artifacts from arbitrary filesystems as long as hadoop is correctly 
configured (in core-site.xml).


Diffs (updated)
-

  src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
  src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 

Diff: https://reviews.apache.org/r/27483/diff/


Testing
---

make check
sudo bin/mesos-tests.sh --verbose
support/mesos-style.py


Thanks,

Ankur Chauhan

Re: Unable to install Mesos on Ubuntu 14.04. Error during 'make'

2014-11-04 Thread Joris Van Remoortere

You need to make the directory and set the permissions to the user you are
running the master as.

For example:

sudo mkdir /var/lib/mesos
sudo chown jmlvanre /var/lib/mesos

[jmlvanre] run mesos master...

On Tue, Nov 4, 2014 at 2:07 PM, Sweta Rani swetarani3...@gmail.com wrote:

 Thanks for your help. I have installed mesos but  unable to start mesos
 master by the command
 ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos.

 The file exists as mesos-master.sh.in in the bin folder.

 It reports:
 Failed to create work directory '/var/lib/mesos': Permission denied

 *Sweta Rani*

 On Tue, Nov 4, 2014 at 2:50 PM, Adam Bordelon a...@mesosphere.io wrote:

  As an alternate workaround, if you don't need the Java support (you're
 not
  running a JVM framework like Spark/Hadoop/Marathon), then you could use
  `configure --disable-java` to skip all the maven/pom build steps.
 
  By disabling Java, you could at least check that the rest of Mesos builds
  and runs fine, which would still work with C++, Python, or Go frameworks,
  or even any Java frameworks using the pure Java bindings (jesos).
 
  On Mon, Nov 3, 2014 at 2:01 PM, Joris Van Remoortere 
 jo...@mesosphere.io
  wrote:
 
   Did you verify that the settings.xml file was parsed using mvn -X?
  
   Specifically after the first few lines you will see something along the
   lines of:
   [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
   [DEBUG] Reading user settings from /home/abc/.m2/settings.xml
   [DEBUG] Using local repository at /home/abc/.m2/repository
  
   Make sure your settings file is in the right location.
  
   On Mon, Nov 3, 2014 at 11:51 AM, Sweta Rani swetarani3...@gmail.com
   wrote:
  
I use campus proxy configuration for internet. But I am unable to run
   make
in installing Mesos on my Ubuntu -14.04, as it ends up with ERROR(1).
   Upon
searching, I created settings.xml file inside .m2 with the required
configuration but failed again. Please help me out, as it is greatly
hindering me from my work.
   
On Sat Nov 01 2014 at 3:38:57 AM Joris Van Remoortere 
   jo...@mesosphere.io

wrote:
   
 I think this suggests you're fetching through a proxy:
 *Proxy request sent, awaiting response... 200 OK*

 When I wget this is my output:
  wget http://repo.maven.apache.org/maven2/org/apache/apache/11/
 apache-11.pom
 --2014-10-31
 
   
  
 
 http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom--2014-10-31

 15:05:32--

  http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
 Resolving repo.maven.apache.org (repo.maven.apache.org)...
   23.235.47.215
 Connecting to repo.maven.apache.org
 (repo.maven.apache.org)|23.235.47.215|:80...
 connected.
 *HTTP request sent, awaiting response... 200 OK*

 I think you might need to set up your proxy along these lines:
 http://maven.apache.org/guides/mini/guide-proxies.html ... but I
 am
   not
 familiar with Maven.

 Joris


 On Fri, Oct 31, 2014 at 2:36 PM, Sweta Rani 
 swetarani3...@gmail.com
  
 wrote:

  Tried wget for all the given https, it was successful but still
  after
 that
  make was not successful and make check says YOU HAVE 3 DISABLED
   TESTS
 
  proxima@Centauri:~/mesos/build$ wget
 
   http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
  --2014-11-01 02:55:51--
 
   http://repo.maven.apache.org/maven2/org/apache/apache/11/apache-11.pom
  Connecting to 10.3.100.207:8080... connected.
  Proxy request sent, awaiting response... 200 OK
  Length: 14811 (14K) [text/xml]
  Saving to: ‘apache-11.pom’
 
   *Joris Van Remoortere*
 
  On Sat Nov 01 2014 at 2:36:27 AM Joris Van Remoortere 
 jo...@mesosphere.io
  
  wrote:
 
   That seems like you might have a connectivity / firewall issue.
   Can you try just doing a wget on that artifact?
  
   On Fri, Oct 31, 2014 at 1:56 PM, Sweta Rani 
   swetarani3...@gmail.com

   wrote:
  
I tried it again but failed with following errors :
   
make[1]: Entering directory `/home/proxima/mesos/build/src'
Building mesos-0.21.0.jar ...
[INFO] Scanning for projects...
Downloading:
http://repo.maven.apache.org/maven2/org/apache/apache/11/
 apache-11.pom
[ERROR] The build could not read 1 project - [Help 1]
[ERROR]
[ERROR]   The project org.apache.mesos:mesos:0.21.0
(/home/proxima/mesos/build/src/java/mesos.pom) has 1 error
[ERROR] Non-resolvable parent POM: Could not transfer
   artifact
org.apache:apache:pom:11 from/to central (
http://repo.maven.apache.org/maven2): Connection to
http://repo.maven.apache.org refused and
 'parent.relativePath'
 points
  at
wrong local POM @ line 18, column 11:

Review Request 27595: Add MESOS-343 and MESOS-1143 to CHANGELOG


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27595/
---

Review request for mesos and Vinod Kone.


Repository: mesos-git


Description
---

see summary


Diffs
-

  CHANGELOG 55d8ff4af8bda37a47d31ee62bc424ef496311c4 

Diff: https://reviews.apache.org/r/27595/diff/


Testing
---


Thanks,

Dominic Hamon

Re: Review Request 27595: Add MESOS-343 and MESOS-1143 to CHANGELOG


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27595/#review59856
---

Ship it!



CHANGELOG
https://reviews.apache.org/r/27595/#comment101157

period or colon at the end.


- Vinod Kone


On Nov. 4, 2014, 10:24 p.m., Dominic Hamon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27595/
 ---
 
 (Updated Nov. 4, 2014, 10:24 p.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 see summary
 
 
 Diffs
 -
 
   CHANGELOG 55d8ff4af8bda37a47d31ee62bc424ef496311c4 
 
 Diff: https://reviews.apache.org/r/27595/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dominic Hamon

Review Request 27588: Remove newlines from master and slave help messages.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27588/
---

Review request for mesos.


Repository: mesos-git


Description
---

Also fixed help message for master's credential flag.


Diffs
-

  src/master/flags.hpp 1cea50c02f3ad7de1e1ae91d65d1accdb9af7b03 
  src/slave/flags.hpp efbd35dd744d8c8a53c36dd5cdb8d1c50fcfec3a 

Diff: https://reviews.apache.org/r/27588/diff/


Testing
---

make check


Thanks,

Kapil Arya

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/
---

(Updated Nov. 4, 2014, 5:33 p.m.)


Review request for mesos, Ian Downes and Till Toenshoff.


Changes
---

Created html table for options.


Bugs: MESOS-2037
https://issues.apache.org/jira/browse/MESOS-2037


Repository: mesos-git


Description (updated)
---

Copied the output of --help messages from configure, master, and slave.

The option tables are now in html format.

Also removed the explanation of the default configure flags.


Diffs (updated)
-

  docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 

Diff: https://reviews.apache.org/r/27556/diff/


Testing
---

Pushed and tested output on github at:
https://github.com/karya0/mesos/blob/modules/docs/configuration.md


Thanks,

Kapil Arya

TaskStatus source and reason fields

As per MESOS-343 https://issues.apache.org/jira/browse/MESOS-343, the
TaskStatus now contains source and reason fields. These are set to allow
frameworks to differentiate between status updates that come from the
Master, Slave, or Executor, and to get more information in a programmatic
way about the update. This will be part of the 0.21.0 release.

As a further enhancement, TASK_ERROR has been added (see MESOS-1143
https://issues.apache.org/jira/browse/MESOS-1143) as part of 0.21.0. It
is not currently being sent, but will be in the next release (0.22.0) to
allow time for frameworks to support it.

The semantics of TASK_LOST vs TASK_ERROR are simple: TASK_LOST means that
attempting to reschedule the task should succeed. TASK_ERROR means that any
attempt to reschedule the task will fail. This allows frameworks to make
better decisions.

Thank you.
-- 
Dominic Hamon | @mrdo | Twitter
*There are no bad ideas; only good ideas that go horribly wrong.*

Re: Review Request 27588: Removed trailing linefeeds from master and slave help messages.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27588/
---

(Updated Nov. 4, 2014, 5:49 p.m.)


Review request for mesos.


Changes
---

Updated summary.


Summary (updated)
-

Removed trailing linefeeds from master and slave help messages.


Repository: mesos-git


Description
---

Also fixed help message for master's credential flag.


Diffs
-

  src/master/flags.hpp 1cea50c02f3ad7de1e1ae91d65d1accdb9af7b03 
  src/slave/flags.hpp efbd35dd744d8c8a53c36dd5cdb8d1c50fcfec3a 

Diff: https://reviews.apache.org/r/27588/diff/


Testing
---

make check


Thanks,

Kapil Arya

Re: Review Request 27588: Removed trailing linefeeds from master and slave help messages.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27588/
---

(Updated Nov. 4, 2014, 5:50 p.m.)


Review request for mesos.


Repository: mesos-git


Description (updated)
---

Also fixed help message for master's credentials flag.


Diffs
-

  src/master/flags.hpp 1cea50c02f3ad7de1e1ae91d65d1accdb9af7b03 
  src/slave/flags.hpp efbd35dd744d8c8a53c36dd5cdb8d1c50fcfec3a 

Diff: https://reviews.apache.org/r/27588/diff/


Testing
---

make check


Thanks,

Kapil Arya

Re: Review Request 27555: Refactored the C++ Resources class to support persistent disk resources.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27555/#review59860
---


Patch looks great!

Reviews applied: [27550, 27552, 27555]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 10:01 p.m., Jie Yu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27555/
 ---
 
 (Updated Nov. 4, 2014, 10:01 p.m.)
 
 
 Review request for mesos, Ben Mahler and Vinod Kone.
 
 
 Bugs: MESOS-1974
 https://issues.apache.org/jira/browse/MESOS-1974
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 The purpose of the refactor is to support persistent disk resources.
 
 Here are the main things I've done in this refactor:
 1) Resource objects in Resources are stored in minimal format 
 (validated/non-zero). That allows us to kill isAllocatable, allocatable, 
 isZero, etc.
 2) 'matches' needs to be split into two pieces: one for combining and one for 
 removing, in order to support persitent disk resource. For example, one 
 cannot combine two Resource object with DiskInfo (it's like two disks), 
 however, you can do removal if they are identical.
 3) Some of the interfaces are not intuitive (e.g., =, see details in the 
 ticket). I removed them in favor of more explicit interfaces.
 4) Unified all the validation code.
 5) Adjusted the tests accordingly.
 
 
 Diffs
 -
 
   include/mesos/resources.hpp 0e37170 
   src/cli/execute.cpp ddaa20d 
   src/common/resources.cpp e9a0c85 
   src/examples/low_level_scheduler_libprocess.cpp 7ef5ea7 
   src/examples/low_level_scheduler_pthread.cpp 6e233a1 
   src/examples/no_executor_framework.cpp f98a073 
   src/examples/test_framework.cpp 187a611 
   src/master/drf_sorter.cpp 5464900 
   src/master/hierarchical_allocator_process.hpp 31dfb2c 
   src/master/http.cpp 3189933 
   src/master/master.cpp d914786 
   src/tests/allocator_tests.cpp 58e15aa 
   src/tests/gc_tests.cpp f7747e2 
   src/tests/master_tests.cpp 2e52574 
   src/tests/mesos.hpp c1d64a7 
   src/tests/resource_offers_tests.cpp fe66432 
   src/tests/resources_tests.cpp 3e50889 
   src/tests/slave_recovery_tests.cpp 98e059f 
   src/tests/sorter_tests.cpp 0516ab5 
 
 Diff: https://reviews.apache.org/r/27555/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Jie Yu

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.



 On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote:
  src/launcher/fetcher.cpp, line 223
  https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223
 
  Why fall through if result.isError()?
 
 Ankur Chauhan wrote:
 The way I am thinking of the logic of fetchers is on the lines of we try 
 all fetchers one by one till we succeed *even* if an unsuccessful fetcher can 
 handle it. This means that if for any reason (maybe mesos-fetcher is running 
 doesn't have permissions or whatever be the case) a fetcher fails we always 
 go to the next one. I understand that a case can be made to do something on 
 the lines of 
 
 ```
 if(local.canHandle(uri)) {
   return local.fetch(uri);
 } else if (os_net.canHandle(uri)) {
   return os_net.fetch(uri);
 } else if ( hadoop.canHandle(uri)) {
   return hadoop.fetch(uri);
 } else {
   return Error(Can't download uri: + uri);
 }
 ```
 
 But this way we are assuming that local fetcher has all possible 
 permissions to handle local/file uris *and* os_net has all the configuration 
 to handle all http(s) | ftp/ftp(s) uris and takes away the fallback. It may 
 be the case that the hadoop fetcher (and hence hadoop client) has some 
 credentials in its config files that aren't present in the urls themselves. 
 An example of this is if ftp uris are provided without user or password but 
 core-site.xml has the required user/password in it.
 
 Does this answer your concern?

I'm concerned about how the logs will look for an user when fetching URIs. If 
it's a non-local URI they would always see a message and failure with local 
copy before using HDFS or Net, which is bad.

I would much rather make it explicit what you are doing, with comments.

```
if (local.canHandle(uri) {
  result = local.fetch(uri);
  if (result.isSome()) {
return result;
  } else {
LOG(WARNING)  Failed to copy local file:   result.Error()  ;
  }
} else if (os_net.canHandle(uri)) {
  result = os_net.fetch(uri);
  if (result.isSome()) {
return result;
  } else {
LOG(WARNING)  Failed to download URI:   result.Error()  ;
  }
}

// If we are here, one of the following is possible.
// 1. Failed to copy a local URI. 
// 2. Failed to download a remote URI using Os.net.
// 3. HDFS compatible URI.
// 4. Unexpected URI.
// For all these cases we try to fetch the URI using Hadoop client as a 
fallback.

return hadoop.fetch(uri);

```

More importantly, are there really file permission issues where cp doesn't 
work but hadoop fs copyToLocal works? Can you give me a concrete example? I 
ask because if the hadoop client hangs for whatever reason (we have seen this 
in production at Twitter), it will take upto the executor registration timeout 
for the slave to kill the executor and transition the task to LOST. Getting rid 
of hadoop dependency was one of the reasons we install our executors locally. 
This change adds the dependency back on hadoop, even for local files, which is 
unfortunate.


- Vinod


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59842
---


On Nov. 4, 2014, 10:20 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 10:20 p.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.



 On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote:
  src/launcher/fetcher.cpp, line 223
  https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223
 
  Why fall through if result.isError()?
 
 Ankur Chauhan wrote:
 The way I am thinking of the logic of fetchers is on the lines of we try 
 all fetchers one by one till we succeed *even* if an unsuccessful fetcher can 
 handle it. This means that if for any reason (maybe mesos-fetcher is running 
 doesn't have permissions or whatever be the case) a fetcher fails we always 
 go to the next one. I understand that a case can be made to do something on 
 the lines of 
 
 ```
 if(local.canHandle(uri)) {
   return local.fetch(uri);
 } else if (os_net.canHandle(uri)) {
   return os_net.fetch(uri);
 } else if ( hadoop.canHandle(uri)) {
   return hadoop.fetch(uri);
 } else {
   return Error(Can't download uri: + uri);
 }
 ```
 
 But this way we are assuming that local fetcher has all possible 
 permissions to handle local/file uris *and* os_net has all the configuration 
 to handle all http(s) | ftp/ftp(s) uris and takes away the fallback. It may 
 be the case that the hadoop fetcher (and hence hadoop client) has some 
 credentials in its config files that aren't present in the urls themselves. 
 An example of this is if ftp uris are provided without user or password but 
 core-site.xml has the required user/password in it.
 
 Does this answer your concern?
 
 Vinod Kone wrote:
 I'm concerned about how the logs will look for an user when fetching 
 URIs. If it's a non-local URI they would always see a message and failure 
 with local copy before using HDFS or Net, which is bad.
 
 I would much rather make it explicit what you are doing, with comments.
 
 ```
 if (local.canHandle(uri) {
   result = local.fetch(uri);
   if (result.isSome()) {
 return result;
   } else {
 LOG(WARNING)  Failed to copy local file:   result.Error()  ;
   }
 } else if (os_net.canHandle(uri)) {
   result = os_net.fetch(uri);
   if (result.isSome()) {
 return result;
   } else {
 LOG(WARNING)  Failed to download URI:   result.Error()  ;
   }
 }
 
 // If we are here, one of the following is possible.
 // 1. Failed to copy a local URI. 
 // 2. Failed to download a remote URI using Os.net.
 // 3. HDFS compatible URI.
 // 4. Unexpected URI.
 // For all these cases we try to fetch the URI using Hadoop client as a 
 fallback.
 
 return hadoop.fetch(uri);
 
 ```
 
 More importantly, are there really file permission issues where cp 
 doesn't work but hadoop fs copyToLocal works? Can you give me a concrete 
 example? I ask because if the hadoop client hangs for whatever reason (we 
 have seen this in production at Twitter), it will take upto the executor 
 registration timeout for the slave to kill the executor and transition the 
 task to LOST. Getting rid of hadoop dependency was one of the reasons we 
 install our executors locally. This change adds the dependency back on 
 hadoop, even for local files, which is unfortunate.

That does make sense, I hadn't considered the logging :-( and I am all up for 
no hard external dependencies if I can avoid it. Let me think a little on that.


- Ankur


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59842
---


On Nov. 4, 2014, 10:20 p.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 4, 2014, 10:20 p.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check
 sudo bin/mesos-tests.sh --verbose
 support/mesos-style.py
 
 
 Thanks,
 
 Ankur Chauhan

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59873
---


Patch looks great!

Reviews applied: [27588, 27573, 27560, 27556]

All tests passed.

- Mesos ReviewBot


On Nov. 4, 2014, 10:33 p.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 4, 2014, 10:33 p.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 The option tables are now in html format.
 
 Also removed the explanation of the default configure flags.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27567: MESOS-2038: removed dead code from _runTask()


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/
---

(Updated Nov. 4, 2014, 11:46 p.m.)


Review request for mesos and Vinod Kone.


Changes
---

Thanks Bernd! I will let vinod take a look since he has the context here.

--bmahler


Bugs: MESOS-2038
https://issues.apache.org/jira/browse/MESOS-2038


Repository: mesos-git


Description
---

Removed a few lines of dead code that coverty discovered.


Diffs
-

  src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 

Diff: https://reviews.apache.org/r/27567/diff/


Testing
---

none.
expecting/waiting for review bot to report no problem.


Thanks,

Bernd Mathiske

Re: Review Request 27567: MESOS-2038: removed dead code from _runTask()

2014-11-04 Thread Adam B



 On Nov. 4, 2014, 3:09 a.m., Adam B wrote:
  Ship It!

Following the logic with my internal C++ compiler/interpreter (aka my brain), 
it's pretty clear that we would have returned earlier in this function if 
framework were in fact NULL, so the null-check branch is indeed unreachable 
code.


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/#review59760
---


On Nov. 4, 2014, 3:46 p.m., Bernd Mathiske wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27567/
 ---
 
 (Updated Nov. 4, 2014, 3:46 p.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-2038
 https://issues.apache.org/jira/browse/MESOS-2038
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Removed a few lines of dead code that coverty discovered.
 
 
 Diffs
 -
 
   src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 
 
 Diff: https://reviews.apache.org/r/27567/diff/
 
 
 Testing
 ---
 
 none.
 expecting/waiting for review bot to report no problem.
 
 
 Thanks,
 
 Bernd Mathiske

Re: Review Request 27550: Added DiskInfo to the Resource protobuf.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27550/#review59880
---



include/mesos/mesos.proto
https://reviews.apache.org/r/27550/#comment101186

why is this optional instead of required?


- Vinod Kone


On Nov. 4, 2014, 12:29 a.m., Jie Yu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27550/
 ---
 
 (Updated Nov. 4, 2014, 12:29 a.m.)
 
 
 Review request for mesos, Ben Mahler and Vinod Kone.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 See summary.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 6b93e9000761857c4f335f2a8c8088e155078f54 
 
 Diff: https://reviews.apache.org/r/27550/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Jie Yu

Re: Review Request 27552: Added a slave flag to opt out using default resources.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27552/#review59864
---



src/slave/flags.hpp
https://reviews.apache.org/r/27552/#comment101170

Instead of adding another flag, why not consider absence of --resources 
flag to mean using default resources and --resources='' to mean using empty 
resources?


- Vinod Kone


On Nov. 4, 2014, 12:30 a.m., Jie Yu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27552/
 ---
 
 (Updated Nov. 4, 2014, 12:30 a.m.)
 
 
 Review request for mesos, Ben Mahler and Vinod Kone.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 This is used in testing of the new Resources class. See the dependent review.
 
 
 Diffs
 -
 
   src/slave/containerizer/containerizer.cpp 
 0254679508167a390fd6fed855f19794354ac081 
   src/slave/flags.hpp 03c62a2fd040768392c7e24d93f64ca3a855c4a1 
 
 Diff: https://reviews.apache.org/r/27552/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Jie Yu

Re: Review Request 27567: MESOS-2038: removed dead code from _runTask()

2014-11-04 Thread Ben Mahler



 On Nov. 4, 2014, 11:09 a.m., Adam B wrote:
  Ship It!
 
 Adam B wrote:
 Following the logic with my internal C++ compiler/interpreter (aka my 
 brain), it's pretty clear that we would have returned earlier in this 
 function if framework were in fact NULL, so the null-check branch is indeed 
 unreachable code.

Definitely unreachable, I just took a glance and couldn't easily figure out 
when this case occurs. I also couldn't quickly figure out whether the creation 
of the framework was something we want to do in this case vs. log a warning and 
bail. If we bail when the framework is NULL and this is something we expect to 
occur, do we need to send task lost?

With these questions, I realized I should just let vinod review this given he 
had more context around this.


- Ben


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27567/#review59760
---


On Nov. 4, 2014, 11:46 p.m., Bernd Mathiske wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27567/
 ---
 
 (Updated Nov. 4, 2014, 11:46 p.m.)
 
 
 Review request for mesos and Vinod Kone.
 
 
 Bugs: MESOS-2038
 https://issues.apache.org/jira/browse/MESOS-2038
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Removed a few lines of dead code that coverty discovered.
 
 
 Diffs
 -
 
   src/slave/slave.cpp 5e9b0e4f93a5100a91340e1f6fb1fe160b2eea4b 
 
 Diff: https://reviews.apache.org/r/27567/diff/
 
 
 Testing
 ---
 
 none.
 expecting/waiting for review bot to report no problem.
 
 
 Thanks,
 
 Bernd Mathiske

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59895
---


Vastly better!

Can you avoid wrapping some of the long flag names like 
--[no-]initialize_driver_logging?

- Ian Downes


On Nov. 4, 2014, 2:33 p.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 4, 2014, 2:33 p.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 The option tables are now in html format.
 
 Also removed the explanation of the default configure flags.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27550: Added DiskInfo to the Resource protobuf.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27550/#review59896
---



include/mesos/mesos.proto
https://reviews.apache.org/r/27550/#comment101212

DiskInfo is a very generic name; it's not clear that it's a *persistent* 
disk resource. Is there a possible need for Resource to included non-persistent 
disk later on and which lead to some unfortunate naming?


- Ian Downes


On Nov. 3, 2014, 4:29 p.m., Jie Yu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27550/
 ---
 
 (Updated Nov. 3, 2014, 4:29 p.m.)
 
 
 Review request for mesos, Ben Mahler and Vinod Kone.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 See summary.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 6b93e9000761857c4f335f2a8c8088e155078f54 
 
 Diff: https://reviews.apache.org/r/27550/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Jie Yu

Re: Review Request 27550: Added DiskInfo to the Resource protobuf.

2014-11-04 Thread Jie Yu



 On Nov. 5, 2014, 12:48 a.m., Ian Downes wrote:
  include/mesos/mesos.proto, line 393
  https://reviews.apache.org/r/27550/diff/1/?file=747884#file747884line393
 
  DiskInfo is a very generic name; it's not clear that it's a 
  *persistent* disk resource. Is there a possible need for Resource to 
  included non-persistent disk later on and which lead to some unfortunate 
  naming?

That's because the name the resource is called disk right now. We may have 
CpusInfo, MemInfo in the future.


- Jie


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27550/#review59896
---


On Nov. 4, 2014, 12:29 a.m., Jie Yu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27550/
 ---
 
 (Updated Nov. 4, 2014, 12:29 a.m.)
 
 
 Review request for mesos, Ben Mahler and Vinod Kone.
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 See summary.
 
 
 Diffs
 -
 
   include/mesos/mesos.proto 6b93e9000761857c4f335f2a8c8088e155078f54 
 
 Diff: https://reviews.apache.org/r/27550/diff/
 
 
 Testing
 ---
 
 make check
 
 
 Thanks,
 
 Jie Yu

Review Request 27605: Add a unit test to stout path

2014-11-04 Thread Cody Maloney


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27605/
---

Review request for mesos and Ben Mahler.


Bugs: MESOS-1733 and mesos-1877
https://issues.apache.org/jira/browse/MESOS-1733
https://issues.apache.org/jira/browse/mesos-1877


Repository: mesos-git


Description
---

Add a unit test to stout path to increase confidence when changing the code.


Diffs
-

  3rdparty/libprocess/3rdparty/Makefile.am 
0dc2871af7d8c6646080263c71f41db855d50be4 
  3rdparty/libprocess/3rdparty/stout/Makefile.am 
7aac3fdb864e838cdd4e0d817ed4dff14923c69d 
  3rdparty/libprocess/3rdparty/stout/include/stout/path.hpp 
bc6920a24d920c809e36a2a3da1810e52d0db101 
  3rdparty/libprocess/3rdparty/stout/tests/path_tests.cpp PRE-CREATION 

Diff: https://reviews.apache.org/r/27605/diff/


Testing
---

make distcheck on Ubuntu 14.04

NOTE: ReviewBot is going to be unhappy about this crossing stout and 
libprocess... I'm not sure what the correct fix for reviewbot is in this case, 
since the Makefile does really cross the project bounds...


Thanks,

Cody Maloney

Review Request 27606: Standardized path handling in files

2014-11-04 Thread Cody Maloney


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27606/
---

Review request for mesos and Ben Mahler.


Bugs: mesos-1877
https://issues.apache.org/jira/browse/mesos-1877


Repository: mesos-git


Description
---

Files attach, detach, and browse now all call a single function to standardize 
the path. Before attach, browse did the normalization. Detach did not.

Switch to strings::tokenize() for splitting apart the path into tokens, as that 
is more canonical across the codebase, used by libprocess for proccessing paths.

Update detach to return a bool of whether or not the detach does anything which 
means that it can be tested.

Add some additional testing to catch the inconsistencies in path handlign 
between attach and detach previously, some of the oddness in current path 
handling so that they can be explicitly seen when changes happen.


Diffs
-

  src/files/files.hpp 818087b13cc787d0bd3186bb3e8a069751629bf9 
  src/files/files.cpp 12e8f75aa7bd77d2e81d5d3a7a4d09dd915854aa 
  src/tests/files_tests.cpp a696aa22d56b37ee70c6e64c81a849da6d436451 

Diff: https://reviews.apache.org/r/27606/diff/


Testing
---

make distcheck on ubuntu 14.04


Thanks,

Cody Maloney

Re: Review Request 27605: Add a unit test to stout path


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27605/#review59925
---


Bad patch!

Reviews applied: [27605]

Failed command: ./support/apply-review.sh -n -r 27605

Error:
 2014-11-05 03:22:17 URL:https://reviews.apache.org/r/27605/diff/raw/ 
[3414/3414] - 27605.patch [1]
Successfully applied: Add a unit test to stout path

Add a unit test to stout path to increase confidence when changing the code.


Review: https://reviews.apache.org/r/27605
Checking 2 files using filter 
--filter=-,+build/class,+build/deprecated,+build/endif_comment,+readability/todo,+readability/namespace,+runtime/vlog,+whitespace/blank_line,+whitespace/comma,+whitespace/end_of_line,+whitespace/ending_newline,+whitespace/forcolon,+whitespace/indent,+whitespace/line_length,+whitespace/operators,+whitespace/semicolon,+whitespace/tab,+whitespace/todo
Total errors found: 0
ERROR: Commit spanning multiple projects.

Please use separate commits for mesos, libprocess and stout.

Paths grouped by project:
stout:
  3rdparty/libprocess/3rdparty/stout/Makefile.am
  3rdparty/libprocess/3rdparty/stout/include/stout/path.hpp
  3rdparty/libprocess/3rdparty/stout/tests/path_tests.cpp
libprocess:
  3rdparty/libprocess/3rdparty/Makefile.am
Failed to commit patch

- Mesos ReviewBot


On Nov. 5, 2014, 2:53 a.m., Cody Maloney wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27605/
 ---
 
 (Updated Nov. 5, 2014, 2:53 a.m.)
 
 
 Review request for mesos and Ben Mahler.
 
 
 Bugs: MESOS-1733 and mesos-1877
 https://issues.apache.org/jira/browse/MESOS-1733
 https://issues.apache.org/jira/browse/mesos-1877
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Add a unit test to stout path to increase confidence when changing the code.
 
 
 Diffs
 -
 
   3rdparty/libprocess/3rdparty/Makefile.am 
 0dc2871af7d8c6646080263c71f41db855d50be4 
   3rdparty/libprocess/3rdparty/stout/Makefile.am 
 7aac3fdb864e838cdd4e0d817ed4dff14923c69d 
   3rdparty/libprocess/3rdparty/stout/include/stout/path.hpp 
 bc6920a24d920c809e36a2a3da1810e52d0db101 
   3rdparty/libprocess/3rdparty/stout/tests/path_tests.cpp PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/27605/diff/
 
 
 Testing
 ---
 
 make distcheck on Ubuntu 14.04
 
 NOTE: ReviewBot is going to be unhappy about this crossing stout and 
 libprocess... I'm not sure what the correct fix for reviewbot is in this 
 case, since the Makefile does really cross the project bounds...
 
 
 Thanks,
 
 Cody Maloney

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/
---

(Updated Nov. 4, 2014, 10:37 p.m.)


Review request for mesos, Ian Downes and Till Toenshoff.


Changes
---

Fixed column widths.


Bugs: MESOS-2037
https://issues.apache.org/jira/browse/MESOS-2037


Repository: mesos-git


Description
---

Copied the output of --help messages from configure, master, and slave.

The option tables are now in html format.

Also removed the explanation of the default configure flags.


Diffs (updated)
-

  docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 

Diff: https://reviews.apache.org/r/27556/diff/


Testing
---

Pushed and tested output on github at:
https://github.com/karya0/mesos/blob/modules/docs/configuration.md


Thanks,

Kapil Arya

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.



 On Nov. 4, 2014, 7:44 p.m., Ian Downes wrote:
  Vastly better!
  
  Can you avoid wrapping some of the long flag names like 
  --[no-]initialize_driver_logging?

I fixed most of them, but there are still three and making them fit is getting 
challenging.  I would leave them as is unless there is a strong preference.


- Kapil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59895
---


On Nov. 4, 2014, 10:37 p.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 4, 2014, 10:37 p.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 The option tables are now in html format.
 
 Also removed the explanation of the default configure flags.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27606: Standardized path handling in files


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27606/#review59930
---


Patch looks great!

Reviews applied: [27606]

All tests passed.

- Mesos ReviewBot


On Nov. 5, 2014, 2:53 a.m., Cody Maloney wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27606/
 ---
 
 (Updated Nov. 5, 2014, 2:53 a.m.)
 
 
 Review request for mesos and Ben Mahler.
 
 
 Bugs: mesos-1877
 https://issues.apache.org/jira/browse/mesos-1877
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Files attach, detach, and browse now all call a single function to 
 standardize the path. Before attach, browse did the normalization. Detach did 
 not.
 
 Switch to strings::tokenize() for splitting apart the path into tokens, as 
 that is more canonical across the codebase, used by libprocess for 
 proccessing paths.
 
 Update detach to return a bool of whether or not the detach does anything 
 which means that it can be tested.
 
 Add some additional testing to catch the inconsistencies in path handlign 
 between attach and detach previously, some of the oddness in current path 
 handling so that they can be explicitly seen when changes happen.
 
 
 Diffs
 -
 
   src/files/files.hpp 818087b13cc787d0bd3186bb3e8a069751629bf9 
   src/files/files.cpp 12e8f75aa7bd77d2e81d5d3a7a4d09dd915854aa 
   src/tests/files_tests.cpp a696aa22d56b37ee70c6e64c81a849da6d436451 
 
 Diff: https://reviews.apache.org/r/27606/diff/
 
 
 Testing
 ---
 
 make distcheck on ubuntu 14.04
 
 
 Thanks,
 
 Cody Maloney

Re: Review Request 27556: Updated docs/configuration.md to reflect current state.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27556/#review59932
---


Patch looks great!

Reviews applied: [27588, 27573, 27560, 27556]

All tests passed.

- Mesos ReviewBot


On Nov. 5, 2014, 3:37 a.m., Kapil Arya wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27556/
 ---
 
 (Updated Nov. 5, 2014, 3:37 a.m.)
 
 
 Review request for mesos, Ian Downes and Till Toenshoff.
 
 
 Bugs: MESOS-2037
 https://issues.apache.org/jira/browse/MESOS-2037
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Copied the output of --help messages from configure, master, and slave.
 
 The option tables are now in html format.
 
 Also removed the explanation of the default configure flags.
 
 
 Diffs
 -
 
   docs/configuration.md 5845ae324181d01cb65990fbf8dd38a621e1c351 
 
 Diff: https://reviews.apache.org/r/27556/diff/
 
 
 Testing
 ---
 
 Pushed and tested output on github at:
 https://github.com/karya0/mesos/blob/modules/docs/configuration.md
 
 
 Thanks,
 
 Kapil Arya

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/
---

(Updated Nov. 5, 2014, 6:39 a.m.)


Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.


Changes
---

Do compatibility check before using a fetcher, add tests.


Bugs: MESOS-1711
https://issues.apache.org/jira/browse/MESOS-1711


Repository: mesos-git


Description
---

Previously, the fetcher used a hardcoded list of schemes to determine what URIs 
could be fetched by hadoop (if available). This is now changed such that we 
first check if hadoop can fetch them for us and then we fallback to the os::net 
and then a local copy method (same as it used to be). This allows users to 
fetch artifacts from arbitrary filesystems as long as hadoop is correctly 
configured (in core-site.xml).


Diffs (updated)
-

  src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
  src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
  src/tests/fetcher_tests.cpp d7754009a59fedb43e3422c56b3a786ce80164aa 

Diff: https://reviews.apache.org/r/27483/diff/


Testing
---

make check
sudo bin/mesos-tests.sh --verbose
support/mesos-style.py


Thanks,

Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.



 On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote:
  src/launcher/fetcher.cpp, line 223
  https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223
 
  Why fall through if result.isError()?
 
 Ankur Chauhan wrote:
 The way I am thinking of the logic of fetchers is on the lines of we try 
 all fetchers one by one till we succeed *even* if an unsuccessful fetcher can 
 handle it. This means that if for any reason (maybe mesos-fetcher is running 
 doesn't have permissions or whatever be the case) a fetcher fails we always 
 go to the next one. I understand that a case can be made to do something on 
 the lines of 
 
 ```
 if(local.canHandle(uri)) {
   return local.fetch(uri);
 } else if (os_net.canHandle(uri)) {
   return os_net.fetch(uri);
 } else if ( hadoop.canHandle(uri)) {
   return hadoop.fetch(uri);
 } else {
   return Error(Can't download uri: + uri);
 }
 ```
 
 But this way we are assuming that local fetcher has all possible 
 permissions to handle local/file uris *and* os_net has all the configuration 
 to handle all http(s) | ftp/ftp(s) uris and takes away the fallback. It may 
 be the case that the hadoop fetcher (and hence hadoop client) has some 
 credentials in its config files that aren't present in the urls themselves. 
 An example of this is if ftp uris are provided without user or password but 
 core-site.xml has the required user/password in it.
 
 Does this answer your concern?
 
 Vinod Kone wrote:
 I'm concerned about how the logs will look for an user when fetching 
 URIs. If it's a non-local URI they would always see a message and failure 
 with local copy before using HDFS or Net, which is bad.
 
 I would much rather make it explicit what you are doing, with comments.
 
 ```
 if (local.canHandle(uri) {
   result = local.fetch(uri);
   if (result.isSome()) {
 return result;
   } else {
 LOG(WARNING)  Failed to copy local file:   result.Error()  ;
   }
 } else if (os_net.canHandle(uri)) {
   result = os_net.fetch(uri);
   if (result.isSome()) {
 return result;
   } else {
 LOG(WARNING)  Failed to download URI:   result.Error()  ;
   }
 }
 
 // If we are here, one of the following is possible.
 // 1. Failed to copy a local URI. 
 // 2. Failed to download a remote URI using Os.net.
 // 3. HDFS compatible URI.
 // 4. Unexpected URI.
 // For all these cases we try to fetch the URI using Hadoop client as a 
 fallback.
 
 return hadoop.fetch(uri);
 
 ```
 
 More importantly, are there really file permission issues where cp 
 doesn't work but hadoop fs copyToLocal works? Can you give me a concrete 
 example? I ask because if the hadoop client hangs for whatever reason (we 
 have seen this in production at Twitter), it will take upto the executor 
 registration timeout for the slave to kill the executor and transition the 
 task to LOST. Getting rid of hadoop dependency was one of the reasons we 
 install our executors locally. This change adds the dependency back on 
 hadoop, even for local files, which is unfortunate.
 
 Ankur Chauhan wrote:
 That does make sense, I hadn't considered the logging :-( and I am all up 
 for no hard external dependencies if I can avoid it. Let me think a little on 
 that.

The latest patch should take care of this. I am following the same approach as 
above.


- Ankur


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59842
---


On Nov. 5, 2014, 6:39 a.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27483/
 ---
 
 (Updated Nov. 5, 2014, 6:39 a.m.)
 
 
 Review request for mesos, Timothy Chen, Timothy St. Clair, and Vinod Kone.
 
 
 Bugs: MESOS-1711
 https://issues.apache.org/jira/browse/MESOS-1711
 
 
 Repository: mesos-git
 
 
 Description
 ---
 
 Previously, the fetcher used a hardcoded list of schemes to determine what 
 URIs could be fetched by hadoop (if available). This is now changed such that 
 we first check if hadoop can fetch them for us and then we fallback to the 
 os::net and then a local copy method (same as it used to be). This allows 
 users to fetch artifacts from arbitrary filesystems as long as hadoop is 
 correctly configured (in core-site.xml).
 
 
 Diffs
 -
 
   src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 
   src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 
   src/tests/fetcher_tests.cpp d7754009a59fedb43e3422c56b3a786ce80164aa 
 
 Diff: https://reviews.apache.org/r/27483/diff/
 
 
 Testing
 ---
 
 make check

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27483/#review59934
---


This may be a side issue but I noticed in a run of `bin/mesos-tests.sh 
--gtest_filter=FetcherTest.*`:

```
$ ./bin/mesos-tests.sh --gtest_filter=FetcherTest.*
Source directory: /Users/achauhan/Projects/mesos
Build directory: /Users/achauhan/Projects/mesos/build
-
We cannot run any Docker tests because:
Docker tests not supported on non-Linux systems
-
Note: Google Test filter = 
FetcherTest.*-DockerContainerizerTest.ROOT_DOCKER_Launch:DockerContainerizerTest.ROOT_DOCKER_Kill:DockerContainerizerTest.ROOT_DOCKER_Usage:DockerContainerizerTest.DISABLED_ROOT_DOCKER_Recover:DockerContainerizerTest.ROOT_DOCKER_Logs:DockerContainerizerTest.ROOT_DOCKER_Default_CMD:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Override:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Args:DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer:DockerContainerizerTest.DISABLED_ROOT_DOCKER_SlaveRecoveryExecutorContainer:DockerContainerizerTest.ROOT_DOCKER_PortMapping:DockerContainerizerTest.ROOT_DOCKER_LaunchSandboxWithColon:DockerTest.ROOT_DOCKER_interface:DockerTest.ROOT_DOCKER_CheckCommandWithShell:DockerTest.ROOT_DOCKER_CheckPortResource:DockerTest.ROOT_DOCKER_CancelPull:SlaveTest.ROOT_RunTaskWithCommandInfoWithoutUser:SlaveTest.DISABLED_ROOT_RunTaskWithCommandInfoWithUser:SlaveCount/Registrar_BENCHMARK_Test.performance/0:SlaveCount/Registrar_BENCH
 
MARK_Test.performance/1:SlaveCount/Registrar_BENCHMARK_Test.performance/2:SlaveCount/Registrar_BENCHMARK_Test.performance/3
[==] Running 4 tests from 1 test case.
[--] Global test environment set-up.
[--] 4 tests from FetcherTest
[ RUN  ] FetcherTest.FileURI
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1104 22:43:38.412901 2068996864 fetcher.cpp:197] Fetching URI 
'file:///private/tmp/FetcherTest_FileURI_ipAYlv/from/test'
I1104 22:43:38.413653 2068996864 fetcher.cpp:178] Copying resource from 
'/private/tmp/FetcherTest_FileURI_ipAYlv/from/test' to 
'/private/tmp/FetcherTest_FileURI_ipAYlv'
I1104 22:43:38.419317 2068996864 fetcher.cpp:300] Skipped extracting path 
'/private/tmp/FetcherTest_FileURI_ipAYlv/test'
[   OK ] FetcherTest.FileURI (110 ms)
[ RUN  ] FetcherTest.FilePath
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1104 22:43:38.507416 2068996864 fetcher.cpp:197] Fetching URI 
'/private/tmp/FetcherTest_FilePath_lAzOZG/from/test'
I1104 22:43:38.508020 2068996864 fetcher.cpp:178] Copying resource from 
'/private/tmp/FetcherTest_FilePath_lAzOZG/from/test' to 
'/private/tmp/FetcherTest_FilePath_lAzOZG'
I1104 22:43:38.512864 2068996864 fetcher.cpp:300] Skipped extracting path 
'/private/tmp/FetcherTest_FilePath_lAzOZG/test'
[   OK ] FetcherTest.FilePath (100 ms)
[ RUN  ] FetcherTest.OSNetUriTest
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1104 22:43:38.608177 2068996864 fetcher.cpp:197] Fetching URI 
'http://www.example.com/index.html'
I1104 22:43:38.608901 2068996864 fetcher.cpp:109] Fetching URI 
'http://www.example.com/index.html' with os::net
I1104 22:43:38.608924 2068996864 fetcher.cpp:119] Downloading 
'http://www.example.com/index.html' to 
'/private/tmp/FetcherTest_OSNetUriTest_ZB4nAg/index.html'
I1104 22:43:38.619645 2068996864 fetcher.cpp:300] Skipped extracting path 
'/private/tmp/FetcherTest_OSNetUriTest_ZB4nAg/index.html'
[   OK ] FetcherTest.OSNetUriTest (105 ms)
[ RUN  ] FetcherTest.FileLocalhostURI
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1104 22:43:38.712461 2068996864 fetcher.cpp:197] Fetching URI 
'file://localhost/private/tmp/FetcherTest_FileLocalhostURI_m1wYnH/from/test'
I1104 22:43:38.713353 2068996864 fetcher.cpp:178] Copying resource from 
'/private/tmp/FetcherTest_FileLocalhostURI_m1wYnH/from/test' to 
'/private/tmp/FetcherTest_FileLocalhostURI_m1wYnH'
I1104 22:43:38.718541 2068996864 fetcher.cpp:300] Skipped extracting path 
'/private/tmp/FetcherTest_FileLocalhostURI_m1wYnH/test'
[   OK ] FetcherTest.FileLocalhostURI (103 ms)
[--] 4 tests from FetcherTest (419 ms total)

[--] Global test environment tear-down
[==] 4 tests from 1 test case ran. (434 ms total)
[  PASSED  ] 4 tests.

  YOU HAVE 5 DISABLED TESTS
```

There are warnings in the output:

```
WARNING: Logging before InitGoogleLogging() is written to STDERR
```

And looking at the code, I don't see a glog initialization codeblock. Should 
there be `google::InitGoogleLogging(argv[0]);` in the `main` method?

- Ankur Chauhan


On Nov. 5, 2014, 6:39 a.m., Ankur Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.