[jira] [Updated] (MESOS-1832) Slave should accept PingSlaveMessage but not PING message.
[ https://issues.apache.org/jira/browse/MESOS-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-1832: -- Target Version/s: 0.24.0 (was: 0.23.0) Slave should accept PingSlaveMessage but not PING message. Key: MESOS-1832 URL: https://issues.apache.org/jira/browse/MESOS-1832 Project: Mesos Issue Type: Task Reporter: Vinod Kone Assignee: Adam B Labels: mesosphere Slave handles both PING message and PingSlaveMessage in until 0.22.0 for backwards compatibility (https://reviews.apache.org/r/25867/). In 0.23.0, slave no longer needs handle PING. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1831) Master should send PingSlaveMessage instead of PING
[ https://issues.apache.org/jira/browse/MESOS-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-1831: -- Target Version/s: 0.24.0 (was: 0.23.0) Master should send PingSlaveMessage instead of PING - Key: MESOS-1831 URL: https://issues.apache.org/jira/browse/MESOS-1831 Project: Mesos Issue Type: Task Reporter: Vinod Kone Assignee: Adam B Labels: mesosphere In 0.21.0 master sends PING message with an embedded PingSlaveMessage for backwards compatibility (https://reviews.apache.org/r/25867/). In 0.22.0, master should send PingSlaveMessage directly instead of PING. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1791) Introduce Master / Offer Resource Reservations aka Quota
[ https://issues.apache.org/jira/browse/MESOS-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-1791: --- Epic Name: Quota (was: Offer Reservations) Component/s: replicated log master allocation Summary: Introduce Master / Offer Resource Reservations aka Quota (was: Introduce Master / Offer Resource Reservations) Introduce Master / Offer Resource Reservations aka Quota Key: MESOS-1791 URL: https://issues.apache.org/jira/browse/MESOS-1791 Project: Mesos Issue Type: Epic Components: allocation, master, replicated log Reporter: Tom Arnfeld Labels: mesosphere Currently Mesos supports the ability to reserve resources (for a given role) on a per-slave basis, as introduced in MESOS-505. This allows you to almost statically partition off a set of resources on a set of machines, to guarantee certain types of frameworks get some resources. This is very useful, though it is also very useful to be able to control these reservations through the master (instead of per-slave) for when I don't care which nodes I get on, as long as I get X cpu and Y RAM, or Z sets of (X,Y). I'm not sure what structure this could take, but apparently it has already been discussed. Would this be a CLI flag? Could there be a (authenticated) web interface to control these reservations? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2774) SIGSEGV received during process::MessageEncoder::encode()
[ https://issues.apache.org/jira/browse/MESOS-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599149#comment-14599149 ] Klaus Ma commented on MESOS-2774: - Where can I download the core file? SIGSEGV received during process::MessageEncoder::encode() - Key: MESOS-2774 URL: https://issues.apache.org/jira/browse/MESOS-2774 Project: Mesos Issue Type: Bug Affects Versions: 0.23.0 Reporter: Yan Xu Priority: Critical Observed in production. {noformat:title=} Program terminated with signal 11, Segmentation fault. #0 0x7fd6ded54b2e in _itoa_word () from /lib64/libc.so.6 (gdb) bt #0 0x7fd6ded54b2e in _itoa_word () from /lib64/libc.so.6 #1 0x7fd6ded57f13 in vfprintf () from /lib64/libc.so.6 #2 0x7fd6ded75b49 in vsprintf () from /lib64/libc.so.6 #3 0x7fd6ded5fe38 in sprintf () from /lib64/libc.so.6 #4 0x7fd6dedf473e in inet_ntop () from /lib64/libc.so.6 #5 0x7fd6e058aed6 in operator (stream=..., pid=Unhandled dwarf expression opcode 0xf3 ) at ./3rdparty/stout/include/stout/ip.hpp:225 #6 operator (stream=..., pid=Unhandled dwarf expression opcode 0xf3 ) at ./include/process/address.hpp:106 #7 process::operator (stream=..., pid=Unhandled dwarf expression opcode 0xf3 ) at src/pid.cpp:65 #8 0x7fd6e05ab29a in process::MessageEncoder::encode (message=0x7fd6b80e2d70) at src/encoder.hpp:123 #9 0x7fd6e05ad1b5 in process::MessageEncoder::MessageEncoder (this=0x7fd6b813aed0, s=..., _message=0x7fd6b80e2d70) at src/encoder.hpp:98 #10 0x7fd6e05a60cc in process::SocketManager::send (this=0x124d100, message=0x7fd6b80e2d70) at src/process.cpp:1583 #11 0x7fd6e05a6750 in process::ProcessBase::send (this=0x7fd6c00097d0, to=..., name=PONG, data=0x0, length=0) at src/process.cpp:2679 #12 0x7fd6e0155f87 in mesos::internal::slave::Slave::pingOld (this=0x7fd6c0008f20, from=..., body=Unhandled dwarf expression opcode 0xf3 ) at slave/slave.cpp:2896 #13 0x7fd6e05a68cb in operator() (this=0x7fd6c00097d0, event=...) at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:2464 #14 process::ProcessBase::visit (this=0x7fd6c00097d0, event=...) at src/process.cpp:2688 #15 0x7fd6e019120d in ProtobufProcessmesos::internal::slave::Slave::visit (this=0x7fd6c0008f20, event=...) at ../3rdparty/libprocess/include/process/protobuf.hpp:94 #16 0x7fd6e05a009a in process::ProcessManager::resume (this=0x124d3b0, process=0x7fd6c00097d0) at src/process.cpp:2175 #17 0x7fd6e05a035c in process::schedule (arg=Unhandled dwarf expression opcode 0xf3 ) at src/process.cpp:613 #18 0x7fd6df5f583d in start_thread () from /lib64/libpthread.so.0 #19 0x7fd6dede7fcd in clone () from /lib64/libc.so.6 {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2154) Port CFS support to Docker Containerizer
[ https://issues.apache.org/jira/browse/MESOS-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599049#comment-14599049 ] Adam B commented on MESOS-2154: --- Pushing this out to 0.24, since nobody seems to be working on it. Port CFS support to Docker Containerizer Key: MESOS-2154 URL: https://issues.apache.org/jira/browse/MESOS-2154 Project: Mesos Issue Type: Improvement Components: docker, isolation Affects Versions: 0.21.0 Environment: Linux (Ubuntu 14.04.1) Reporter: Andrew Ortman Priority: Minor Port the CFS support the Mesos Containerizer has to the Docker Containerizer. Whenever the --cgroup_enable_cfs flag is set, the Docker Containerizer should update the cfs_period_us and cfs_quota_us values to allow hard CPU capping on the container. Current workaround is to pass those values as LXC configuration parameters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2154) Port CFS support to Docker Containerizer
[ https://issues.apache.org/jira/browse/MESOS-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-2154: -- Target Version/s: 0.24.0 (was: 0.23.0) Port CFS support to Docker Containerizer Key: MESOS-2154 URL: https://issues.apache.org/jira/browse/MESOS-2154 Project: Mesos Issue Type: Improvement Components: docker, isolation Affects Versions: 0.21.0 Environment: Linux (Ubuntu 14.04.1) Reporter: Andrew Ortman Priority: Minor Port the CFS support the Mesos Containerizer has to the Docker Containerizer. Whenever the --cgroup_enable_cfs flag is set, the Docker Containerizer should update the cfs_period_us and cfs_quota_us values to allow hard CPU capping on the container. Current workaround is to pass those values as LXC configuration parameters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2664) Modernize the codebase to C++11
[ https://issues.apache.org/jira/browse/MESOS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599114#comment-14599114 ] Adam B commented on MESOS-2664: --- Retargeting this epic to 0.24, since we won't complete all of the issues in time for 0.23. The issues that have been resolved will be listed individually in the CHANGELOG, but we'll have to wait at least one more release before we can claim to have modernized the codebase to C++11. Great work so far though! Modernize the codebase to C++11 --- Key: MESOS-2664 URL: https://issues.apache.org/jira/browse/MESOS-2664 Project: Mesos Issue Type: Epic Components: technical debt Reporter: Michael Park Assignee: Michael Park Labels: mesosphere Since [this commit|https://github.com/apache/mesos/commit/0f5c78fad3423181f7227027eb42d162811514e7], we officially require GCC-4.8+ and Clang-3.5+. This means that we now have full C++11 support and therefore can start to modernize our codebase to be more readable, safer and efficient! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-1807: -- Target Version/s: 0.24.0 (was: 0.23.0) Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599058#comment-14599058 ] Adam B commented on MESOS-1807: --- Thanks for the explanation, Vinod. I'm targeting this out of 0.23 since nobody is actively working on it. Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2664) Modernize the codebase to C++11
[ https://issues.apache.org/jira/browse/MESOS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-2664: -- Target Version/s: 0.24.0 (was: 0.23.0) Modernize the codebase to C++11 --- Key: MESOS-2664 URL: https://issues.apache.org/jira/browse/MESOS-2664 Project: Mesos Issue Type: Epic Components: technical debt Reporter: Michael Park Assignee: Michael Park Labels: mesosphere Since [this commit|https://github.com/apache/mesos/commit/0f5c78fad3423181f7227027eb42d162811514e7], we officially require GCC-4.8+ and Clang-3.5+. This means that we now have full C++11 support and therefore can start to modernize our codebase to be more readable, safer and efficient! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (MESOS-1153) configure does not check for patch
[ https://issues.apache.org/jira/browse/MESOS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kapil Arya closed MESOS-1153. - Resolution: Fixed commit 349884a9159c6dc62d82a9ddd6e0dca11df2e287 Author: haosdent huang haosd...@gmail.com Date: Mon Apr 27 01:55:02 2015 +0200 Added check for availability of the patch command in configure.ac. Review: https://reviews.apache.org/r/33266 configure does not check for patch -- Key: MESOS-1153 URL: https://issues.apache.org/jira/browse/MESOS-1153 Project: Mesos Issue Type: Bug Components: build Environment: Linux Reporter: Vinson Lee The build uses the patch command but configure never checks for it's existence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2894) web UI shows YYYY for year instead of year
[ https://issues.apache.org/jira/browse/MESOS-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-2894: -- Assignee: haosdent web UI shows for year instead of year Key: MESOS-2894 URL: https://issues.apache.org/jira/browse/MESOS-2894 Project: Mesos Issue Type: Bug Components: webui Reporter: brian wickman Assignee: haosdent Priority: Minor Labels: newbie The directory listing formats on when it should probably format on : {noformat} drwxr-xr-x3 ads twitter 4 KBJun 18 15:27 .pex drwxr-xr-x2 ads twitter 4 KBDec 01 aurora {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2894) web UI shows YYYY for year instead of year
[ https://issues.apache.org/jira/browse/MESOS-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598994#comment-14598994 ] Adam B commented on MESOS-2894: --- Review: https://reviews.apache.org/r/35782 web UI shows for year instead of year Key: MESOS-2894 URL: https://issues.apache.org/jira/browse/MESOS-2894 Project: Mesos Issue Type: Bug Components: webui Reporter: brian wickman Assignee: haosdent Priority: Minor Labels: newbie The directory listing formats on when it should probably format on : {noformat} drwxr-xr-x3 ads twitter 4 KBJun 18 15:27 .pex drwxr-xr-x2 ads twitter 4 KBDec 01 aurora {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2670) Update existing lambdas to meet style guide
[ https://issues.apache.org/jira/browse/MESOS-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599107#comment-14599107 ] Adam B commented on MESOS-2670: --- [~jvanremoortere] [~benjaminhindman] Now that the above patches have been committed, is there anything left to do before we can close this ticket? Update existing lambdas to meet style guide --- Key: MESOS-2670 URL: https://issues.apache.org/jira/browse/MESOS-2670 Project: Mesos Issue Type: Task Reporter: Joris Van Remoortere Assignee: haosdent Labels: c++11 There are already some lambdas in C++11 specific files. Modify these to meet the updated style guide. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2632) Remove capture by reference of temporaries in mesos
[ https://issues.apache.org/jira/browse/MESOS-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599045#comment-14599045 ] Adam B commented on MESOS-2632: --- [~jvanremoortere] Do you think we'll get this into 0.23, or should we push it out to the next release? Remove capture by reference of temporaries in mesos --- Key: MESOS-2632 URL: https://issues.apache.org/jira/browse/MESOS-2632 Project: Mesos Issue Type: Task Components: technical debt Reporter: Joris Van Remoortere -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MESOS-2894) web UI shows YYYY for year instead of year
[ https://issues.apache.org/jira/browse/MESOS-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B resolved MESOS-2894. --- Resolution: Fixed Fix Version/s: 0.23.0 commit d86341b216dc9bb0f97663007c04d147b663f427 Author: haosdent huang haosd...@gmail.com Date: Wed Jun 24 01:29:57 2015 -0700 Fixed web UI showing for year instead of year. Review: https://reviews.apache.org/r/35782 web UI shows for year instead of year Key: MESOS-2894 URL: https://issues.apache.org/jira/browse/MESOS-2894 Project: Mesos Issue Type: Bug Components: webui Reporter: brian wickman Assignee: haosdent Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: MESOS-2894-after.png, MESOS-2894-before.png The directory listing formats on when it should probably format on : {noformat} drwxr-xr-x3 ads twitter 4 KBJun 18 15:27 .pex drwxr-xr-x2 ads twitter 4 KBDec 01 aurora {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2572) Add memory statistics tests.
[ https://issues.apache.org/jira/browse/MESOS-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599051#comment-14599051 ] Adam B commented on MESOS-2572: --- [~jieyu] [~idownes] Do either of you have time to review these patches? Looks like [~chzhcn] updated them recently after Ian's latest reviews. They've been sitting around for a while now, and they won't make it into 0.23 without some attention. Add memory statistics tests. Key: MESOS-2572 URL: https://issues.apache.org/jira/browse/MESOS-2572 Project: Mesos Issue Type: Task Reporter: Chi Zhang Assignee: Chi Zhang Labels: twitter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2757) Can't Use - operator with OptionT
[ https://issues.apache.org/jira/browse/MESOS-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599102#comment-14599102 ] Adam B commented on MESOS-2757: --- [~jvanremoortere] This review was discarded (no reason given). Could you please explain why and update this ticket if it's no longer reviewable? Is it still an issue? Can't Use - operator with OptionT Key: MESOS-2757 URL: https://issues.apache.org/jira/browse/MESOS-2757 Project: Mesos Issue Type: Improvement Components: stout Reporter: Joris Van Remoortere Assignee: Joris Van Remoortere Labels: c++11, mesosphere, option, stout Let's add operator overloads to OptionT to allow access to the underlying T using the `-` operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2673) Follow Google Style Guide for header file include order completely.
[ https://issues.apache.org/jira/browse/MESOS-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599195#comment-14599195 ] Adam B commented on MESOS-2673: --- [~js84] If, as described in the gist, this Epic describes not just updating the style guide, but also making the changes in stout/libprocess/mesos, then I highly doubt that we will complete this Epic in time for 0.23. Could you please retarget to 0.24 if appropriate? Thanks! Follow Google Style Guide for header file include order completely. --- Key: MESOS-2673 URL: https://issues.apache.org/jira/browse/MESOS-2673 Project: Mesos Issue Type: Improvement Reporter: Joerg Schad Assignee: Joerg Schad Priority: Minor Labels: mesosphere Fix For: 0.23.0 The header include order for Mesos actually follows the Google Styleguide but omits step 1 without mentioning this exception in the Mesos styleguide. This proposal suggests to adapt to the include order explained in the Google Styleguide i.e. include the direct headers first in the .cpp files implementing them. A gist of the proposal can be found here: https://gist.github.com/joerg84/65cb9611d24b2e35b69b The corresponding Review Board review can be found here: https://reviews.apache.org/r/33646/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2888) Add SSL socket tests
[ https://issues.apache.org/jira/browse/MESOS-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-2888: -- Target Version/s: 0.23.0 Add SSL socket tests Key: MESOS-2888 URL: https://issues.apache.org/jira/browse/MESOS-2888 Project: Mesos Issue Type: Improvement Components: libprocess Reporter: Joris Van Remoortere Assignee: Joris Van Remoortere Labels: libprocess, mesosphere, ssl, tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2884) Allow isolators to specify required namespaces
[ https://issues.apache.org/jira/browse/MESOS-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599202#comment-14599202 ] Adam B commented on MESOS-2884: --- [~karya] These RRs have been committed. Is there anything else left to do for this ticket? commit 1a82a3fb2bc717c468218384190a115b770f88c3 Author: Kapil Arya ka...@mesosphere.io Date: Tue Jun 23 12:33:41 2015 -0700 Updated LinuxLauncher to receive list of namespaces. MesosContainerizer looks up the list of required namespaces by calling Isolator::namespaces() for all enabled isolators and passes on this value to LinuxLauncher. Review: https://reviews.apache.org/r/35586 commit 2143ae0315990ed663bf5810a801adeacff3a986 Author: Kapil Arya ka...@mesosphere.io Date: Tue Jun 23 12:32:32 2015 -0700 Updated Isolator to return required namespaces. This would enable the MesosContainerizer to pass on a list of namespaces to LinuxLauncher instead of having LinuxLauncher guess it from the isolation flags. Review: https://reviews.apache.org/r/35585 Allow isolators to specify required namespaces -- Key: MESOS-2884 URL: https://issues.apache.org/jira/browse/MESOS-2884 Project: Mesos Issue Type: Task Components: isolation Reporter: Kapil Arya Assignee: Kapil Arya Labels: mesosphere Currently, the LinuxLauncher looks into SlaveFlags to compute the namespaces that should be enabled when launching the executor. This means that a custom Isolator module doesn't have any way to specify dependency on a set of namespaces. The proposed solution is to extend the Isolator interface to also export the namespaces dependency. This way the MesosContainerizer can directly query all loaded Isolators (inbuilt and custom modules) to compute the set of namespaces required by the executor. This set of namespaces is then passed on to the LinuxLauncher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2637) Consolidate 'foo', 'bar', ... string constants in test and example code
[ https://issues.apache.org/jira/browse/MESOS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600087#comment-14600087 ] Colin Williams commented on MESOS-2637: --- I'll do that, thanks for your help so far. On Jun 24, 2015 2:19 PM, Niklas Quarfot Nielsen (JIRA) j...@apache.org Consolidate 'foo', 'bar', ... string constants in test and example code --- Key: MESOS-2637 URL: https://issues.apache.org/jira/browse/MESOS-2637 Project: Mesos Issue Type: Bug Components: technical debt Reporter: Niklas Quarfot Nielsen Assignee: Colin Williams We are using 'foo', 'bar', ... string constants and pairs in src/tests/master_tests.cpp, src/tests/slave_tests.cpp, src/tests/hook_tests.cpp and src/examples/test_hook_module.cpp for label and hooks tests. These values should be stored in local variables to avoid the possibility of assignment getting out of sync with checking for that same value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-999) Slave should wait() and start executor registration timeout after launch
[ https://issues.apache.org/jira/browse/MESOS-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600134#comment-14600134 ] Yan Xu commented on MESOS-999: -- So we ended up not taking on this. Our motivation for addressing this was because with docker and other new containerization efforts such as MESOS-2386, some considerable amount of time can be spent on pulling and preparing the container images before the executor is launched so we don't want the slave to kill it due to a small {{--executor_registration_timeout}}. While implementing this we realized that adding a slave level launch timeout flag doesn't solve the fundamental problem of the long image preparation having a noticeable impact on the slave and its tasks. We should instead working towards a solution that minimizes such impact. The {{--executor_registration_timeout}} flag was originally introduced to account for the time required to fetch the executor so for now giving it a large value is the reasonable way for tasks that use container images. Ultimately only the task knows what the expected preparation time is so such timeouts should probably go to ExecutorInfo or TaskInfo. I feel like this ticket as it is currently phrased could be closed as {{Won't Fix}}. Do you agree [~idownes]? Slave should wait() and start executor registration timeout after launch - Key: MESOS-999 URL: https://issues.apache.org/jira/browse/MESOS-999 Project: Mesos Issue Type: Bug Components: isolation Affects Versions: 0.18.0 Reporter: Ian Downes Assignee: Yan Xu Priority: Minor The current code will start launch a container and wait on it before the launch is complete. We should do this only after the container has successfully launched. Likewise for the executor registration timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1187) precision errors with allocation calculations
[ https://issues.apache.org/jira/browse/MESOS-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-1187: -- Priority: Major (was: Minor) precision errors with allocation calculations - Key: MESOS-1187 URL: https://issues.apache.org/jira/browse/MESOS-1187 Project: Mesos Issue Type: Bug Components: allocation, master Reporter: aniruddha sathaye As allocations are stored/transmitted as doubles many a times precision errors creep in. we have seen erroneous share calculations happen only because of floating point arithmetic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-2889) Add SSL switch to python configuration
[ https://issues.apache.org/jira/browse/MESOS-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Harutyunyan reassigned MESOS-2889: Assignee: Artem Harutyunyan (was: Joris Van Remoortere) Add SSL switch to python configuration -- Key: MESOS-2889 URL: https://issues.apache.org/jira/browse/MESOS-2889 Project: Mesos Issue Type: Bug Reporter: Joris Van Remoortere Assignee: Artem Harutyunyan Labels: mesosphere The python egg requires explicit dependencies for SSL. Add these to the python configuration if ssl is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
[ https://issues.apache.org/jira/browse/MESOS-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600121#comment-14600121 ] Paul Brett commented on MESOS-2925: --- [~jvanremoortere] - I think the init macro in the initializer list use looks much better but the compiler warns against it because the behavior is undefined and therefore unsafe. BTW, I'm using clang on Linux, so I don't know if the proposed Apple tweak would help me. Invalid usage of ATOMIC_FLAG_INIT in member initialization -- Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-999) Slave should wait() and start executor registration timeout after launch
[ https://issues.apache.org/jira/browse/MESOS-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Xu updated MESOS-999: - Labels: (was: twitter) Slave should wait() and start executor registration timeout after launch - Key: MESOS-999 URL: https://issues.apache.org/jira/browse/MESOS-999 Project: Mesos Issue Type: Bug Components: isolation Affects Versions: 0.18.0 Reporter: Ian Downes Priority: Minor The current code will start launch a container and wait on it before the launch is complete. We should do this only after the container has successfully launched. Likewise for the executor registration timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600254#comment-14600254 ] Alex Clemmer commented on MESOS-898: [~vinodkone]: {quote} -- Looks like you are adding windows support too (I'm guessing that's your main motivation to work on this? Stoked to see this btw!). It's probably better to work on WIN support in a separate patch. {quote} First, yes, I'm attempting to contribute Windows (and Windows container) support to the Mesos project, and yes, this is the first big step. But, please don't spread it around to too many people just yet -- to the extent possible, I am hoping to avoid impacting Mesosphere's ability to effectively launch the feature. :) To your suggestion that we split Windows support into a different patch: it seems like you're saying it might be better to first contribute a baseline CMake-based build system, and then extend that to support Windows later. (That is, I assume you're not talking about how to package up the changes to the C++ code that we will need in order to support Windows, which I assume we all agree belong in a different patch.) Is that all right? I'm actually hoping to make x-plat support out of the box a core goal of this new build system. My rationale is basically twofold: (1) I am basically certain that I will end up developing a very different build system if we don't make x-plat support a priority right out of the gate (right now, I'm testing that each iteration works on both Linux and Windows, which definitely guides the design a lot), and (2) I've taken care to clearly mark off the Windows-specific stuff so that we can pull it out easily later if this Windows stuff doesn't work out. More particularly, I am suggesting this course of action because I think it will result in the least aggregate work in expectation. That all said, I understand the hesitation to put stuff into the codebase if it isn't used, and I'm totally willing to be convinced that I'm wrong. Let me know if you see something I don't. To your other issues, that boost issue that you and [~xujyan] is good to know, so thanks for pointing it out. And, I was actually hoping that one of you had good knowledge of CMake idioms. :) If we don't have someone with that expertise on deck, I agree we need to go find it before merge. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600263#comment-14600263 ] Cody Maloney commented on MESOS-898: I would suggest that with the move to CMake we switch to using a raw upstream packaged version of boost. There isn't a lot we gain by stripping out some of the headers, and it adds a lot more complexity. CMake has a lot of stuff ready-made for finding, downloading boost if and only if it isn't present on the host machine, isn't of the right version, etc. Forcing rebuilding all of that logic/code so that we can remove some files in a tarball which shouldn't be embedded inside the repository anyways seems like not the best idea. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
[ https://issues.apache.org/jira/browse/MESOS-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600084#comment-14600084 ] Joris Van Remoortere commented on MESOS-2925: - Hi [~pbrett]. Clang 3.7 does error out on OSX due to the braced scaled initializer warning. According to [~tillt] this is something that Apple tends to tweak before they provide the stable releases of clang for OSX? AFAIK the macro expands to either `{false}` or {0}. I don't see why it is harmful to keep using this to initialize the locks. I find the code in the review request harder to read due to the noise. What do you think? Invalid usage of ATOMIC_FLAG_INIT in member initialization -- Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MESOS-1187) precision errors with allocation calculations
[ https://issues.apache.org/jira/browse/MESOS-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu reopened MESOS-1187: --- I was able to reproduce. 0.1 + 0.1 + 0.1 - 0.1 - 0.1 = 0.09892 0.1 That would cause issues if the code relies on the fact that 0.1 + 0.1 + 0.1 - 0.1 - 0.1 == 0.1 precision errors with allocation calculations - Key: MESOS-1187 URL: https://issues.apache.org/jira/browse/MESOS-1187 Project: Mesos Issue Type: Bug Components: allocation, master Reporter: aniruddha sathaye Priority: Minor As allocations are stored/transmitted as doubles many a times precision errors creep in. we have seen erroneous share calculations happen only because of floating point arithmetic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1187) precision errors with allocation calculations
[ https://issues.apache.org/jira/browse/MESOS-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600257#comment-14600257 ] Jie Yu commented on MESOS-1187: --- This is a simple test to reproduce: https://reviews.apache.org/r/35849/ precision errors with allocation calculations - Key: MESOS-1187 URL: https://issues.apache.org/jira/browse/MESOS-1187 Project: Mesos Issue Type: Bug Components: allocation, master Reporter: aniruddha sathaye As allocations are stored/transmitted as doubles many a times precision errors creep in. we have seen erroneous share calculations happen only because of floating point arithmetic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600293#comment-14600293 ] Alex Clemmer commented on MESOS-898: [~vi...@twitter.com] I would be very, very disappointed in any Apache project that did not have community involvement in a decision like this. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2637) Consolidate 'foo', 'bar', ... string constants in test and example code
[ https://issues.apache.org/jira/browse/MESOS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklas Quarfot Nielsen updated MESOS-2637: -- Shepherd: (was: Niklas Quarfot Nielsen) Consolidate 'foo', 'bar', ... string constants in test and example code --- Key: MESOS-2637 URL: https://issues.apache.org/jira/browse/MESOS-2637 Project: Mesos Issue Type: Bug Components: technical debt Reporter: Niklas Quarfot Nielsen Assignee: Colin Williams We are using 'foo', 'bar', ... string constants and pairs in src/tests/master_tests.cpp, src/tests/slave_tests.cpp, src/tests/hook_tests.cpp and src/examples/test_hook_module.cpp for label and hooks tests. These values should be stored in local variables to avoid the possibility of assignment getting out of sync with checking for that same value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-999) Slave should wait() and start executor registration timeout after launch
[ https://issues.apache.org/jira/browse/MESOS-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Xu updated MESOS-999: - Assignee: (was: Yan Xu) Slave should wait() and start executor registration timeout after launch - Key: MESOS-999 URL: https://issues.apache.org/jira/browse/MESOS-999 Project: Mesos Issue Type: Bug Components: isolation Affects Versions: 0.18.0 Reporter: Ian Downes Priority: Minor The current code will start launch a container and wait on it before the launch is complete. We should do this only after the container has successfully launched. Likewise for the executor registration timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600279#comment-14600279 ] Vinod Kone commented on MESOS-898: -- You know that this JIRA is public right :) Also, since this support is being added to Apache Mesos, we expect the community to be in the know during development of changes like these. Regarding cross platform support off the bat, I'll trust your instincts on this. From what I saw in your commits, windows support didn't seem to complicate things. So, it's ok with me. Regarding CMake idioms, [~tstclair] (Mesos committer), might be able to help you out here. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
[ https://issues.apache.org/jira/browse/MESOS-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600084#comment-14600084 ] Joris Van Remoortere edited comment on MESOS-2925 at 6/24/15 8:26 PM: -- Hi [~pbrett]. Clang 3.7 does error out on OSX due to the braced scaled initializer warning. According to [~tillt] this is something that Apple tends to tweak before they provide the stable releases of clang for OSX? AFAIK the macro expands to either \{false\} or \{0\}. I don't see why it is harmful to keep using this to initialize the locks. I find the code in the review request harder to read due to the noise. What do you think? was (Author: jvanremoortere): Hi [~pbrett]. Clang 3.7 does error out on OSX due to the braced scaled initializer warning. According to [~tillt] this is something that Apple tends to tweak before they provide the stable releases of clang for OSX? AFAIK the macro expands to either `{false}` or {0}. I don't see why it is harmful to keep using this to initialize the locks. I find the code in the review request harder to read due to the noise. What do you think? Invalid usage of ATOMIC_FLAG_INIT in member initialization -- Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
[ https://issues.apache.org/jira/browse/MESOS-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600104#comment-14600104 ] Paul Brett commented on MESOS-2925: --- Up for review at https://reviews.apache.org/r/35841/ Invalid usage of ATOMIC_FLAG_INIT in member initialization -- Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2924) Allow simple construction via initializer list on hashset.
Till Toenshoff created MESOS-2924: - Summary: Allow simple construction via initializer list on hashset. Key: MESOS-2924 URL: https://issues.apache.org/jira/browse/MESOS-2924 Project: Mesos Issue Type: Improvement Reporter: Till Toenshoff {{hashmap}} already has a initializer-list constructor, {{hashset}} should offer the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2924) Allow simple construction via initializer list on hashset.
[ https://issues.apache.org/jira/browse/MESOS-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Toenshoff updated MESOS-2924: -- Priority: Minor (was: Major) Allow simple construction via initializer list on hashset. -- Key: MESOS-2924 URL: https://issues.apache.org/jira/browse/MESOS-2924 Project: Mesos Issue Type: Improvement Reporter: Till Toenshoff Priority: Minor {{hashmap}} already has a initializer-list constructor, {{hashset}} should offer the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2923) fetcher.cpp - problem with certificates..?
Tomasz Mieszkowski created MESOS-2923: - Summary: fetcher.cpp - problem with certificates..? Key: MESOS-2923 URL: https://issues.apache.org/jira/browse/MESOS-2923 Project: Mesos Issue Type: Bug Affects Versions: 0.22.1, 0.22.0 Environment: Ubuntu 14.04 (build + test) Reporter: Tomasz Mieszkowski Mesos 0.22.0/0.22.1 built and installed from sources accordingly to the instructions given [here|http://mesos.apache.org/gettingstarted/] has some problem with certificates. Every time I try to deploy something that requires downloading any resource via HTTPS (with URI specified via Marathon), such deployment fails and I get this message in failed app's sandbox: {code} E0617 09:58:44.339409 12380 fetcher.cpp:138] Error downloading resource: Problem with the SSL CA cert (path? access rights?) {code} Trying to download the same resource on the same slave with {{curl}} or {{wget}} works without problems. Moreover, when I install exactly the same version of Mesos from Mesosphere's debs on identical machines (i.e., set up by the same Ansible scripts), everything works fine as well. I guess it must be something related to the way how Mesos is built - maybe some missing switch for {{configure}} or {{make}}..? Any ideas..? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo
[ https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600410#comment-14600410 ] Charles Allen commented on MESOS-2340: -- Is rolling back from 0.24 to 0.23 during a failed rolling upgrade or immediately after something that is intended to be supported? Publish JSON in ZK instead of serialized MasterInfo --- Key: MESOS-2340 URL: https://issues.apache.org/jira/browse/MESOS-2340 Project: Mesos Issue Type: Improvement Components: leader election Reporter: Zameer Manji Assignee: Marco Massenzio Currently to discover the master a client needs the ZK node location and access to the MasterInfo protobuf so it can deserialize the binary blob in the node. I think it would be nice to publish JSON (like Twitter's ServerSets) so clients are not tied to protobuf to do service discovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2931) Add explanation of master bootstrap process to Operational Guide
[ https://issues.apache.org/jira/browse/MESOS-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Nugent updated MESOS-2931: - Description: When Mesos starts up, masters come up in an empty bootstrap state. End users may find that they experience failures without an apparent cause. The documentation for the operational guide should lay out the requirements for the bootstrap process and its behavior. See MESOS-2148 for an example of an issue filed when someone wasn't aware of the bootstrap process. was: When Mesos starts up, masters come up in an empty bootstrap state. End users may find that they experience failures without an apparent cause. The documentation for the operational guide should lay out the requirements for the bootstrap process and its behavior. See MESOS-2148 for an example of an issue filed when someone wasn't aware of the problem. Add explanation of master bootstrap process to Operational Guide Key: MESOS-2931 URL: https://issues.apache.org/jira/browse/MESOS-2931 Project: Mesos Issue Type: Documentation Components: documentation Affects Versions: 0.22.1 Reporter: Daniel Nugent Priority: Minor When Mesos starts up, masters come up in an empty bootstrap state. End users may find that they experience failures without an apparent cause. The documentation for the operational guide should lay out the requirements for the bootstrap process and its behavior. See MESOS-2148 for an example of an issue filed when someone wasn't aware of the bootstrap process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo
[ https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600350#comment-14600350 ] Marco Massenzio commented on MESOS-2340: Update sent to [the user mailing list](http://www.mail-archive.com/user@mesos.apache.org/msg03691.html) Publish JSON in ZK instead of serialized MasterInfo --- Key: MESOS-2340 URL: https://issues.apache.org/jira/browse/MESOS-2340 Project: Mesos Issue Type: Improvement Components: leader election Reporter: Zameer Manji Assignee: Marco Massenzio Currently to discover the master a client needs the ZK node location and access to the MasterInfo protobuf so it can deserialize the binary blob in the node. I think it would be nice to publish JSON (like Twitter's ServerSets) so clients are not tied to protobuf to do service discovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2926) Extend mesos-style.py/cpplint.py to check #include files
Paul Brett created MESOS-2926: - Summary: Extend mesos-style.py/cpplint.py to check #include files Key: MESOS-2926 URL: https://issues.apache.org/jira/browse/MESOS-2926 Project: Mesos Issue Type: Bug Reporter: Paul Brett Assignee: Paul Brett cpplint.py provides the capability to enforce the style guide requirements for #including everything you use and ordering files based on type but it does not work for mesos because we do use #include ... for project files where it expects #include We should update the style checker to support our include usage and then turn it on by default in the commit hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2832) Enable configuring Mesos with environment variables without having them leak to tasks launched
[ https://issues.apache.org/jira/browse/MESOS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Hindman updated MESOS-2832: Story Points: 8 Enable configuring Mesos with environment variables without having them leak to tasks launched -- Key: MESOS-2832 URL: https://issues.apache.org/jira/browse/MESOS-2832 Project: Mesos Issue Type: Wish Reporter: Cody Maloney Assignee: Benjamin Hindman Priority: Critical Labels: mesosphere Fix For: 0.23.0 Currently if mesos is configured with environment variables (MESOS_MODULES), those show up in every task which is launched unless the executor explicitly cleans them up. If the task being launched happens to be something libprocess / mesos based, this can often prevent the task from starting up (A scheduler has issues loading a module intended for the slave). There are also cases where it would be nice to be able to change what the PATH is that tasks launch with (the host may have more in the path than tasks are supposed to / allowed to depend upon). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2824) Support pre-fetching default container image on slave startup
[ https://issues.apache.org/jira/browse/MESOS-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600356#comment-14600356 ] Yan Xu commented on MESOS-2824: --- We ([~vinodkone], [~idownes] and myself) discussed the pre-fetching strategies but it seemed unacceptable to have either - the slave to be blocked for extended period of time (minutes) which delays the communication between the executor and scheduler, or - the first task that uses this image to be blocked for a long time to wait for the container image to be ready. Pre-fetching, either started automatically or upon explicit requests is a good idea but we'll probably not do it on slave startup. Support pre-fetching default container image on slave startup - Key: MESOS-2824 URL: https://issues.apache.org/jira/browse/MESOS-2824 Project: Mesos Issue Type: Improvement Components: isolation Affects Versions: 0.23.0 Reporter: Ian Downes Assignee: Yan Xu Priority: Minor Labels: twitter Default container images can be specified with the --default_container_info flag to the slave. This may be a large image that will take a long time to initially fetch/hash/extract when the first container is provisioned. Add optional support to start fetching the image when the slave starts and consider not registering until the fetch is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2927) Update mesos to #include headers for symbols we rely on
Paul Brett created MESOS-2927: - Summary: Update mesos to #include headers for symbols we rely on Key: MESOS-2927 URL: https://issues.apache.org/jira/browse/MESOS-2927 Project: Mesos Issue Type: Bug Reporter: Paul Brett Assignee: Paul Brett Update mesos to #include headers for symbols we rely on -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2929) Update libprocess to #include headers for symbols we rely on
Paul Brett created MESOS-2929: - Summary: Update libprocess to #include headers for symbols we rely on Key: MESOS-2929 URL: https://issues.apache.org/jira/browse/MESOS-2929 Project: Mesos Issue Type: Bug Reporter: Paul Brett Assignee: Paul Brett -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo
[ https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600428#comment-14600428 ] Vinod Kone commented on MESOS-2340: --- 0.23.0 and 0.24.0 components (scheduler, master) should always be able to talk to each other. Publish JSON in ZK instead of serialized MasterInfo --- Key: MESOS-2340 URL: https://issues.apache.org/jira/browse/MESOS-2340 Project: Mesos Issue Type: Improvement Components: leader election Reporter: Zameer Manji Assignee: Marco Massenzio Currently to discover the master a client needs the ZK node location and access to the MasterInfo protobuf so it can deserialize the binary blob in the node. I think it would be nice to publish JSON (like Twitter's ServerSets) so clients are not tied to protobuf to do service discovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MESOS-2832) Enable configuring Mesos with environment variables without having them leak to tasks launched
[ https://issues.apache.org/jira/browse/MESOS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Hindman resolved MESOS-2832. - Resolution: Fixed Fix Version/s: 0.23.0 commit 46dc9979e9cc38d36dc7300db13af39bdfbfd52e Author: Benjamin Hindman benjamin.hind...@gmail.com Date: Mon Jun 15 00:29:20 2015 -0700 Added 'executor_environment_variables' flag to slave. This new flag, 'executor_environment_variables', let's an operator specify the environment variables that should be passed to an executor, and thus, any of it's tasks. By default, an executor will inherit the environment variables of the slave. Review: https://reviews.apache.org/r/35567 Enable configuring Mesos with environment variables without having them leak to tasks launched -- Key: MESOS-2832 URL: https://issues.apache.org/jira/browse/MESOS-2832 Project: Mesos Issue Type: Wish Reporter: Cody Maloney Assignee: Benjamin Hindman Priority: Critical Labels: mesosphere Fix For: 0.23.0 Currently if mesos is configured with environment variables (MESOS_MODULES), those show up in every task which is launched unless the executor explicitly cleans them up. If the task being launched happens to be something libprocess / mesos based, this can often prevent the task from starting up (A scheduler has issues loading a module intended for the slave). There are also cases where it would be nice to be able to change what the PATH is that tasks launch with (the host may have more in the path than tasks are supposed to / allowed to depend upon). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2931) Add explanation of master bootstrap process to Operational Guide
Daniel Nugent created MESOS-2931: Summary: Add explanation of master bootstrap process to Operational Guide Key: MESOS-2931 URL: https://issues.apache.org/jira/browse/MESOS-2931 Project: Mesos Issue Type: Documentation Components: documentation Affects Versions: 0.22.1 Reporter: Daniel Nugent Priority: Minor When Mesos starts up, masters come up in an empty bootstrap state. End users may find that they experience failures without an apparent cause. The documentation for the operational guide should lay out the requirements for the bootstrap process and its behavior. See MESOS-2148 for an example of an issue filed when someone wasn't aware of the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2930) Allow the Resource Estimator to express over-allocation of revocable resources.
Benjamin Mahler created MESOS-2930: -- Summary: Allow the Resource Estimator to express over-allocation of revocable resources. Key: MESOS-2930 URL: https://issues.apache.org/jira/browse/MESOS-2930 Project: Mesos Issue Type: Improvement Components: slave Reporter: Benjamin Mahler Currently the resource estimator returns the amount of oversubscription resources that are available, since resources cannot be negative, this allows the resource estimator to express the following: (1) Return empty resources: We are fully allocated for oversubscription resources. (2) Return non-empty resources: We are under-allocated for oversubscription resources. In other words, some are available. However, there is an additional situation that we cannot express: (3) Analogous to returning non-empty negative resources: We are over-allocated for oversubscription resources. Do not re-offer any of the over-allocated oversubscription resources that are recovered. Without (3), the slave can only shrink the total pool of oversubscription resources by returning (1) as resources are recovered, until the pool is shrunk to the desired size. However, this approach is only best-effort, it's possible for a framework to launch more tasks in the window of time (15 seconds by default) that the slave polls the estimator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-2928) Update stout to #include headers for symbols we rely on
[ https://issues.apache.org/jira/browse/MESOS-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600455#comment-14600455 ] Paul Brett edited comment on MESOS-2928 at 6/25/15 12:58 AM: - https://reviews.apache.org/r/35861/ was (Author: pbrett): https://reviews.apache.org/r/35860/ Update stout to #include headers for symbols we rely on --- Key: MESOS-2928 URL: https://issues.apache.org/jira/browse/MESOS-2928 Project: Mesos Issue Type: Bug Reporter: Paul Brett Assignee: Paul Brett Update mesos to #include headers for symbols we rely on -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2928) Update stout to #include headers for symbols we rely on
[ https://issues.apache.org/jira/browse/MESOS-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600455#comment-14600455 ] Paul Brett commented on MESOS-2928: --- https://reviews.apache.org/r/35860/ Update stout to #include headers for symbols we rely on --- Key: MESOS-2928 URL: https://issues.apache.org/jira/browse/MESOS-2928 Project: Mesos Issue Type: Bug Reporter: Paul Brett Assignee: Paul Brett Update mesos to #include headers for symbols we rely on -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
[ https://issues.apache.org/jira/browse/MESOS-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600512#comment-14600512 ] Michael Park commented on MESOS-2925: - [~pbrett] I gave it a {{ShipIt}} on the review but how about we use the inline member initializer syntax instead? {code} class Obj { public: std::atomic_flag lock = ATOMIC_FLAG_INIT; }; {code} Invalid usage of ATOMIC_FLAG_INIT in member initialization -- Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2664) Modernize the codebase to C++11
[ https://issues.apache.org/jira/browse/MESOS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599623#comment-14599623 ] Michael Park commented on MESOS-2664: - [~bmahler] I think {{-Wswitch}} _almost_ captures what we want. Yes, to both (a) and (b). The missing part for {{-Wswitch}} for us is: (c) If we covered all the cases *and* have a {{default}}, it would be nice for that to also be a warning. When we add a new enum case, I think what we want is a warning saying you need to handle this new enum value here! rather than silently going into the {{default}} case. An example from our codebase: {code} switch (volume.mode()) { case Volume::RW: volumeConfig += :rw; break; case Volume::RO: volumeConfig += :ro; break; default: LOG(FATAL) Unknown Volume mode: volume.mode(); break; } {code} I think we can capture what we want with {{-Wswitch}} + a style guide around the use {{default}}. What do you think? P.S. We currently have {{-Wall}} turned on, which includes {{-Wswitch}} :) Modernize the codebase to C++11 --- Key: MESOS-2664 URL: https://issues.apache.org/jira/browse/MESOS-2664 Project: Mesos Issue Type: Epic Components: technical debt Reporter: Michael Park Assignee: Michael Park Labels: mesosphere Since [this commit|https://github.com/apache/mesos/commit/0f5c78fad3423181f7227027eb42d162811514e7], we officially require GCC-4.8+ and Clang-3.5+. This means that we now have full C++11 support and therefore can start to modernize our codebase to be more readable, safer and efficient! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2545) Developer guide for libprocess
[ https://issues.apache.org/jira/browse/MESOS-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599613#comment-14599613 ] Bernd Mathiske commented on MESOS-2545: --- commit 98f85e1f81b86502013fdf1783f9d8ec65eae3d0 Author: Joerg Schad jo...@mesosphere.io Date: Wed Jun 24 17:47:30 2015 +0200 Remove (almost all) html from libprocess Developer Guide. Review: https://reviews.apache.org/r/35568 Developer guide for libprocess -- Key: MESOS-2545 URL: https://issues.apache.org/jira/browse/MESOS-2545 Project: Mesos Issue Type: Documentation Components: libprocess Reporter: Bernd Mathiske Assignee: Joerg Schad Labels: documentation, mesosphere Create a developer guide for libprocess that explains the philosophy behind it and explains the most important features as well as the prevalent use patterns in Mesos with examples. This could be similar to stout/README.md. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2932) There is a typo in docs/docker-containerizer.md file
Chen Zhiwei created MESOS-2932: -- Summary: There is a typo in docs/docker-containerizer.md file Key: MESOS-2932 URL: https://issues.apache.org/jira/browse/MESOS-2932 Project: Mesos Issue Type: Bug Components: documentation Affects Versions: 0.22.1 Reporter: Chen Zhiwei Priority: Minor A docker image currently supports having a entrypoint and/or a default command. The `a entrypoint` should be `an entrypoint`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599706#comment-14599706 ] Alex Clemmer commented on MESOS-898: I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599706#comment-14599706 ] Alex Clemmer edited comment on MESOS-898 at 6/24/15 4:39 PM: - I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for screwing up the posting of this the first time. I don't interact with JIRA much. was (Author: hausdorff): I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a
[jira] [Comment Edited] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599706#comment-14599706 ] Alex Clemmer edited comment on MESOS-898 at 6/24/15 4:47 PM: - I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. * This commit spans Mesos, libprocess, and stout (rather than separating them out) mainly to make it easy to consume -- for the real solution, I will happily split them out appropriately. * So far, this prototype commit follows [~haosd...@gmail.com]'s supposition that we want to build the CMake system incrementally and in parallel (though I want to note that stout is a header-only library, and does not need to be built). * I agree that with [~tstclair] that it is probably best that we coalesce libprocess and stout into one sort of systems layer, but given the progress here so far, I'd like to postpone that discussion for another time. I'm willing to rewrite this part of the build system in the event that those changes materialize. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for double posting this. I don't interact with JIRA much. was (Author: hausdorff): I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. * This commit spans Mesos, libprocess, and stout (rather than separating them out) mainly to make it easy to consume -- for the real solution, I will happily split them out appropriately. * So far, this prototype commit follows [~haosd...@gmail.com]'s supposition that we want to build the CMake system incrementally and in parallel (though I want to note that stout is a header-only library, and does not need to be built). Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for double posting this. I don't interact with JIRA much. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599705#comment-14599705 ] Alex Clemmer commented on MESOS-898: I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599706#comment-14599706 ] Alex Clemmer edited comment on MESOS-898 at 6/24/15 4:40 PM: - I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for double posting this. I don't interact with JIRA much. was (Author: hausdorff): I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for screwing up the posting of this the first time. I don't interact with JIRA much. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've
[jira] [Comment Edited] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599706#comment-14599706 ] Alex Clemmer edited comment on MESOS-898 at 6/24/15 4:43 PM: - I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. * This commit spans Mesos, libprocess, and stout (rather than separating them out) mainly to make it easy to consume -- for the real solution, I will happily split them out appropriately. * So far, this prototype commit follows [~haosd...@gmail.com]'s supposition that we want to build the CMake system incrementally and in parallel (though I want to note that stout is a header-only library, and does not need to be built). Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for double posting this. I don't interact with JIRA much. was (Author: hausdorff): I'd like to start this conversation with (what I think is) the simplest step forward. I've put together a prototype CMake-based build system that covers libprocess and the third-party libraries, which successfully builds on several flavors of Linux. It is the [last commit of this branch|https://github.com/hausdorff/mesos/commits/test_cmake]. In general, I'm hoping that this will help focus the discussion around concrete things that I need to fix or do less stupidly, to accomplish our goals in this project. Some notes: * I DON'T KNOW CMAKE (OR AUTOCONF), so if it looks like I'm doing something silly, it's because I am! * It's pretty well-commented. I'm hoping it's not too hard to navigate. * It is build to be x-plat at the outset. So, you should be able to take this file and generate makefiles or, like, nmake files. Some things on the roadmap if you all like the progress so far: [x] Third-party dependencies are downloaded, configured, and built when you run `make` (or whatever). [ ] Support for system installations of the third-party dependencies (i.e., we should allow users to use glog if it's already installed on their machine) [x] Tests build and run on multiple flavors of Linux [ ] Benchmarks build and run on multiple flavors of Linux n.b., for issue 2 I have not decided what the right way of communicating the system dependencies to CMake is. In the autoconf solution this comes from `configure`, but CMake will not have a configure step because it needs to run not only on POSIX machines. EDIT: sorry, folks, for double posting this. I don't interact with JIRA much. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where
[jira] [Assigned] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
[ https://issues.apache.org/jira/browse/MESOS-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Brett reassigned MESOS-2925: - Assignee: Paul Brett Invalid usage of ATOMIC_FLAG_INIT in member initialization -- Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett Assignee: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2855) Update operational guide to include growing from standalone to high availability
[ https://issues.apache.org/jira/browse/MESOS-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599603#comment-14599603 ] Michael Schenck commented on MESOS-2855: https://reviews.apache.org/r/35361/ Update operational guide to include growing from standalone to high availability Key: MESOS-2855 URL: https://issues.apache.org/jira/browse/MESOS-2855 Project: Mesos Issue Type: Documentation Reporter: Michael Schenck Assignee: Michael Schenck Labels: documentation The [Operational Guide|http://mesos.apache.org/documentation/latest/operational-guide/] covers increasing quorum size from {{--quorum=2}}, but does not cover how to move from a _standalone_ master to a high availability configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599866#comment-14599866 ] Alex Clemmer edited comment on MESOS-898 at 6/24/15 6:13 PM: - Only a few people have given a quick once-over look to the prototype [CMake-based build system for libprocess|https://github.com/hausdorff/mesos/commit/073d42cea1bed8e7fb801f59685f52d3be06e510?diff=unified], but they have been pretty positive, so I would like to target EOD tomorrow (i.e., Th, June 24th) for posting something to ReviewBoard -- I don't want to let the commit hang too long, but I am also new to the community, and I want to give people a chance to build consensus that this commit is at least directionally correct. :) was (Author: hausdorff): Only a few people have given a quick once-over look to the prototype [CMake-based build system for libprocess|https://github.com/hausdorff/mesos/commit/073d42cea1bed8e7fb801f59685f52d3be06e510?diff=unified], but they have been pretty positive, so I would like to target EOD tomorrow (i.e., Th, June 24th) for posting something to ReviewBoard. This slight delay is mainly because I am new to the community, and I want to give people a chance to build consensus that this commit is at least directionally correct. :) Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2637) Consolidate 'foo', 'bar', ... string constants in test and example code
[ https://issues.apache.org/jira/browse/MESOS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599886#comment-14599886 ] Niklas Quarfot Nielsen commented on MESOS-2637: --- Hi [~lackita] - Did you get to look at this? I will be out for the next two weeks from tomorrow. If you are planning on getting reviews/feedback in the mean time - can you ping one of the other committers? Thanks! Niklas Consolidate 'foo', 'bar', ... string constants in test and example code --- Key: MESOS-2637 URL: https://issues.apache.org/jira/browse/MESOS-2637 Project: Mesos Issue Type: Bug Components: technical debt Reporter: Niklas Quarfot Nielsen Assignee: Colin Williams We are using 'foo', 'bar', ... string constants and pairs in src/tests/master_tests.cpp, src/tests/slave_tests.cpp, src/tests/hook_tests.cpp and src/examples/test_hook_module.cpp for label and hooks tests. These values should be stored in local variables to avoid the possibility of assignment getting out of sync with checking for that same value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2919) Framework can overcommit oversubscribable resources during master failover.
[ https://issues.apache.org/jira/browse/MESOS-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599902#comment-14599902 ] Jie Yu commented on MESOS-2919: --- https://reviews.apache.org/r/35836/ Framework can overcommit oversubscribable resources during master failover. --- Key: MESOS-2919 URL: https://issues.apache.org/jira/browse/MESOS-2919 Project: Mesos Issue Type: Bug Reporter: Jie Yu Assignee: Jie Yu Priority: Critical Labels: twitter This is due to a bug in the hierarchical allocator. Here is the sequence of events: 1) slave uses a fixed resource estimator which advertise 4 revocable cpus 2) a framework A launches a task that uses all the 4 revocable cpus 3) master fails over 4) slave re-registers with the new master, and sends UpdateSlaveMessage with 4 revocable cpus as oversubscribed resources 5) framework A hasn't registered yet, therefore, the slave's available resources will be 4 revocable cpus 6) framework A registered and will receive an additional 4 revocable cpus. So it can launch another task with 4 revocable cpus (that means 8 total!) The problem is due to the way we calculate 'allocated' resource in allocator when 'updateSlave'. If the framework is not registered, the 'allocation' below is not accurate (check that if block in 'addSlave'). {code} template class RoleSorter, class FrameworkSorter void HierarchicalAllocatorProcessRoleSorter, FrameworkSorter::updateSlave( const SlaveID slaveId, const Resources oversubscribed) { CHECK(initialized); CHECK(slaves.contains(slaveId)); // Check that all the oversubscribed resources are revocable. CHECK_EQ(oversubscribed, oversubscribed.revocable()); // Update the total resources. // First remove the old oversubscribed resources from the total. slaves[slaveId].total -= slaves[slaveId].total.revocable(); // Now add the new estimate of oversubscribed resources. slaves[slaveId].total += oversubscribed; // Now, update the total resources in the role sorter. roleSorter-update( slaveId, slaves[slaveId].total.unreserved()); // Calculate the current allocation of oversubscribed resources. Resources allocation; foreachkey (const std::string role, roles) { allocation += roleSorter-allocation(role, slaveId).revocable(); } // Update the available resources. // First remove the old oversubscribed resources from available. slaves[slaveId].available -= slaves[slaveId].available.revocable(); // Now add the new estimate of available oversubscribed resources. slaves[slaveId].available += oversubscribed - allocation; LOG(INFO) Slave slaveId ( slaves[slaveId].hostname ) updated with oversubscribed resources oversubscribed (total: slaves[slaveId].total , available: slaves[slaveId].available ); allocate(slaveId); } template class RoleSorter, class FrameworkSorter void HierarchicalAllocatorProcessRoleSorter, FrameworkSorter::addSlave( const SlaveID slaveId, const SlaveInfo slaveInfo, const Resources total, const hashmapFrameworkID, Resources used) { CHECK(initialized); CHECK(!slaves.contains(slaveId)); roleSorter-add(slaveId, total.unreserved()); foreachpair (const FrameworkID frameworkId, const Resources allocated, used) { if (frameworks.contains(frameworkId)) { const std::string role = frameworks[frameworkId].role; // TODO(bmahler): Validate that the reserved resources have the // framework's role. roleSorter-allocated(role, slaveId, allocated.unreserved()); frameworkSorters[role]-add(slaveId, allocated); frameworkSorters[role]-allocated( frameworkId.value(), slaveId, allocated); } } ... } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599866#comment-14599866 ] Alex Clemmer commented on MESOS-898: Only a few people have given a quick once-over look to the prototype [CMake-based build system for libprocess|https://github.com/hausdorff/mesos/commit/073d42cea1bed8e7fb801f59685f52d3be06e510?diff=unified], but they have been pretty positive, so I would like to target EOD tomorrow (i.e., Th, June 24th) for posting something to ReviewBoard. This slight delay is mainly because I am new to the community, and I want to give people a chance to build consensus that this commit is at least directionally correct. :) Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599866#comment-14599866 ] Alex Clemmer edited comment on MESOS-898 at 6/24/15 6:17 PM: - Only a few people have given a quick once-over look to the [prototype CMake-based build system for libprocess|https://github.com/hausdorff/mesos/commit/073d42cea1bed8e7fb801f59685f52d3be06e510?diff=unified], but they have been pretty positive, so I would like to target EOD tomorrow (i.e., Th, June 24th) for posting something to ReviewBoard -- I don't want to let the commit hang too long, but I am also new to the community, and I want to give people a chance to build consensus that this commit is at least directionally correct. :) was (Author: hausdorff): Only a few people have given a quick once-over look to the prototype [CMake-based build system for libprocess|https://github.com/hausdorff/mesos/commit/073d42cea1bed8e7fb801f59685f52d3be06e510?diff=unified], but they have been pretty positive, so I would like to target EOD tomorrow (i.e., Th, June 24th) for posting something to ReviewBoard -- I don't want to let the commit hang too long, but I am also new to the community, and I want to give people a chance to build consensus that this commit is at least directionally correct. :) Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2919) Framework can overcommit oversubscribable resources during master failover.
[ https://issues.apache.org/jira/browse/MESOS-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-2919: -- Story Points: 3 Framework can overcommit oversubscribable resources during master failover. --- Key: MESOS-2919 URL: https://issues.apache.org/jira/browse/MESOS-2919 Project: Mesos Issue Type: Bug Reporter: Jie Yu Assignee: Jie Yu Priority: Critical Labels: twitter This is due to a bug in the hierarchical allocator. Here is the sequence of events: 1) slave uses a fixed resource estimator which advertise 4 revocable cpus 2) a framework A launches a task that uses all the 4 revocable cpus 3) master fails over 4) slave re-registers with the new master, and sends UpdateSlaveMessage with 4 revocable cpus as oversubscribed resources 5) framework A hasn't registered yet, therefore, the slave's available resources will be 4 revocable cpus 6) framework A registered and will receive an additional 4 revocable cpus. So it can launch another task with 4 revocable cpus (that means 8 total!) The problem is due to the way we calculate 'allocated' resource in allocator when 'updateSlave'. If the framework is not registered, the 'allocation' below is not accurate (check that if block in 'addSlave'). {code} template class RoleSorter, class FrameworkSorter void HierarchicalAllocatorProcessRoleSorter, FrameworkSorter::updateSlave( const SlaveID slaveId, const Resources oversubscribed) { CHECK(initialized); CHECK(slaves.contains(slaveId)); // Check that all the oversubscribed resources are revocable. CHECK_EQ(oversubscribed, oversubscribed.revocable()); // Update the total resources. // First remove the old oversubscribed resources from the total. slaves[slaveId].total -= slaves[slaveId].total.revocable(); // Now add the new estimate of oversubscribed resources. slaves[slaveId].total += oversubscribed; // Now, update the total resources in the role sorter. roleSorter-update( slaveId, slaves[slaveId].total.unreserved()); // Calculate the current allocation of oversubscribed resources. Resources allocation; foreachkey (const std::string role, roles) { allocation += roleSorter-allocation(role, slaveId).revocable(); } // Update the available resources. // First remove the old oversubscribed resources from available. slaves[slaveId].available -= slaves[slaveId].available.revocable(); // Now add the new estimate of available oversubscribed resources. slaves[slaveId].available += oversubscribed - allocation; LOG(INFO) Slave slaveId ( slaves[slaveId].hostname ) updated with oversubscribed resources oversubscribed (total: slaves[slaveId].total , available: slaves[slaveId].available ); allocate(slaveId); } template class RoleSorter, class FrameworkSorter void HierarchicalAllocatorProcessRoleSorter, FrameworkSorter::addSlave( const SlaveID slaveId, const SlaveInfo slaveInfo, const Resources total, const hashmapFrameworkID, Resources used) { CHECK(initialized); CHECK(!slaves.contains(slaveId)); roleSorter-add(slaveId, total.unreserved()); foreachpair (const FrameworkID frameworkId, const Resources allocated, used) { if (frameworks.contains(frameworkId)) { const std::string role = frameworks[frameworkId].role; // TODO(bmahler): Validate that the reserved resources have the // framework's role. roleSorter-allocated(role, slaveId, allocated.unreserved()); frameworkSorters[role]-add(slaveId, allocated); frameworkSorters[role]-allocated( frameworkId.value(), slaveId, allocated); } } ... } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2795) Introduce filesystem provisioner abstraction to Mesos containerizer
[ https://issues.apache.org/jira/browse/MESOS-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599834#comment-14599834 ] Ian Downes commented on MESOS-2795: --- ContainerImage protobuf: https://reviews.apache.org/r/34136 Add support for container image provisioners. This includes the Provisioner interface and support for provisioners in the MesosContainerizer, including checkpointing and recovery of the optional container rootfs. https://reviews.apache.org/r/34137 Introduce filesystem provisioner abstraction to Mesos containerizer --- Key: MESOS-2795 URL: https://issues.apache.org/jira/browse/MESOS-2795 Project: Mesos Issue Type: Improvement Components: isolation Affects Versions: 0.22.1 Reporter: Ian Downes Assignee: Ian Downes Labels: twitter Optional filesystem provisioner component for the Mesos containerizer that can provision per-container filesystems. This is different to a filesystem isolators because it just provisions a root filesystem for a container and doesn't actually do any isolation (e.g., through a mount namespace + pivot or chroot). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2925) Invalid usage of ATOMIC_FLAG_INIT in member initialization
Paul Brett created MESOS-2925: - Summary: Invalid usage of ATOMIC_FLAG_INIT in member initialization Key: MESOS-2925 URL: https://issues.apache.org/jira/browse/MESOS-2925 Project: Mesos Issue Type: Bug Components: libprocess Affects Versions: 0.23.0 Reporter: Paul Brett The C++ specification states: The macro ATOMIC_FLAG_INIT shall be defined in such a way that it can be used to initialize an object of type atomic_flag to the clear state. The macro can be used in the form: atomic_flag guard = ATOMIC_FLAG_INIT; It is unspecified whether the macro can be used in other initialization contexts. Clang catches this (although reports it erroneously as a braced scaled init issue) and refuses to compile libprocess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-898) Introduce CMake as an alternative build system.
[ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599955#comment-14599955 ] Yan Xu commented on MESOS-898: -- {quote} -- Looks like you are pulling in the whole of boost? While it's not clear, we bundle the boost ourselves. It basically includes only the boost headers that we need. There should be a README somewhere that explains the process. {quote} Yeah it's super unclear. The README is hidden within the tarball. https://github.com/apache/mesos/blob/master/3rdparty/libprocess/3rdparty/boost-1.53.0.tar.gz We should move it out. Introduce CMake as an alternative build system. --- Key: MESOS-898 URL: https://issues.apache.org/jira/browse/MESOS-898 Project: Mesos Issue Type: Epic Components: build Reporter: Timothy St. Clair Assignee: Alex Clemmer Labels: build This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment. The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack. This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden. Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels yet. This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject). I do not know of any equivalent in the autotools world, other then to brew your own solution. I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. The primary benefits are: 1. Enable downstream channels to easily distro without carrying a large patch sets. 2. Still support existing non-proper distribution methods. 3. Harden / future proof dependent interfaces. Side Benefits: Audit current build mechanics. - Presently the language specific binding are not installed. (.py .jar) - make -jX currently fails - optionally look in arm support. Costs: 1. Time 2. Potential temporary destabilization 3. Infrastructure around build+test may need to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2551) C++ Scheduler library should send Call messages to Master
[ https://issues.apache.org/jira/browse/MESOS-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599969#comment-14599969 ] Marco Massenzio commented on MESOS-2551: Thanks, [~vi...@twitter.com] for taking this up. Just for historical records purposes, [this review](https://reviews.apache.org/r/35806/) was originally posted. [~vi...@twitter.com] - can you please keep [~adam-mesos] and me posted on progress: 0.23RC is coming up fast and we'd like to have this in too, if we possibly can. C++ Scheduler library should send Call messages to Master - Key: MESOS-2551 URL: https://issues.apache.org/jira/browse/MESOS-2551 Project: Mesos Issue Type: Story Reporter: Vinod Kone Assignee: Vinod Kone Labels: tech-debt Currently, the C++ library sends different messages to Master instead of a single Call message. To vet the new Call API it should send Call messages. Master should be updated to handle all types of Calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MESOS-2621) Create documentation for observability metrics
[ https://issues.apache.org/jira/browse/MESOS-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone resolved MESOS-2621. --- Resolution: Fixed Assignee: Ricardo Cervera-Navarro Fix Version/s: 0.23.0 commit f16d73852623ee05cc13d2757115f7815e608964 Author: Ricardo Cervera-Navarro rcerveranava...@twopensource.com Date: Wed Jun 24 12:01:39 2015 -0700 Added documentation on monitoring metrics and alerts. Review: https://reviews.apache.org/r/33241 Create documentation for observability metrics -- Key: MESOS-2621 URL: https://issues.apache.org/jira/browse/MESOS-2621 Project: Mesos Issue Type: Documentation Reporter: Ricardo Cervera-Navarro Assignee: Ricardo Cervera-Navarro Priority: Minor Fix For: 0.23.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2551) C++ Scheduler library should send Call messages to Master
[ https://issues.apache.org/jira/browse/MESOS-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Massenzio updated MESOS-2551: --- Labels: tech-debt twitter (was: tech-debt) C++ Scheduler library should send Call messages to Master - Key: MESOS-2551 URL: https://issues.apache.org/jira/browse/MESOS-2551 Project: Mesos Issue Type: Story Reporter: Vinod Kone Assignee: Vinod Kone Labels: tech-debt, twitter Currently, the C++ library sends different messages to Master instead of a single Call message. To vet the new Call API it should send Call messages. Master should be updated to handle all types of Calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-2551) C++ Scheduler library should send Call messages to Master
[ https://issues.apache.org/jira/browse/MESOS-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599969#comment-14599969 ] Marco Massenzio edited comment on MESOS-2551 at 6/24/15 7:08 PM: - Thanks, [~vi...@twitter.com] for taking this up. I've marked it in progress, as I'm assuming you're working on it? [~vi...@twitter.com] - can you please keep [~adam-mesos] and me posted on progress: 0.23RC is coming up fast and we'd like to have this in too, if we possibly can. (Just for historical records purposes, [this review](https://reviews.apache.org/r/35806/) was originally posted by [~ijimenez]) was (Author: marco-mesos): Thanks, [~vi...@twitter.com] for taking this up. Just for historical records purposes, [this review](https://reviews.apache.org/r/35806/) was originally posted. [~vi...@twitter.com] - can you please keep [~adam-mesos] and me posted on progress: 0.23RC is coming up fast and we'd like to have this in too, if we possibly can. C++ Scheduler library should send Call messages to Master - Key: MESOS-2551 URL: https://issues.apache.org/jira/browse/MESOS-2551 Project: Mesos Issue Type: Story Reporter: Vinod Kone Assignee: Vinod Kone Labels: tech-debt, twitter Currently, the C++ library sends different messages to Master instead of a single Call message. To vet the new Call API it should send Call messages. Master should be updated to handle all types of Calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)