[jira] [Created] (MESOS-2086) Update messages.proto to use a raw bytestream instead of a string for AuthenticationStartMessage.
Till Toenshoff created MESOS-2086:
-
Summary: Update messages.proto to use a raw bytestream instead of
a string for AuthenticationStartMessage.
Key: MESOS-2086
URL: https://issues.apache.org/jira/browse/MESOS-2086
Project: Mesos
Issue Type: Improvement
Reporter: Till Toenshoff
Assignee: Till Toenshoff
{{messages.proto}} has to be slightly changed to use a raw bytestream instead
of a string for {{AuthenticationStartMessage}} as non CRAM-MD5 authentication
mechanisms may transmit binary data.
*NOTE* that the above change does basically have no impact on C++ based proto
code other than the prevention of a warning due to non-UTF8 characters being
encoded.
For more on this subject, please see
https://www.mail-archive.com/protobuf@googlegroups.com/msg01486.html
Even though the above is pretty much determining that we wont run into
compatibility issues (old master vs. new slave and vice versa), it still should
be tested briefly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-2086) Update messages.proto to use a raw bytestream instead of a string for AuthenticationStartMessage.
[
https://issues.apache.org/jira/browse/MESOS-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207905#comment-14207905
]
Till Toenshoff commented on MESOS-2086:
---
https://reviews.apache.org/r/27494/
Update messages.proto to use a raw bytestream instead of a string for
AuthenticationStartMessage.
-
Key: MESOS-2086
URL: https://issues.apache.org/jira/browse/MESOS-2086
Project: Mesos
Issue Type: Improvement
Reporter: Till Toenshoff
Assignee: Till Toenshoff
{{messages.proto}} has to be slightly changed to use a raw bytestream instead
of a string for {{AuthenticationStartMessage}} as non CRAM-MD5 authentication
mechanisms may transmit binary data.
*NOTE* that the above change does basically have no impact on C++ based proto
code other than the prevention of a warning due to non-UTF8 characters being
encoded.
For more on this subject, please see
https://www.mail-archive.com/protobuf@googlegroups.com/msg01486.html
Even though the above is pretty much determining that we wont run into
compatibility issues (old master vs. new slave and vice versa), it still
should be tested briefly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-2086) Update messages.proto to use a raw bytestream instead of a string for AuthenticationStartMessage.
[
https://issues.apache.org/jira/browse/MESOS-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207905#comment-14207905
]
Till Toenshoff edited comment on MESOS-2086 at 11/12/14 10:36 AM:
--
https://reviews.apache.org/r/27494/ (change introduced)
https://reviews.apache.org/r/27675/ (change documented)
was (Author: tillt):
https://reviews.apache.org/r/27494/
Update messages.proto to use a raw bytestream instead of a string for
AuthenticationStartMessage.
-
Key: MESOS-2086
URL: https://issues.apache.org/jira/browse/MESOS-2086
Project: Mesos
Issue Type: Improvement
Reporter: Till Toenshoff
Assignee: Till Toenshoff
{{messages.proto}} has to be slightly changed to use a raw bytestream instead
of a string for {{AuthenticationStartMessage}} as non CRAM-MD5 authentication
mechanisms may transmit binary data.
*NOTE* that the above change does basically have no impact on C++ based proto
code other than the prevention of a warning due to non-UTF8 characters being
encoded.
For more on this subject, please see
https://www.mail-archive.com/protobuf@googlegroups.com/msg01486.html
Even though the above is pretty much determining that we wont run into
compatibility issues (old master vs. new slave and vice versa), it still
should be tested briefly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-2056) Refactor fetcher code in preparation for fetcher cache
[ https://issues.apache.org/jira/browse/MESOS-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207916#comment-14207916 ] Bernd Mathiske commented on MESOS-2056: --- This is somehow a continuation of MESOS-1316. Refactor fetcher code in preparation for fetcher cache -- Key: MESOS-2056 URL: https://issues.apache.org/jira/browse/MESOS-2056 Project: Mesos Issue Type: Improvement Components: fetcher, slave Reporter: Bernd Mathiske Assignee: Bernd Mathiske Priority: Minor Original Estimate: 72h Remaining Estimate: 72h Refactor/rearrange fetcher-related code so that cache functionality can be dropped in. One could do both together in one go. This is splitting up reviews into smaller chunks. It will not immediately be obvious how this change will be used later, but it will look better-factored and still do the exact same thing as before. In particular, a download routine to be reused several times in launcher/fetcher will be factored out and the remainder of fetcher-related code can be moved from the containerizer realm into fetcher.cpp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2087) Make libprocess support X-Mesos header on the receiving end
Dario Rexin created MESOS-2087: -- Summary: Make libprocess support X-Mesos header on the receiving end Key: MESOS-2087 URL: https://issues.apache.org/jira/browse/MESOS-2087 Project: Mesos Issue Type: Improvement Components: libprocess Reporter: Dario Rexin Priority: Minor The changes proposed in https://issues.apache.org/jira/browse/MESOS-2071 are not supported by libprocess at the receiving end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1486) Introduce an optional master whitelist for slaves
[ https://issues.apache.org/jira/browse/MESOS-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-1486: --- Summary: Introduce an optional master whitelist for slaves (was: Add authentication of masters in slaves.) Introduce an optional master whitelist for slaves - Key: MESOS-1486 URL: https://issues.apache.org/jira/browse/MESOS-1486 Project: Mesos Issue Type: Improvement Components: slave Reporter: Niklas Quarfot Nielsen Like masters can whitelist slaves (and only announce available resources from slaves whitelisted), slaves should be able to whitelist masters they are willing/allowed to connect to. I have a proof-of-concept ready which ties into the slave::detected() method and prevents non-whitelisted masters to register. If * is provided - whitelisting is not enforced (which would be the usual case). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2088) relative dates are parsed incorrectly.
Adam Shannon created MESOS-2088: --- Summary: relative dates are parsed incorrectly. Key: MESOS-2088 URL: https://issues.apache.org/jira/browse/MESOS-2088 Project: Mesos Issue Type: Bug Components: webui Affects Versions: 0.20.1 Reporter: Adam Shannon The relative-date library seems to be parsing dates without regard to the timezone in the iso 8601 timestamp. I found this file that seems to be responsible for this. https://github.com/apache/mesos/blob/master/src/webui/master/static/js/relative-date.js I would recommend using a different library that will handle this properly. http://momentjs.com/docs/#/parsing/string/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1486) Introduce an optional master whitelist for slaves
[ https://issues.apache.org/jira/browse/MESOS-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208225#comment-14208225 ] Alexander Rukletsov commented on MESOS-1486: A follow-up on this issue. We would like to authorize (not authenticate) masters in slaves. For example, if a new (rogue) master becomes a leader, a slave checks it against its list of whitelisted masters and refuses to communicate with it if it is not authorized. As a short-term solution, we can use a mechanism similar to slave whitelisting in master, which will be [deprecated in favour of ACLs|https://issues.apache.org/jira/browse/MESOS-2089]. In a long run, this feature should resemble [MESOS-1546|https://issues.apache.org/jira/browse/MESOS-1546]. Introduce an optional master whitelist for slaves - Key: MESOS-1486 URL: https://issues.apache.org/jira/browse/MESOS-1486 Project: Mesos Issue Type: Improvement Components: slave Reporter: Niklas Quarfot Nielsen Like masters can whitelist slaves (and only announce available resources from slaves whitelisted), slaves should be able to whitelist masters they are willing/allowed to connect to. I have a proof-of-concept ready which ties into the slave::detected() method and prevents non-whitelisted masters to register. If * is provided - whitelisting is not enforced (which would be the usual case). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2090) Introduce trackable JSON-based flags
Alexander Rukletsov created MESOS-2090:
--
Summary: Introduce trackable JSON-based flags
Key: MESOS-2090
URL: https://issues.apache.org/jira/browse/MESOS-2090
Project: Mesos
Issue Type: Task
Reporter: Alexander Rukletsov
Some flags represent configuration that may change over time, e.g.
{{--whitelist}} master flag. Such flags mostly contain JSON formatted data.
Provide a common mechanism to facilitate tracking changes and reloading the
flag value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1694) Future::failure should return a const string
[ https://issues.apache.org/jira/browse/MESOS-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone updated MESOS-1694: -- Story Points: 1 Future::failure should return a const string - Key: MESOS-1694 URL: https://issues.apache.org/jira/browse/MESOS-1694 Project: Mesos Issue Type: Task Components: technical debt Reporter: Dominic Hamon Assignee: Dominic Hamon Priority: Minor Labels: newbie Fix For: 0.22.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2093) Allow live-updating credential/s on a running master/slave/framework.
Till Toenshoff created MESOS-2093: - Summary: Allow live-updating credential/s on a running master/slave/framework. Key: MESOS-2093 URL: https://issues.apache.org/jira/browse/MESOS-2093 Project: Mesos Issue Type: Improvement Reporter: Till Toenshoff For improving a transition from/to an authenticated slave/framework or changes within principal/s and/or password/s, we would have to permanently watch a given credential/s file and allow the authentication logic to get re-triggered by any changes on these files on both ends (slave/framework and master). There are likely subtasks involved and hence this JIRA might get converted into an EPIC once this improvement should get implemented (if at all). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2094) Libprocess: Introduce make_shared
Joris Van Remoortere created MESOS-2094: --- Summary: Libprocess: Introduce make_shared Key: MESOS-2094 URL: https://issues.apache.org/jira/browse/MESOS-2094 Project: Mesos Issue Type: Improvement Components: libprocess Reporter: Joris Van Remoortere Assignee: Joris Van Remoortere add make_shared to the configure check -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2095) Introduce make_shared
Joris Van Remoortere created MESOS-2095: --- Summary: Introduce make_shared Key: MESOS-2095 URL: https://issues.apache.org/jira/browse/MESOS-2095 Project: Mesos Issue Type: Improvement Reporter: Joris Van Remoortere Assignee: Joris Van Remoortere add make_shared to the configure check -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2095) Introduce make_shared
[ https://issues.apache.org/jira/browse/MESOS-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208580#comment-14208580 ] Joris Van Remoortere commented on MESOS-2095: - https://reviews.apache.org/r/27921/ Introduce make_shared - Key: MESOS-2095 URL: https://issues.apache.org/jira/browse/MESOS-2095 Project: Mesos Issue Type: Improvement Reporter: Joris Van Remoortere Assignee: Joris Van Remoortere add make_shared to the configure check -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2086) Update messages.proto to use a raw bytestream instead of a string for AuthenticationStartMessage.
[
https://issues.apache.org/jira/browse/MESOS-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam B updated MESOS-2086:
--
Shepherd: Adam B
Update messages.proto to use a raw bytestream instead of a string for
AuthenticationStartMessage.
-
Key: MESOS-2086
URL: https://issues.apache.org/jira/browse/MESOS-2086
Project: Mesos
Issue Type: Improvement
Reporter: Till Toenshoff
Assignee: Till Toenshoff
{{messages.proto}} has to be slightly changed to use a raw bytestream instead
of a string for {{AuthenticationStartMessage}} as non CRAM-MD5 authentication
mechanisms may transmit binary data.
*NOTE* that the above change does basically have no impact on C++ based proto
code other than the prevention of a warning due to non-UTF8 characters being
encoded.
For more on this subject, please see
https://www.mail-archive.com/protobuf@googlegroups.com/msg01486.html
Even though the above is pretty much determining that we wont run into
compatibility issues (old master vs. new slave and vice versa), it still
should be tested briefly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-2096) Allow Mesos modules to be built outside the Mesos source tree
[ https://issues.apache.org/jira/browse/MESOS-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kapil Arya updated MESOS-2096: -- Description: We want to be able to provide a mechanism to allow third parties to build Mesos modules outside the Mesos source tree. Ideally, we would want to build Mesos modules without the need to build Mesos itself, provided that the public headers and corresponding shared libraries are available. Building a Mesos module requires access to the following components: 1. stout headers 2. libprocess headers and corresponding dynamic libraries (current, just libmesos.so) 3. protobufs 4. glog 5. gmock/gtest (optional) 6. boost Of the above components, stout headers and libprocess headers are already installed as part of standard Mesos installation. The problem lies with the bundled packages such as protobufs. It is the understanding that Mesos modules should be built with the same version of these packages with which Mesos was built. Since Mesos may have used the bundled versions, there is no way to build a module without having access to the Mesos source tree and the build directory. Having access to the source and build directory solves only part of the problem. The module build system needs to be aware of the configure parameters that were used for building Mesos (and accordingly, the bundled packages). A hackish solution is to force Mesos build system to emit the necessary information in a format that can be consumed by the module build system. A better solution is to remove the bundled packages -- protobuf, gmock, boost, and glog -- from the Mesos distribution and instead rely on system provided packages. Since these packages are usually available on most distros, we just need to do a configure check in Mesos and fail if they are not installed. Once these dependencies are resolved, we can export the header files defining module kinds such as isolator.hpp, authenticator.hpp, etc. At this point, there won't any known blockers that could prevent building modules out of Mesos tree. was: We want to be able to provide a mechanism to allow third parties to build Mesos modules outside the Mesos source tree. Ideally, we would want to build Mesos modules without the need to build Mesos itself, provided that the public headers and corresponding shared libraries are available. Building a Mesos module requires access to the following components: 1. stout headers 2. libprocess headers and corresponding dynamic libraries (current, just libmesos.so) 3. protobufs 4. glog 5. gmock/gtest (optional) 6. boost Of the above components, stout headers and libprocess headers are already installed as part of standard Mesos installation. The problem lies with the bundled versions of protobufs, etc. It is the understanding that Mesos modules should be built with the same version of these packages with which Mesos was built. Since Mesos may have used the bundled versions, there is no way to build a module without having access to Mesos source tree and the build directory. Having access to the build directory solves only part of the problem. The module build system needs to be aware of the bundled packages that were used for building Mesos. A hackish solution is to force Mesos build system to emit the necessary information in a format that can be consumed by the module build system. A better solution is to remove the bundled packages protobuf, gmock, boost, and glog from Mesos distribution and instead rely on system provided packages. Since these packages are usually available on most distros, we just need to do a configure check in Mesos and fail if they are not installed. Once these dependencies are resolved, we can export the file defining module kinds such as isolator.hpp, authenticator.hpp, etc. At this point, there won't any known blockers that could prevent building modules out of Mesos tree. Allow Mesos modules to be built outside the Mesos source tree - Key: MESOS-2096 URL: https://issues.apache.org/jira/browse/MESOS-2096 Project: Mesos Issue Type: Bug Reporter: Kapil Arya Priority: Blocker We want to be able to provide a mechanism to allow third parties to build Mesos modules outside the Mesos source tree. Ideally, we would want to build Mesos modules without the need to build Mesos itself, provided that the public headers and corresponding shared libraries are available. Building a Mesos module requires access to the following components: 1. stout headers 2. libprocess headers and corresponding dynamic libraries (current, just libmesos.so) 3. protobufs 4. glog 5. gmock/gtest (optional) 6. boost Of the above components, stout headers and libprocess headers are already installed as part of standard
[jira] [Updated] (MESOS-1912) Decouple libev from clock implementation
[ https://issues.apache.org/jira/browse/MESOS-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklas Quarfot Nielsen updated MESOS-1912: -- Sprint: Mesosphere Q4 Sprint 2 - 11/14 Decouple libev from clock implementation Key: MESOS-1912 URL: https://issues.apache.org/jira/browse/MESOS-1912 Project: Mesos Issue Type: Task Reporter: Niklas Quarfot Nielsen Assignee: Benjamin Hindman -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1912) Decouple libev from clock implementation
[ https://issues.apache.org/jira/browse/MESOS-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklas Quarfot Nielsen updated MESOS-1912: -- Assignee: Benjamin Hindman Decouple libev from clock implementation Key: MESOS-1912 URL: https://issues.apache.org/jira/browse/MESOS-1912 Project: Mesos Issue Type: Task Reporter: Niklas Quarfot Nielsen Assignee: Benjamin Hindman -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2042) Authenticatee Module: Integrate authenticatee module in scheduler
[ https://issues.apache.org/jira/browse/MESOS-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Toenshoff updated MESOS-2042: -- Sprint: Mesosphere Q4 Sprint 2 - 11/14 Authenticatee Module: Integrate authenticatee module in scheduler - Key: MESOS-2042 URL: https://issues.apache.org/jira/browse/MESOS-2042 Project: Mesos Issue Type: Improvement Components: modules Reporter: Till Toenshoff Assignee: Till Toenshoff Allow for frameworks to use the authenticatee module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1894) Authenticator Module: Tests
[
https://issues.apache.org/jira/browse/MESOS-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Toenshoff updated MESOS-1894:
--
Target Version/s: 0.22.0
Authenticator Module: Tests
---
Key: MESOS-1894
URL: https://issues.apache.org/jira/browse/MESOS-1894
Project: Mesos
Issue Type: Improvement
Components: modules
Reporter: Till Toenshoff
Assignee: Till Toenshoff
Priority: Blocker
h4. Motivation
Make sure that an authenticator module can be fully unit- and integration-
tested within the mesos test suite.
h4. Status Quo
We currently have a few CRAM-MD5 specific authentication integration tests (
{{tests/sasl_tests.cpp}} ) as well as a complete authentication test list for
authentication as a whole with some CRAM-MD5 specifics again (
{{tests/authentication.cpp}} ).
h4. Approach
1. MESOS-1864 needs to get fleshed out, providing a module integration test
scheme for our mesos test suite
2. refactor / move / rename integration tests of the CRAM-MD5 specific tests
{{tests/sasl_tests.cpp}} {{tests/authentication.cpp}}
3. implement tests via a parameterized version of {{tests/authentication.cpp}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1974) Update the C++ Resources abstraction for DiskInfo
[
https://issues.apache.org/jira/browse/MESOS-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu updated MESOS-1974:
--
Description:
WeThe existing C++ 'Resources' interfaces are poorly designed. Some of them are
confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may wanna
introduce a concept about Resource that indicates that a resource cannot be
merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to refactor
it.
was:
The existing C++ 'Resources' interfaces are poorly designed. Some of them are
confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may wanna
introduce a concept about Resource that indicates that a resource cannot be
merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to refactor
it.
Update the C++ Resources abstraction for DiskInfo
-
Key: MESOS-1974
URL: https://issues.apache.org/jira/browse/MESOS-1974
Project: Mesos
Issue Type: Improvement
Reporter: Jie Yu
Assignee: Jie Yu
WeThe existing C++ 'Resources' interfaces are poorly designed. Some of them
are confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may
wanna introduce a concept about Resource that indicates that a resource
cannot be merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to
refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1974) Update the C++ Resources abstraction for DiskInfo
[
https://issues.apache.org/jira/browse/MESOS-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu updated MESOS-1974:
--
Summary: Update the C++ Resources abstraction for DiskInfo (was: Refactor
the C++ 'Resources' abstraction.)
Update the C++ Resources abstraction for DiskInfo
-
Key: MESOS-1974
URL: https://issues.apache.org/jira/browse/MESOS-1974
Project: Mesos
Issue Type: Improvement
Reporter: Jie Yu
Assignee: Jie Yu
The existing C++ 'Resources' interfaces are poorly designed. Some of them are
confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may
wanna introduce a concept about Resource that indicates that a resource
cannot be merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to
refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1974) Update the C++ Resources abstraction for DiskInfo
[
https://issues.apache.org/jira/browse/MESOS-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu updated MESOS-1974:
--
Description:
As we introduce DiskInfo and reservation for Resource. We need to change the
C++ Resources abstraction to properly deal with merge/split of resources with
those additional fields.
Also, the existing C++ 'Resources' interfaces are poorly designed. Some of them
are confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may wanna
introduce a concept about Resource that indicates that a resource cannot be
merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to refactor
it.
was:
WeThe existing C++ 'Resources' interfaces are poorly designed. Some of them are
confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may wanna
introduce a concept about Resource that indicates that a resource cannot be
merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to refactor
it.
Update the C++ Resources abstraction for DiskInfo
-
Key: MESOS-1974
URL: https://issues.apache.org/jira/browse/MESOS-1974
Project: Mesos
Issue Type: Improvement
Reporter: Jie Yu
Assignee: Jie Yu
As we introduce DiskInfo and reservation for Resource. We need to change the
C++ Resources abstraction to properly deal with merge/split of resources with
those additional fields.
Also, the existing C++ 'Resources' interfaces are poorly designed. Some of
them are confusing and unintuitive. Some of them are overloaded with too many
functionalities. For instance,
{noformat}
bool operator = (const Resource left, const Resource right);
{noformat}
This interface in non-intuitive because A = B doesn't imply !(B = A).
{noformat}
Resource operator + (const Resource left, const Resource right);
{noformat}
This one is also non-intuitive because if 'left' is not compatible with
'right', the result is 'left' (why not right???). Similar for operator '-'.
{noformat}
OptionResource Resources::get(const Resource r) const;
{noformat}
This one assume Resources is flattened, but it might not be.
As we start to introduce persistent disk resources (MESOS-1554), things will
get more complicated. For example, one may want to get two types of 'disk()'
functions: one returns the ephemeral disk bytes (with no disk info), one
returns the total disk bytes (including ones that have disk info). We may
wanna introduce a concept about Resource that indicates that a resource
cannot be merged or split (e.g., atomic?).
Since we need to change this class anyway. I wanna take this chance to
refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-2097) Update Resource protobuf with DiskInfo
Jie Yu created MESOS-2097:
-
Summary: Update Resource protobuf with DiskInfo
Key: MESOS-2097
URL: https://issues.apache.org/jira/browse/MESOS-2097
Project: Mesos
Issue Type: Task
Reporter: Jie Yu
{noformat}
message Resource {
required string name = 1;
required Value.Type type = 2;
optional Value.Scalar scalar = 3;
optional Value.Ranges ranges = 4;
optional Value.Set set = 5;
optional string role = 6 [default = *];
// Used for describing persistent disk resource.
message DiskInfo {
// A unique identifier for the persistent disk resource. The id
// needs to be unique within a role for a slave.
required string id = 1;
// The volume mapping for the persistent disk resource.
required Volume volume = 2;
}
optional DiskInfo disk = 8;
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-2098) Update task validation to be after task authorization.
Jie Yu created MESOS-2098: - Summary: Update task validation to be after task authorization. Key: MESOS-2098 URL: https://issues.apache.org/jira/browse/MESOS-2098 Project: Mesos Issue Type: Task Reporter: Jie Yu So that we can simply the task validation because we no longer need to check with pendingTasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-2030) Maintain persistent disk resources in master memory.
[ https://issues.apache.org/jira/browse/MESOS-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-2030: -- Component/s: master Description: Maintain an in-memory data structure to track persistent disk resources on each slave. Update this data structure when slaves register/re-register/disconnect, etc. (was: We need to do the following in master in order to support persistent disk resource: 1) Add an API allowing the framework to release a persistent disk resource. 2) Maintain an in-memory data structure to track persistent disk resources on each slave. Update this data structure when slaves register/re-register/disconnect, etc. 3) Relay releasing of persistent disk resource to the corresponding slave according to the data structure maintained in 2)) Labels: twitter (was: ) Story Points: 3 (was: 5) Summary: Maintain persistent disk resources in master memory. (was: Support persistent disk resource in master.) Maintain persistent disk resources in master memory. Key: MESOS-2030 URL: https://issues.apache.org/jira/browse/MESOS-2030 Project: Mesos Issue Type: Task Components: master Reporter: Jie Yu Assignee: Jie Yu Labels: twitter Maintain an in-memory data structure to track persistent disk resources on each slave. Update this data structure when slaves register/re-register/disconnect, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2099) Support acquiring/releasing resources with DiskInfo in allocator.
Jie Yu created MESOS-2099: - Summary: Support acquiring/releasing resources with DiskInfo in allocator. Key: MESOS-2099 URL: https://issues.apache.org/jira/browse/MESOS-2099 Project: Mesos Issue Type: Task Components: allocation Reporter: Jie Yu The allocator needs to be changed because the resources are changing while we acquiring or releasing persistent disk resources (resources with DiskInfo). For example, when we release a persistent disk resource, we are changing the release with DiskInfo to a resource with the DiskInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (MESOS-1902) Support persistent disk resource.
[ https://issues.apache.org/jira/browse/MESOS-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Mahler closed MESOS-1902. -- Resolution: Duplicate Support persistent disk resource. - Key: MESOS-1902 URL: https://issues.apache.org/jira/browse/MESOS-1902 Project: Mesos Issue Type: Task Reporter: Jie Yu Mesos needs to provide a way to allow tasks to write persistent data which won’t be garbage collected. For example, a task can write its persistent data to some predefined directory. When this task finishes, the framework can launch a new task which is able to access the persistent data written by the previous task which Mesos would have usually garbage-collected. One way to achieve that is to provide a new type of disk resources which are persistent. We call it persistent disk resource. When a framework launches a task using persistent disk resources, the data the task writes will be persisted. When the framework launches a new task using the same persistent disk resource (after the previous task finishes), the new task will be able to access the data written by the previous task. The persistent disk resource should be able to survive slave reboot or slave info/id change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-2097) Update Resource protobuf with DiskInfo
[
https://issues.apache.org/jira/browse/MESOS-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu reassigned MESOS-2097:
-
Assignee: Jie Yu
Update Resource protobuf with DiskInfo
--
Key: MESOS-2097
URL: https://issues.apache.org/jira/browse/MESOS-2097
Project: Mesos
Issue Type: Task
Reporter: Jie Yu
Assignee: Jie Yu
{noformat}
message Resource {
required string name = 1;
required Value.Type type = 2;
optional Value.Scalar scalar = 3;
optional Value.Ranges ranges = 4;
optional Value.Set set = 5;
optional string role = 6 [default = *];
// Used for describing persistent disk resource.
message DiskInfo {
// A unique identifier for the persistent disk resource. The id
// needs to be unique within a role for a slave.
required string id = 1;
// The volume mapping for the persistent disk resource.
required Volume volume = 2;
}
optional DiskInfo disk = 8;
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-2100) Implement master to slave protocol for persistent disk resources.
[ https://issues.apache.org/jira/browse/MESOS-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-2100: -- Story Points: 8 (was: 5) Implement master to slave protocol for persistent disk resources. - Key: MESOS-2100 URL: https://issues.apache.org/jira/browse/MESOS-2100 Project: Mesos Issue Type: Task Components: master, slave Reporter: Jie Yu Labels: twitter We need to do the following: 1) Slave needs to send persisted resources when registering (or re-registering). 2) Master needs to send total persisted resources to slave by either re-using RunTask/UpdateFrameworkInfo or introduce new type of messages (like UpdateResources). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2100) Implement master to slave protocol for persistent disk resources.
Jie Yu created MESOS-2100: - Summary: Implement master to slave protocol for persistent disk resources. Key: MESOS-2100 URL: https://issues.apache.org/jira/browse/MESOS-2100 Project: Mesos Issue Type: Task Components: master, slave Reporter: Jie Yu We need to do the following: 1) Slave needs to send persisted resources when registering (or re-registering). 2) Master needs to send total persisted resources to slave by either re-using RunTask/UpdateFrameworkInfo or introduce new type of messages (like UpdateResources). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MESOS-1902) Support persistent disk resource.
[ https://issues.apache.org/jira/browse/MESOS-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Mahler reopened MESOS-1902: Assignee: (was: Jie Yu) Support persistent disk resource. - Key: MESOS-1902 URL: https://issues.apache.org/jira/browse/MESOS-1902 Project: Mesos Issue Type: Task Reporter: Jie Yu Mesos needs to provide a way to allow tasks to write persistent data which won’t be garbage collected. For example, a task can write its persistent data to some predefined directory. When this task finishes, the framework can launch a new task which is able to access the persistent data written by the previous task which Mesos would have usually garbage-collected. One way to achieve that is to provide a new type of disk resources which are persistent. We call it persistent disk resource. When a framework launches a task using persistent disk resources, the data the task writes will be persisted. When the framework launches a new task using the same persistent disk resource (after the previous task finishes), the new task will be able to access the data written by the previous task. The persistent disk resource should be able to survive slave reboot or slave info/id change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-2101) Add the persistent resources release primitive to the framework API
Jie Yu created MESOS-2101: - Summary: Add the persistent resources release primitive to the framework API Key: MESOS-2101 URL: https://issues.apache.org/jira/browse/MESOS-2101 Project: Mesos Issue Type: Task Reporter: Jie Yu We are thinking about introducing a Release protobuf message which specifies persistent disk resources (w/ DiskInfo) to release. The Release message could be piggybacked on the Launch/Decline message. This probably will overlap with the dynamic reservation work (MESOS-2018). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MESOS-1902) Support persistent disk resource.
[ https://issues.apache.org/jira/browse/MESOS-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Mahler reopened MESOS-1902: Assignee: (was: Jie Yu) Support persistent disk resource. - Key: MESOS-1902 URL: https://issues.apache.org/jira/browse/MESOS-1902 Project: Mesos Issue Type: Task Reporter: Jie Yu Mesos needs to provide a way to allow tasks to write persistent data which won’t be garbage collected. For example, a task can write its persistent data to some predefined directory. When this task finishes, the framework can launch a new task which is able to access the persistent data written by the previous task which Mesos would have usually garbage-collected. One way to achieve that is to provide a new type of disk resources which are persistent. We call it persistent disk resource. When a framework launches a task using persistent disk resources, the data the task writes will be persisted. When the framework launches a new task using the same persistent disk resource (after the previous task finishes), the new task will be able to access the data written by the previous task. The persistent disk resource should be able to survive slave reboot or slave info/id change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
