[mesos] 03/03: Added MESOS-9624 and MESOS-9626 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit f6159cb1caab84c6d3725b29b3f46a1c40a21b2b Author: Chun-Hung Hsiao AuthorDate: Tue Apr 9 13:14:56 2019 -0700 Added MESOS-9624 and MESOS-9626 to the 1.8.0 CHANGELOG. --- CHANGELOG | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index a4689d2..eec6093 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -295,7 +295,9 @@ All Resolved Issues: * [MESOS-9620] - Add metrics for volume gid manager * [MESOS-9622] - Refactor SLRP with a CSI volume manager. * [MESOS-9623] - Implement CSI volume manager with CSI v1. + * [MESOS-9624] - Bundle CSI spec v1.0 in Mesos. * [MESOS-9625] - Make `DiskProfileAdaptor` agnostic to CSI spec version. + * [MESOS-9626] - Make SLRP pick the appropriate CSI versions for plugins. * [MESOS-9632] - Refactor SLRP with a CSI service manager. * [MESOS-9639] - Make CSI plugin RPC metrics agnostic to CSI versions. * [MESOS-9648] - Make operation reconciliation send asynchronous updates
[mesos] 01/03: Fixed a container ID generation issue in the CSI service manager.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 315340f5c287fa72d35011ba7fedabf91a66514d
Author: Chun-Hung Hsiao
AuthorDate: Sat Apr 6 21:36:54 2019 -0700
Fixed a container ID generation issue in the CSI service manager.
Review: https://reviews.apache.org/r/70427
---
src/csi/service_manager.cpp | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/src/csi/service_manager.cpp b/src/csi/service_manager.cpp
index 0a3663c..d6d0d4a 100644
--- a/src/csi/service_manager.cpp
+++ b/src/csi/service_manager.cpp
@@ -23,6 +23,7 @@
#include
#include
+#include
#include
@@ -51,7 +52,6 @@
#include "csi/paths.hpp"
#include "csi/v0_client.hpp"
-#include "csi/v0_utils.hpp"
#include "internal/devolve.hpp"
#include "internal/evolve.hpp"
@@ -105,11 +105,19 @@ static ContainerID getContainerId(
const string& containerPrefix,
const CSIPluginContainerInfo& container)
{
+ // NOTE: We cannot simply stringify `container.services()` since it returns
+ // `RepeatedField`, so we reconstruct the list of services here.
+ vector services;
+ services.reserve(container.services_size());
+ for (int i = 0; i < container.services_size(); i++) {
+services.push_back(container.services(i));
+ }
+
ContainerID containerId;
containerId.set_value(
containerPrefix +
strings::join("-", strings::replace(info.type(), ".", "-"), info.name())
+
- "--" + strings::join("-", container.services()));
+ "--" + strings::join("-", services));
return containerId;
}
[mesos] branch master updated (65d0a44 -> 475b5e1)
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from 65d0a44 Added MESOS-9667 to the 1.8.0 CHANGELOG. new 0f9f9e7 Fixed a container ID generation issue in the CSI service manager. new 05a228f Auto-detect CSI API version to create the proper volume manager. new 475b5e1 Added MESOS-9624 and MESOS-9626 to the 1.8.0 CHANGELOG. The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 2 + src/csi/service_manager.cpp| 194 - src/csi/service_manager.hpp| 1 + src/csi/v0_volume_manager.cpp | 28 +-- src/csi/v0_volume_manager.hpp | 4 +- src/csi/v0_volume_manager_process.hpp | 10 +- src/csi/v1_volume_manager.cpp | 28 +-- src/csi/v1_volume_manager.hpp | 4 +- src/csi/v1_volume_manager_process.hpp | 10 +- src/csi/volume_manager.cpp | 31 ++-- src/csi/volume_manager.hpp | 7 +- src/resource_provider/storage/provider.cpp | 77 +--- .../storage_local_resource_provider_tests.cpp | 8 + 13 files changed, 255 insertions(+), 149 deletions(-)
[mesos] 02/03: Auto-detect CSI API version to create the proper volume manager.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 20f076b28e3eaf06f7c942c6117a26d4dad8ed32
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 8 13:49:27 2019 -0700
Auto-detect CSI API version to create the proper volume manager.
The `mesos::csi::VolumeManager::create` function now first creates a
`ServiceManager` to detect the API version, then instantiate the proper
`VolumeManager` based on it.
NOTE: This patch enables CSI v1 by default for all SLRP-related unit
tests except for test `RetryRpcWithExponentialBackoff`.
Review: https://reviews.apache.org/r/70428
---
src/csi/service_manager.cpp| 182 +++--
src/csi/service_manager.hpp| 1 +
src/csi/v0_volume_manager.cpp | 28 +---
src/csi/v0_volume_manager.hpp | 4 +-
src/csi/v0_volume_manager_process.hpp | 10 +-
src/csi/v1_volume_manager.cpp | 28 +---
src/csi/v1_volume_manager.hpp | 4 +-
src/csi/v1_volume_manager_process.hpp | 10 +-
src/csi/volume_manager.cpp | 31 ++--
src/csi/volume_manager.hpp | 7 +-
src/resource_provider/storage/provider.cpp | 77 ++---
.../storage_local_resource_provider_tests.cpp | 8 +
12 files changed, 243 insertions(+), 147 deletions(-)
diff --git a/src/csi/service_manager.cpp b/src/csi/service_manager.cpp
index d6d0d4a..a87df96 100644
--- a/src/csi/service_manager.cpp
+++ b/src/csi/service_manager.cpp
@@ -52,6 +52,7 @@
#include "csi/paths.hpp"
#include "csi/v0_client.hpp"
+#include "csi/v1_client.hpp"
#include "internal/devolve.hpp"
#include "internal/evolve.hpp"
@@ -139,6 +140,7 @@ public:
Future recover();
Future getServiceEndpoint(const Service& service);
+ Future getApiVersion();
private:
// Returns the container info of the specified container for this CSI plugin.
@@ -155,9 +157,12 @@ private:
// Kills the specified plugin container.
Future killContainer(const ContainerID& containerId);
- // Waits for the endpoint (URI to a Unix domain socket) to be ready.
+ // Waits for the endpoint (URI to a Unix domain socket) to be created.
Future waitEndpoint(const string& endpoint);
+ // Probes the endpoint to detect the API version and check for readiness.
+ Future probeEndpoint(const string& endpoint);
+
// Returns the URI of the latest service endpoint for the specified plugin
// container. If the container is not already running, this method will start
// a new container.
@@ -174,6 +179,7 @@ private:
Metrics* metrics;
http::Headers headers;
+ Option apiVersion;
hashmap serviceContainers;
hashmap> daemons;
@@ -350,6 +356,20 @@ Future
ServiceManagerProcess::getServiceEndpoint(const Service& service)
}
+Future ServiceManagerProcess::getApiVersion()
+{
+ if (apiVersion.isSome()) {
+return apiVersion.get();
+ }
+
+ // Ensure that the plugin has been probed (which does the API version
+ // detection) through `getEndpoint` before returning the API version.
+ CHECK(!serviceContainers.empty());
+ return getEndpoint(serviceContainers.begin()->second)
+.then(process::defer(self(), [=] { return CHECK_NOTNONE(apiVersion); }));
+}
+
+
Option ServiceManagerProcess::getContainerInfo(
const ContainerID& containerId)
{
@@ -384,8 +404,8 @@ ServiceManagerProcess::getContainers()
httpResponse.status + "' (" + httpResponse.body + ")");
}
- Try v1Response =
-internal::deserialize(
+ Try v1Response =
+internal::deserialize(
contentType, httpResponse.body);
if (v1Response.isError()) {
@@ -473,55 +493,117 @@ Future
ServiceManagerProcess::waitEndpoint(const string& endpoint)
const string endpointPath =
strings::remove(endpoint, "unix://", strings::PREFIX);
- Future created = Nothing();
- if (!os::exists(endpointPath)) {
-// Wait for the endpoint socket to appear until the timeout expires.
-Timeout timeout = Timeout::in(CSI_ENDPOINT_CREATION_TIMEOUT);
+ if (os::exists(endpointPath)) {
+return Nothing();
+ }
-created = process::loop(
-[=]() -> Future {
- if (timeout.expired()) {
-return Failure("Timed out waiting for endpoint '" + endpoint +
"'");
- }
+ // Wait for the endpoint socket to appear until the timeout expires.
+ Timeout timeout = Timeout::in(CSI_ENDPOINT_CREATION_TIMEOUT);
- return process::after(Milliseconds(10));
-},
-[=](const Nothing&) -> ControlFlow {
- if (os::exists(endpointPath)) {
-return Break();
- }
+ return process::loop(
+ [=]() -> Future {
+if (timeout.expired()) {
+ return Failure("Timed out waiting for
[mesos] branch 1.8.x updated (0baa382 -> f6159cb)
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a change to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 0baa382 Added MESOS-9667 to the 1.8.0 CHANGELOG. new 315340f Fixed a container ID generation issue in the CSI service manager. new 20f076b Auto-detect CSI API version to create the proper volume manager. new f6159cb Added MESOS-9624 and MESOS-9626 to the 1.8.0 CHANGELOG. The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 2 + src/csi/service_manager.cpp| 194 - src/csi/service_manager.hpp| 1 + src/csi/v0_volume_manager.cpp | 28 +-- src/csi/v0_volume_manager.hpp | 4 +- src/csi/v0_volume_manager_process.hpp | 10 +- src/csi/v1_volume_manager.cpp | 28 +-- src/csi/v1_volume_manager.hpp | 4 +- src/csi/v1_volume_manager_process.hpp | 10 +- src/csi/volume_manager.cpp | 31 ++-- src/csi/volume_manager.hpp | 7 +- src/resource_provider/storage/provider.cpp | 77 +--- .../storage_local_resource_provider_tests.cpp | 8 + 13 files changed, 255 insertions(+), 149 deletions(-)
[mesos] 03/03: Added MESOS-9624 and MESOS-9626 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 475b5e14085eef88edd868d331bd08b7ba5443bf Author: Chun-Hung Hsiao AuthorDate: Tue Apr 9 13:14:56 2019 -0700 Added MESOS-9624 and MESOS-9626 to the 1.8.0 CHANGELOG. --- CHANGELOG | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 02f20db..65ac83a 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -295,7 +295,9 @@ All Resolved Issues: * [MESOS-9620] - Add metrics for volume gid manager * [MESOS-9622] - Refactor SLRP with a CSI volume manager. * [MESOS-9623] - Implement CSI volume manager with CSI v1. + * [MESOS-9624] - Bundle CSI spec v1.0 in Mesos. * [MESOS-9625] - Make `DiskProfileAdaptor` agnostic to CSI spec version. + * [MESOS-9626] - Make SLRP pick the appropriate CSI versions for plugins. * [MESOS-9632] - Refactor SLRP with a CSI service manager. * [MESOS-9639] - Make CSI plugin RPC metrics agnostic to CSI versions. * [MESOS-9648] - Make operation reconciliation send asynchronous updates
[mesos] 01/03: Fixed a container ID generation issue in the CSI service manager.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 0f9f9e7be80af936fdd49d4c9f6e41edba62e87f
Author: Chun-Hung Hsiao
AuthorDate: Sat Apr 6 21:36:54 2019 -0700
Fixed a container ID generation issue in the CSI service manager.
Review: https://reviews.apache.org/r/70427
---
src/csi/service_manager.cpp | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/src/csi/service_manager.cpp b/src/csi/service_manager.cpp
index 0a3663c..d6d0d4a 100644
--- a/src/csi/service_manager.cpp
+++ b/src/csi/service_manager.cpp
@@ -23,6 +23,7 @@
#include
#include
+#include
#include
@@ -51,7 +52,6 @@
#include "csi/paths.hpp"
#include "csi/v0_client.hpp"
-#include "csi/v0_utils.hpp"
#include "internal/devolve.hpp"
#include "internal/evolve.hpp"
@@ -105,11 +105,19 @@ static ContainerID getContainerId(
const string& containerPrefix,
const CSIPluginContainerInfo& container)
{
+ // NOTE: We cannot simply stringify `container.services()` since it returns
+ // `RepeatedField`, so we reconstruct the list of services here.
+ vector services;
+ services.reserve(container.services_size());
+ for (int i = 0; i < container.services_size(); i++) {
+services.push_back(container.services(i));
+ }
+
ContainerID containerId;
containerId.set_value(
containerPrefix +
strings::join("-", strings::replace(info.type(), ".", "-"), info.name())
+
- "--" + strings::join("-", container.services()));
+ "--" + strings::join("-", services));
return containerId;
}
[mesos] 02/03: Auto-detect CSI API version to create the proper volume manager.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 05a228f3a5eb49ff4f325884e09bda933d45a715
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 8 13:49:27 2019 -0700
Auto-detect CSI API version to create the proper volume manager.
The `mesos::csi::VolumeManager::create` function now first creates a
`ServiceManager` to detect the API version, then instantiate the proper
`VolumeManager` based on it.
NOTE: This patch enables CSI v1 by default for all SLRP-related unit
tests except for test `RetryRpcWithExponentialBackoff`.
Review: https://reviews.apache.org/r/70428
---
src/csi/service_manager.cpp| 182 +++--
src/csi/service_manager.hpp| 1 +
src/csi/v0_volume_manager.cpp | 28 +---
src/csi/v0_volume_manager.hpp | 4 +-
src/csi/v0_volume_manager_process.hpp | 10 +-
src/csi/v1_volume_manager.cpp | 28 +---
src/csi/v1_volume_manager.hpp | 4 +-
src/csi/v1_volume_manager_process.hpp | 10 +-
src/csi/volume_manager.cpp | 31 ++--
src/csi/volume_manager.hpp | 7 +-
src/resource_provider/storage/provider.cpp | 77 ++---
.../storage_local_resource_provider_tests.cpp | 8 +
12 files changed, 243 insertions(+), 147 deletions(-)
diff --git a/src/csi/service_manager.cpp b/src/csi/service_manager.cpp
index d6d0d4a..a87df96 100644
--- a/src/csi/service_manager.cpp
+++ b/src/csi/service_manager.cpp
@@ -52,6 +52,7 @@
#include "csi/paths.hpp"
#include "csi/v0_client.hpp"
+#include "csi/v1_client.hpp"
#include "internal/devolve.hpp"
#include "internal/evolve.hpp"
@@ -139,6 +140,7 @@ public:
Future recover();
Future getServiceEndpoint(const Service& service);
+ Future getApiVersion();
private:
// Returns the container info of the specified container for this CSI plugin.
@@ -155,9 +157,12 @@ private:
// Kills the specified plugin container.
Future killContainer(const ContainerID& containerId);
- // Waits for the endpoint (URI to a Unix domain socket) to be ready.
+ // Waits for the endpoint (URI to a Unix domain socket) to be created.
Future waitEndpoint(const string& endpoint);
+ // Probes the endpoint to detect the API version and check for readiness.
+ Future probeEndpoint(const string& endpoint);
+
// Returns the URI of the latest service endpoint for the specified plugin
// container. If the container is not already running, this method will start
// a new container.
@@ -174,6 +179,7 @@ private:
Metrics* metrics;
http::Headers headers;
+ Option apiVersion;
hashmap serviceContainers;
hashmap> daemons;
@@ -350,6 +356,20 @@ Future
ServiceManagerProcess::getServiceEndpoint(const Service& service)
}
+Future ServiceManagerProcess::getApiVersion()
+{
+ if (apiVersion.isSome()) {
+return apiVersion.get();
+ }
+
+ // Ensure that the plugin has been probed (which does the API version
+ // detection) through `getEndpoint` before returning the API version.
+ CHECK(!serviceContainers.empty());
+ return getEndpoint(serviceContainers.begin()->second)
+.then(process::defer(self(), [=] { return CHECK_NOTNONE(apiVersion); }));
+}
+
+
Option ServiceManagerProcess::getContainerInfo(
const ContainerID& containerId)
{
@@ -384,8 +404,8 @@ ServiceManagerProcess::getContainers()
httpResponse.status + "' (" + httpResponse.body + ")");
}
- Try v1Response =
-internal::deserialize(
+ Try v1Response =
+internal::deserialize(
contentType, httpResponse.body);
if (v1Response.isError()) {
@@ -473,55 +493,117 @@ Future
ServiceManagerProcess::waitEndpoint(const string& endpoint)
const string endpointPath =
strings::remove(endpoint, "unix://", strings::PREFIX);
- Future created = Nothing();
- if (!os::exists(endpointPath)) {
-// Wait for the endpoint socket to appear until the timeout expires.
-Timeout timeout = Timeout::in(CSI_ENDPOINT_CREATION_TIMEOUT);
+ if (os::exists(endpointPath)) {
+return Nothing();
+ }
-created = process::loop(
-[=]() -> Future {
- if (timeout.expired()) {
-return Failure("Timed out waiting for endpoint '" + endpoint +
"'");
- }
+ // Wait for the endpoint socket to appear until the timeout expires.
+ Timeout timeout = Timeout::in(CSI_ENDPOINT_CREATION_TIMEOUT);
- return process::after(Milliseconds(10));
-},
-[=](const Nothing&) -> ControlFlow {
- if (os::exists(endpointPath)) {
-return Break();
- }
+ return process::loop(
+ [=]() -> Future {
+if (timeout.expired()) {
+ return Failure("Timed out waiting for
[mesos] 01/02: Initialized resource provider manager earlier when recovering.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 456e382d73a7f036cefbf67502a456a1d920788c
Author: Benjamin Bannier
AuthorDate: Wed Apr 3 11:51:41 2019 +0200
Initialized resource provider manager earlier when recovering.
When recovering and reusing the same agent ID the resource provider
manager can be initialized before e.g., recovering executors. This patch
move the initialization to such an earlier point. This e.g., allows to
successfully publish resources via the manager when HTTP-based executors
resubscribe which previously ran into an assertion failure.
If the agent ID is not reused we still need to wait for the agent to
register with the master which would assign an agent ID. In that case we
do not expect any executors to resubscribe.
Review: https://reviews.apache.org/r/70368/
---
src/slave/slave.cpp | 8 ++
src/tests/slave_tests.cpp | 192 ++
2 files changed, 200 insertions(+)
diff --git a/src/slave/slave.cpp b/src/slave/slave.cpp
index 5373cee..794a9c9 100644
--- a/src/slave/slave.cpp
+++ b/src/slave/slave.cpp
@@ -1519,6 +1519,9 @@ void Slave::registered(
CHECK_SOME(state::checkpoint(path, info));
+ // If we registered with this agent ID for the first time initialize
+ // the resource provider manager with it; if the manager was already
+ // initialized with a recovered agent ID this is a no-op.
initializeResourceProviderManager(flags, info.id());
// We start the local resource providers daemon once the agent is
@@ -7385,6 +7388,11 @@ Future Slave::recover(const Try&
state)
requiredMasterCapabilities.agentUpdate = true;
}
+ // If we restarted the agent process and will reuse the same agent ID
+ // we can immediately start the resource provider manager. This allows
+ // executors recovered later on to resubscribe immediately.
+ initializeResourceProviderManager(flags, info.id());
+
// Recover the frameworks.
foreachvalue (const FrameworkState& frameworkState,
slaveState->frameworks) {
diff --git a/src/tests/slave_tests.cpp b/src/tests/slave_tests.cpp
index 528a25a..b1c3a01 100644
--- a/src/tests/slave_tests.cpp
+++ b/src/tests/slave_tests.cpp
@@ -11566,6 +11566,198 @@ TEST_F(SlaveTest,
RetryOperationStatusUpdateAfterRecovery)
Clock::settle();
}
+
+// This test verifies that on agent failover HTTP-based executors using
+// resource provider resources can resubscribe without crashing the
+// agent. This is a regression test for MESOS-9667.
+TEST_F(SlaveTest, AgentFailoverHTTPExecutorUsingResourceProviderResources)
+{
+ // This test is run with paused clock to avoid
+ // dealing with retried task status updates.
+ Clock::pause();
+
+ master::Flags masterFlags = CreateMasterFlags();
+ Try> master = StartMaster(masterFlags);
+ ASSERT_SOME(master);
+
+ Future updateSlaveMessage =
+FUTURE_PROTOBUF(UpdateSlaveMessage(), _, _);
+
+ slave::Flags slaveFlags = CreateSlaveFlags();
+
+ // Use the same process ID so the executor can resubscribe.
+ string processId = process::ID::generate("slave");
+
+ StandaloneMasterDetector detector(master.get()->pid);
+ Try> slave =
+StartSlave(, processId, slaveFlags);
+ ASSERT_SOME(slave);
+
+ Clock::advance(slaveFlags.registration_backoff_factor);
+
+ AWAIT_READY(updateSlaveMessage);
+
+ mesos::v1::ResourceProviderInfo resourceProviderInfo;
+ resourceProviderInfo.set_type("org.apache.mesos.resource_provider.test");
+ resourceProviderInfo.set_name("test");
+
+ // Register a resource provider with the agent.
+ v1::Resources resourceProviderResources = v1::createDiskResource(
+ "200",
+ "*",
+ None(),
+ None(),
+ v1::createDiskSourceRaw());
+
+ v1::MockResourceProvider resourceProvider(
+ resourceProviderInfo,
+ resourceProviderResources);
+
+ Owned endpointDetector(
+ resource_provider::createEndpointDetector(slave.get()->pid));
+
+ updateSlaveMessage = FUTURE_PROTOBUF(UpdateSlaveMessage(), _, _);
+
+ resourceProvider.start(std::move(endpointDetector), ContentType::PROTOBUF);
+
+ AWAIT_READY(updateSlaveMessage);
+
+ // Register a framework to excercise operations.
+ auto scheduler = std::make_shared();
+
+ Future connected;
+ EXPECT_CALL(*scheduler, connected(_))
+.WillOnce(FutureSatisfy());
+
+ v1::scheduler::TestMesos mesos(
+ master.get()->pid,
+ ContentType::PROTOBUF,
+ scheduler);
+
+ AWAIT_READY(connected);
+
+ Future subscribed;
+ Future offers;
+
+ EXPECT_CALL(*scheduler, heartbeat(_))
+.WillRepeatedly(Return()); // Ignore heartbeats.
+
+ EXPECT_CALL(*scheduler, offers(_, _))
+.WillRepeatedly(v1::scheduler::DeclineOffers());
+
+ EXPECT_CALL(
+ *scheduler,
+ offers(
+ _,
+
[mesos] 01/02: Initialized resource provider manager earlier when recovering.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit ef6912a27d164f74acab19b7da880b129e21837a
Author: Benjamin Bannier
AuthorDate: Wed Apr 3 11:51:41 2019 +0200
Initialized resource provider manager earlier when recovering.
When recovering and reusing the same agent ID the resource provider
manager can be initialized before e.g., recovering executors. This patch
move the initialization to such an earlier point. This e.g., allows to
successfully publish resources via the manager when HTTP-based executors
resubscribe which previously ran into an assertion failure.
If the agent ID is not reused we still need to wait for the agent to
register with the master which would assign an agent ID. In that case we
do not expect any executors to resubscribe.
Review: https://reviews.apache.org/r/70368/
---
src/slave/slave.cpp | 8 ++
src/tests/slave_tests.cpp | 192 ++
2 files changed, 200 insertions(+)
diff --git a/src/slave/slave.cpp b/src/slave/slave.cpp
index 5373cee..794a9c9 100644
--- a/src/slave/slave.cpp
+++ b/src/slave/slave.cpp
@@ -1519,6 +1519,9 @@ void Slave::registered(
CHECK_SOME(state::checkpoint(path, info));
+ // If we registered with this agent ID for the first time initialize
+ // the resource provider manager with it; if the manager was already
+ // initialized with a recovered agent ID this is a no-op.
initializeResourceProviderManager(flags, info.id());
// We start the local resource providers daemon once the agent is
@@ -7385,6 +7388,11 @@ Future Slave::recover(const Try&
state)
requiredMasterCapabilities.agentUpdate = true;
}
+ // If we restarted the agent process and will reuse the same agent ID
+ // we can immediately start the resource provider manager. This allows
+ // executors recovered later on to resubscribe immediately.
+ initializeResourceProviderManager(flags, info.id());
+
// Recover the frameworks.
foreachvalue (const FrameworkState& frameworkState,
slaveState->frameworks) {
diff --git a/src/tests/slave_tests.cpp b/src/tests/slave_tests.cpp
index 528a25a..b1c3a01 100644
--- a/src/tests/slave_tests.cpp
+++ b/src/tests/slave_tests.cpp
@@ -11566,6 +11566,198 @@ TEST_F(SlaveTest,
RetryOperationStatusUpdateAfterRecovery)
Clock::settle();
}
+
+// This test verifies that on agent failover HTTP-based executors using
+// resource provider resources can resubscribe without crashing the
+// agent. This is a regression test for MESOS-9667.
+TEST_F(SlaveTest, AgentFailoverHTTPExecutorUsingResourceProviderResources)
+{
+ // This test is run with paused clock to avoid
+ // dealing with retried task status updates.
+ Clock::pause();
+
+ master::Flags masterFlags = CreateMasterFlags();
+ Try> master = StartMaster(masterFlags);
+ ASSERT_SOME(master);
+
+ Future updateSlaveMessage =
+FUTURE_PROTOBUF(UpdateSlaveMessage(), _, _);
+
+ slave::Flags slaveFlags = CreateSlaveFlags();
+
+ // Use the same process ID so the executor can resubscribe.
+ string processId = process::ID::generate("slave");
+
+ StandaloneMasterDetector detector(master.get()->pid);
+ Try> slave =
+StartSlave(, processId, slaveFlags);
+ ASSERT_SOME(slave);
+
+ Clock::advance(slaveFlags.registration_backoff_factor);
+
+ AWAIT_READY(updateSlaveMessage);
+
+ mesos::v1::ResourceProviderInfo resourceProviderInfo;
+ resourceProviderInfo.set_type("org.apache.mesos.resource_provider.test");
+ resourceProviderInfo.set_name("test");
+
+ // Register a resource provider with the agent.
+ v1::Resources resourceProviderResources = v1::createDiskResource(
+ "200",
+ "*",
+ None(),
+ None(),
+ v1::createDiskSourceRaw());
+
+ v1::MockResourceProvider resourceProvider(
+ resourceProviderInfo,
+ resourceProviderResources);
+
+ Owned endpointDetector(
+ resource_provider::createEndpointDetector(slave.get()->pid));
+
+ updateSlaveMessage = FUTURE_PROTOBUF(UpdateSlaveMessage(), _, _);
+
+ resourceProvider.start(std::move(endpointDetector), ContentType::PROTOBUF);
+
+ AWAIT_READY(updateSlaveMessage);
+
+ // Register a framework to excercise operations.
+ auto scheduler = std::make_shared();
+
+ Future connected;
+ EXPECT_CALL(*scheduler, connected(_))
+.WillOnce(FutureSatisfy());
+
+ v1::scheduler::TestMesos mesos(
+ master.get()->pid,
+ ContentType::PROTOBUF,
+ scheduler);
+
+ AWAIT_READY(connected);
+
+ Future subscribed;
+ Future offers;
+
+ EXPECT_CALL(*scheduler, heartbeat(_))
+.WillRepeatedly(Return()); // Ignore heartbeats.
+
+ EXPECT_CALL(*scheduler, offers(_, _))
+.WillRepeatedly(v1::scheduler::DeclineOffers());
+
+ EXPECT_CALL(
+ *scheduler,
+ offers(
+ _,
+
[mesos] 02/02: Added MESOS-9667 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. bbannier pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 65d0a4475e8b7b46737e8a7708f3b121a3cc17e5 Author: Benjamin Bannier AuthorDate: Tue Apr 9 21:42:37 2019 +0200 Added MESOS-9667 to the 1.8.0 CHANGELOG. --- CHANGELOG | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 0c0e30f..02f20db 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -16,7 +16,6 @@ This release contains the following highlights: Unresolved Critical Issues: * [MESOS-9697] Release RPMs are not uploaded to bintray * [MESOS-9672] Docker containerizer should ignore pids of executors that do not pass the connection check. - * [MESOS-9667] Check failure when executor for task using resource provider resources subscribes before agent is registered * [MESOS-9654] `PUBLISH_RESOURCES` should fail if the resource version changes. * [MESOS-9619] Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9616] `Filters.refuse_seconds` declines resources not in offers. @@ -194,6 +193,7 @@ All Resolved Issues: * [MESOS-9692] - Quota may be under allocated for disk resources. * [MESOS-9696] - Test MasterQuotaTest.AvailableResourcesSingleDisconnectedAgent is flaky * [MESOS-9707] - Calling link::lo() may cause runtime error + * [MESOS-9667] - Check failure when executor for task using resource provider resources subscribes before agent is registered. ** Epic * [MESOS-8054] - Feedback for operations
[mesos] branch 1.8.x updated (61764bd -> 0baa382)
This is an automated email from the ASF dual-hosted git repository. bbannier pushed a change to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 61764bd Added MESOS-9707 to the 1.8.0 CHANGELOG. new ef6912a Initialized resource provider manager earlier when recovering. new 0baa382 Added MESOS-9667 to the 1.8.0 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 2 +- src/slave/slave.cpp | 8 ++ src/tests/slave_tests.cpp | 192 ++ 3 files changed, 201 insertions(+), 1 deletion(-)
[mesos] 02/02: Added MESOS-9667 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. bbannier pushed a commit to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 0baa382931edcab0671caff617c78b192444192f Author: Benjamin Bannier AuthorDate: Tue Apr 9 21:42:37 2019 +0200 Added MESOS-9667 to the 1.8.0 CHANGELOG. --- CHANGELOG | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 526be71..a4689d2 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -16,7 +16,6 @@ This release contains the following highlights: Unresolved Critical Issues: * [MESOS-9697] Release RPMs are not uploaded to bintray * [MESOS-9672] Docker containerizer should ignore pids of executors that do not pass the connection check. - * [MESOS-9667] Check failure when executor for task using resource provider resources subscribes before agent is registered * [MESOS-9654] `PUBLISH_RESOURCES` should fail if the resource version changes. * [MESOS-9619] Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9616] `Filters.refuse_seconds` declines resources not in offers. @@ -194,6 +193,7 @@ All Resolved Issues: * [MESOS-9692] - Quota may be under allocated for disk resources. * [MESOS-9696] - Test MasterQuotaTest.AvailableResourcesSingleDisconnectedAgent is flaky * [MESOS-9707] - Calling link::lo() may cause runtime error + * [MESOS-9667] - Check failure when executor for task using resource provider resources subscribes before agent is registered. ** Epic * [MESOS-8054] - Feedback for operations
[mesos] branch master updated (1d4858e -> 65d0a44)
This is an automated email from the ASF dual-hosted git repository. bbannier pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from 1d4858e Added MESOS-9707 to the 1.8.0 CHANGELOG. new 456e382 Initialized resource provider manager earlier when recovering. new 65d0a44 Added MESOS-9667 to the 1.8.0 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 2 +- src/slave/slave.cpp | 8 ++ src/tests/slave_tests.cpp | 192 ++ 3 files changed, 201 insertions(+), 1 deletion(-)
[mesos] branch master updated (0dbb535 -> 1d4858e)
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from 0dbb535 Fix `link::lo()` to deal with `None` value. new 39be632 Added MESOS-9707 to the 1.5.4 CHANGELOG. new 806cc6c Added MESOS-9707 to the 1.6.3 CHANGELOG. new b997c97 Added MESOS-9707 to the 1.7.3 CHANGELOG. new 1d4858e Added MESOS-9707 to the 1.8.0 CHANGELOG. The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 4 1 file changed, 4 insertions(+)
[mesos] 02/04: Added MESOS-9707 to the 1.6.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 806cc6c89a1547bcaf21bef695bb20bbedeba7d3 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:22:35 2019 +0200 Added MESOS-9707 to the 1.6.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 88909b9..51c5423 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -782,6 +782,7 @@ Release Notes - Mesos - Version 1.6.3 (WIP) * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. * [MESOS-9564] - Logrotate container logger lets tasks execute arbitrary commands in the Mesos agent's namespace. * [MESOS-9692] - Quota may be under allocated for disk resources. + * [MESOS-9707] - Calling link::lo() may cause runtime error ** Improvement * [MESOS-8880] - Add minimum capabilities in the master.
[mesos] 04/04: Added MESOS-9707 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 1d4858e991a60d0742c317008ff59f151dcb9d58 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:05:11 2019 +0200 Added MESOS-9707 to the 1.8.0 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index a3d6ff4..0c0e30f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -193,6 +193,7 @@ All Resolved Issues: * [MESOS-9691] - Quota headroom calculation is off when subroles are involved. * [MESOS-9692] - Quota may be under allocated for disk resources. * [MESOS-9696] - Test MasterQuotaTest.AvailableResourcesSingleDisconnectedAgent is flaky + * [MESOS-9707] - Calling link::lo() may cause runtime error ** Epic * [MESOS-8054] - Feedback for operations
[mesos] 03/04: Added MESOS-9707 to the 1.7.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit b997c97a7d16770b012cd29d9a549f3e677a2c99 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:19:11 2019 +0200 Added MESOS-9707 to the 1.7.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 51c5423..a3d6ff4 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -320,6 +320,7 @@ Release Notes - Mesos - Version 1.7.3 (WIP) * [MESOS-9610] - Fetcher vulnerability - escaping from sandbox. * [MESOS-9661] - Agent crashes when SLRP recovers dropped operations. * [MESOS-9692] - Quota may be under allocated for disk resources. + * [MESOS-9707] - Calling link::lo() may cause runtime error ** Improvements * [MESOS-8880] - Add minimum capabilities in the master.
[mesos] 01/04: Added MESOS-9707 to the 1.5.4 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 39be632063d650f0017451dd2287f84f9f49ab82 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:26:16 2019 +0200 Added MESOS-9707 to the 1.5.4 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index a02cb7f..88909b9 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1240,6 +1240,7 @@ Release Notes - Mesos - Version 1.5.4 (WIP) ** Bug * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. + * [MESOS-9707] - Calling link::lo() may cause runtime error Release Notes - Mesos - Version 1.5.3
[mesos] 01/02: Fix `link::lo()` to deal with `None` value.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit d24d0c71f6e4b6f5c8b98c0f90ec2728fc0372b2
Author: Pavel Kirillov <49415588+pashakiril...@users.noreply.github.com>
AuthorDate: Tue Apr 9 15:44:24 2019 +0200
Fix `link::lo()` to deal with `None` value.
Some network links may cause `link::internal::test(link, IFF_LOOPBACK)`
to be `None`.
`Link::lo` should deal with `None` values of test var.
This closes #330
---
src/linux/routing/link/link.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/linux/routing/link/link.cpp b/src/linux/routing/link/link.cpp
index 5388a3d..ef83bc8 100644
--- a/src/linux/routing/link/link.cpp
+++ b/src/linux/routing/link/link.cpp
@@ -99,7 +99,7 @@ Result lo()
Result test = link::internal::test(link, IFF_LOOPBACK);
if (test.isError()) {
return Error("Failed to check the flag on link: " + link);
-} else if (test.get()) {
+} else if (test.isSome() && test.get()) {
return link;
}
}
[mesos] branch 1.5.x updated (136178e -> 89e9505)
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a change to branch 1.5.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 136178e Added MESOS-9529 to 1.5.4 CHANGELOG. new d24d0c7 Fix `link::lo()` to deal with `None` value. new 89e9505 Added MESOS-9707 to the 1.5.4 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/linux/routing/link/link.cpp | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
[mesos] 02/02: Added MESOS-9707 to the 1.5.4 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch 1.5.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 89e9505637726f38cf115c5ed52dec9f2a4a17e0 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:26:16 2019 +0200 Added MESOS-9707 to the 1.5.4 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index ba2f739..023c927 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ Release Notes - Mesos - Version 1.5.4 (WIP) ** Bug * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. + * [MESOS-9707] - Calling link::lo() may cause runtime error Release Notes - Mesos - Version 1.5.3
[mesos] branch 1.6.x updated (34ffae5 -> 80c0673)
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a change to branch 1.6.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 34ffae5 Added MESOS-9529 to 1.6.3 CHANGELOG. new ee630c7 Fix `link::lo()` to deal with `None` value. new 80c0673 Added MESOS-9707 to the 1.6.3 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/linux/routing/link/link.cpp | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
[mesos] 01/02: Fix `link::lo()` to deal with `None` value.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit ee630c7f8db2e59ca92f286f246bd1b5b4f39d21
Author: Pavel Kirillov <49415588+pashakiril...@users.noreply.github.com>
AuthorDate: Tue Apr 9 15:44:24 2019 +0200
Fix `link::lo()` to deal with `None` value.
Some network links may cause `link::internal::test(link, IFF_LOOPBACK)`
to be `None`.
`Link::lo` should deal with `None` values of test var.
This closes #330
---
src/linux/routing/link/link.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/linux/routing/link/link.cpp b/src/linux/routing/link/link.cpp
index 9dc30c5..bff172d 100644
--- a/src/linux/routing/link/link.cpp
+++ b/src/linux/routing/link/link.cpp
@@ -99,7 +99,7 @@ Result lo()
Result test = link::internal::test(link, IFF_LOOPBACK);
if (test.isError()) {
return Error("Failed to check the flag on link: " + link);
-} else if (test.get()) {
+} else if (test.isSome() && test.get()) {
return link;
}
}
[mesos] 02/02: Added MESOS-9707 to the 1.6.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch 1.6.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 80c0673f1844771b6f4baf0433ba9ac27479e722 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:22:35 2019 +0200 Added MESOS-9707 to the 1.6.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 8d22141..392fd25 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -7,6 +7,7 @@ Release Notes - Mesos - Version 1.6.3 (WIP) * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. * [MESOS-9564] - Logrotate container logger lets tasks execute arbitrary commands in the Mesos agent's namespace. * [MESOS-9692] - Quota may be under allocated for disk resources. + * [MESOS-9707] - Calling link::lo() may cause runtime error ** Improvement * [MESOS-8880] - Add minimum capabilities in the master.
[mesos] 02/02: Added MESOS-9707 to the 1.7.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch 1.7.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 9f22a3b8cad9a7e171752c898a0bb65f2a9021a9 Author: Andrei Budnik AuthorDate: Tue Apr 9 17:19:11 2019 +0200 Added MESOS-9707 to the 1.7.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 293a6a9..6dee2da 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -12,6 +12,7 @@ Release Notes - Mesos - Version 1.7.3 (WIP) * [MESOS-9610] - Fetcher vulnerability - escaping from sandbox. * [MESOS-9661] - Agent crashes when SLRP recovers dropped operations. * [MESOS-9692] - Quota may be under allocated for disk resources. + * [MESOS-9707] - Calling link::lo() may cause runtime error ** Improvements * [MESOS-8880] - Add minimum capabilities in the master.
[mesos] branch 1.7.x updated (fc12da5 -> 9f22a3b)
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a change to branch 1.7.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from fc12da5 Fixed compile error with a built-in logrotate module. new 8ac30a5 Fix `link::lo()` to deal with `None` value. new 9f22a3b Added MESOS-9707 to the 1.7.3 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/linux/routing/link/link.cpp | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
[mesos] 01/02: Fix `link::lo()` to deal with `None` value.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 8ac30a58373b4575892cd8647bd5f731cd31ffd6
Author: Pavel Kirillov <49415588+pashakiril...@users.noreply.github.com>
AuthorDate: Tue Apr 9 15:44:24 2019 +0200
Fix `link::lo()` to deal with `None` value.
Some network links may cause `link::internal::test(link, IFF_LOOPBACK)`
to be `None`.
`Link::lo` should deal with `None` values of test var.
This closes #330
---
src/linux/routing/link/link.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/linux/routing/link/link.cpp b/src/linux/routing/link/link.cpp
index 9dc30c5..bff172d 100644
--- a/src/linux/routing/link/link.cpp
+++ b/src/linux/routing/link/link.cpp
@@ -99,7 +99,7 @@ Result lo()
Result test = link::internal::test(link, IFF_LOOPBACK);
if (test.isError()) {
return Error("Failed to check the flag on link: " + link);
-} else if (test.get()) {
+} else if (test.isSome() && test.get()) {
return link;
}
}
[mesos] branch 1.8.x updated (b5fa022 -> 61764bd)
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a change to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from b5fa022 Added MESOS-9623 to the 1.8.0 CHANGELOG. new 7d7d8db Fix `link::lo()` to deal with `None` value. new 61764bd Added MESOS-9707 to the 1.8.0 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/linux/routing/link/link.cpp | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
[mesos] 02/02: Added MESOS-9707 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 61764bde84577090c8eab504c4b1f823634b36cc Author: Andrei Budnik AuthorDate: Tue Apr 9 17:05:11 2019 +0200 Added MESOS-9707 to the 1.8.0 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index a02cb7f..526be71 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -193,6 +193,7 @@ All Resolved Issues: * [MESOS-9691] - Quota headroom calculation is off when subroles are involved. * [MESOS-9692] - Quota may be under allocated for disk resources. * [MESOS-9696] - Test MasterQuotaTest.AvailableResourcesSingleDisconnectedAgent is flaky + * [MESOS-9707] - Calling link::lo() may cause runtime error ** Epic * [MESOS-8054] - Feedback for operations
[mesos] 01/02: Fix `link::lo()` to deal with `None` value.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 7d7d8db9b9963e9a079bee54f06e3aac97f7910c
Author: Pavel Kirillov <49415588+pashakiril...@users.noreply.github.com>
AuthorDate: Tue Apr 9 15:44:24 2019 +0200
Fix `link::lo()` to deal with `None` value.
Some network links may cause `link::internal::test(link, IFF_LOOPBACK)`
to be `None`.
`Link::lo` should deal with `None` values of test var.
This closes #330
---
src/linux/routing/link/link.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/linux/routing/link/link.cpp b/src/linux/routing/link/link.cpp
index 9dc30c5..bff172d 100644
--- a/src/linux/routing/link/link.cpp
+++ b/src/linux/routing/link/link.cpp
@@ -99,7 +99,7 @@ Result lo()
Result test = link::internal::test(link, IFF_LOOPBACK);
if (test.isError()) {
return Error("Failed to check the flag on link: " + link);
-} else if (test.get()) {
+} else if (test.isSome() && test.get()) {
return link;
}
}
[mesos] branch master updated: Fix `link::lo()` to deal with `None` value.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 0dbb535 Fix `link::lo()` to deal with `None` value.
0dbb535 is described below
commit 0dbb535fa7a6790d824c75df1b79917343403115
Author: Pavel Kirillov <49415588+pashakiril...@users.noreply.github.com>
AuthorDate: Tue Apr 9 15:44:24 2019 +0200
Fix `link::lo()` to deal with `None` value.
Some network links may cause `link::internal::test(link, IFF_LOOPBACK)`
to be `None`.
`Link::lo` should deal with `None` values of test var.
This closes #330
---
src/linux/routing/link/link.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/linux/routing/link/link.cpp b/src/linux/routing/link/link.cpp
index 9dc30c5..bff172d 100644
--- a/src/linux/routing/link/link.cpp
+++ b/src/linux/routing/link/link.cpp
@@ -99,7 +99,7 @@ Result lo()
Result test = link::internal::test(link, IFF_LOOPBACK);
if (test.isError()) {
return Error("Failed to check the flag on link: " + link);
-} else if (test.get()) {
+} else if (test.isSome() && test.get()) {
return link;
}
}
[mesos] branch 1.8.x updated: Added MESOS-9623 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a commit to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git The following commit(s) were added to refs/heads/1.8.x by this push: new b5fa022 Added MESOS-9623 to the 1.8.0 CHANGELOG. b5fa022 is described below commit b5fa022935f24b2cf7d271b4694b2c47daaf749f Author: Benno Evers AuthorDate: Tue Apr 9 15:05:37 2019 +0200 Added MESOS-9623 to the 1.8.0 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index e0eb881..a02cb7f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -293,6 +293,7 @@ All Resolved Issues: * [MESOS-9615] - Example framework for feedback on agent default resources * [MESOS-9620] - Add metrics for volume gid manager * [MESOS-9622] - Refactor SLRP with a CSI volume manager. + * [MESOS-9623] - Implement CSI volume manager with CSI v1. * [MESOS-9625] - Make `DiskProfileAdaptor` agnostic to CSI spec version. * [MESOS-9632] - Refactor SLRP with a CSI service manager. * [MESOS-9639] - Make CSI plugin RPC metrics agnostic to CSI versions.
[mesos] branch master updated (a2f0f8b -> 9d3a52c)
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from a2f0f8b Supported CSI v1 in the test CSI plugin. new 2fc4784 Updated CHANGELOG for 1.8.0. new 9d3a52c Added MESOS-9623 to the 1.8.0 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 296 +- 1 file changed, 294 insertions(+), 2 deletions(-)
[mesos] 01/02: Updated CHANGELOG for 1.8.0.
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 2fc47846a1d146843e66b4548bc9db82f39b0041 Author: Benno Evers AuthorDate: Fri Apr 5 22:58:51 2019 +0200 Updated CHANGELOG for 1.8.0. --- CHANGELOG | 295 +- 1 file changed, 293 insertions(+), 2 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 9d11de1..e0eb881 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,5 +1,5 @@ -Release Notes - Mesos - Version 1.8.0 (WIP) +Release Notes - Mesos - Version 1.8.0 +- This release contains the following highlights: * Performance Improvements: @@ -13,6 +13,297 @@ This release contains the following highlights: the scheduler re-subscribes each time it wants to mutate the minimum resource quantity offer filter information, see MESOS-7258. +Unresolved Critical Issues: + * [MESOS-9697] Release RPMs are not uploaded to bintray + * [MESOS-9672] Docker containerizer should ignore pids of executors that do not pass the connection check. + * [MESOS-9667] Check failure when executor for task using resource provider resources subscribes before agent is registered + * [MESOS-9654] `PUBLISH_RESOURCES` should fail if the resource version changes. + * [MESOS-9619] Mesos Master Crashes with Launch Group when using Port Resources + * [MESOS-9616] `Filters.refuse_seconds` declines resources not in offers. + * [MESOS-9609] Master check failure when marking agent unreachable + * [MESOS-9579] ExecutorHttpApiTest.HeartbeatCalls is flaky. + * [MESOS-9560] ContentType/AgentAPITest.MarkResourceProviderGone/1 is flaky + * [MESOS-9536] Nested container launched with non-root user may not be able to write to its sandbox via the environment variable + * [MESOS-9520] IOTest.Read hangs on Windows + * [MESOS-9500] spark submit with docker image on mesos cluster fails. + * [MESOS-9426] ZK master detection can become forever pending. + * [MESOS-9393] Fetcher crashes extracting archives with non-ASCII filenames. + * [MESOS-9365] Windows - GET_CONTAINERS API call causes the Mesos agent to fail + * [MESOS-9355] Persistence volume does not unmount correctly with wrong artifact URI + * [MESOS-9352] Data in persistent volume deleted accidentally when using Docker container and Persistent volume + * [MESOS-9306] Mesos containerizer can get stuck during cgroup cleanup + * [MESOS-9180] tasks get stuck in TASK_KILLING on the default executor + * [MESOS-9053] Network ports isolator can falsely trigger while destroying containers. + * [MESOS-9006] The agent's GET_AGENT leaks resource information when using authorization + * [MESOS-8946] CURL 7.58 causes Mesos to fail decoding raw responses. + * [MESOS-8840] `cpu.cfs_quota_us` may be accidentally set for command task using docker during agent recovery. + * [MESOS-8803] Libprocess deadlocks in a test. + * [MESOS-8769] Agent crashes when CNI config not defined + * [MESOS-8679] If the first KILL stuck in the default executor, all other KILLs will be ignored. + * [MESOS-8608] RmdirContinueOnErrorTest.RemoveWithContinueOnError fails. + * [MESOS-8467] Destroyed executors might be used after `Slave::publishResource()`. + * [MESOS-8257] Unified Containerizer "leaks" a target container mount path to the host FS when the target resolves to an absolute path + * [MESOS-8256] Libprocess can silently deadlock due to worker thread exhaustion. + * [MESOS-8096] Enqueueing events in MockHTTPScheduler can lead to segfaults. + * [MESOS-8038] Launching GPU task sporadically fails. + * [MESOS-7971] PersistentVolumeEndpointsTest.EndpointCreateThenOfferRemove test is flaky + * [MESOS-7911] Non-checkpointing framework's tasks should not be marked LOST when agent disconnects. + * [MESOS-7748] Slow subscribers of streaming APIs can lead to Mesos OOMing. + * [MESOS-7721] Master's agent removal rate limit also applies to agent unreachability. + * [MESOS-7566] Master crash due to failed check in DRFSorter::remove + * [MESOS-7386] Executor not cleaning up existing running docker containers if external logrotate/logger processes die/killed + * [MESOS-5989] Libevent SSL Socket downgrade code accesses uninitialized memory / assumes single peek is sufficient. + * [MESOS-5754] CommandInfo.user not honored in docker containerizer + * [MESOS-2842] Master crashes when framework changes principal on re-registration + +All Resolved Issues: +** Bug + * [MESOS-5048] - MesosContainerizerSlaveRecoveryTest.ResourceStatistics is flaky + * [MESOS-5189] - SSLTest.ProtocolMismatch is slow + * [MESOS-6874] - Agent silently ignores FS isolation when protobuf is malformed + * [MESOS-6949] - SchedulerTest.MasterFailover is flaky + * [MESOS-6990] - PartitionTest.TaskCompletedOnPartitionedAgent is flaky. + *
[mesos] 02/02: Added MESOS-9623 to the 1.8.0 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 9d3a52c3a3bf2333d1eee1cb5865983200173fd6 Author: Benno Evers AuthorDate: Tue Apr 9 15:05:37 2019 +0200 Added MESOS-9623 to the 1.8.0 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index e0eb881..a02cb7f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -293,6 +293,7 @@ All Resolved Issues: * [MESOS-9615] - Example framework for feedback on agent default resources * [MESOS-9620] - Add metrics for volume gid manager * [MESOS-9622] - Refactor SLRP with a CSI volume manager. + * [MESOS-9623] - Implement CSI volume manager with CSI v1. * [MESOS-9625] - Make `DiskProfileAdaptor` agnostic to CSI spec version. * [MESOS-9632] - Refactor SLRP with a CSI service manager. * [MESOS-9639] - Make CSI plugin RPC metrics agnostic to CSI versions.
[mesos] 06/06: Supported CSI v1 in the test CSI plugin.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 2e86f64b9ec9c0f4e2892b4c2d0f152f821d72d9
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 21:36:12 2019 -0700
Supported CSI v1 in the test CSI plugin.
A new `--api_version` optional flag is added to the test CSI plugin. By
default, it will serve both CSI v0 and v1 requests. When this flag is
specified, only one CSI version will be served.
Review: https://reviews.apache.org/r/70404
---
include/mesos/csi/v0.hpp | 2 +
include/mesos/csi/v1.hpp | 2 +
src/examples/test_csi_plugin.cpp | 543 +--
3 files changed, 531 insertions(+), 16 deletions(-)
diff --git a/include/mesos/csi/v0.hpp b/include/mesos/csi/v0.hpp
index 8be070b..eb303ff 100644
--- a/include/mesos/csi/v0.hpp
+++ b/include/mesos/csi/v0.hpp
@@ -39,6 +39,8 @@ namespace v0 {
using namespace ::csi::v0;
+constexpr char API_VERSION[] = "v0";
+
} // namespace v0 {
} // namespace csi {
} // namespace mesos {
diff --git a/include/mesos/csi/v1.hpp b/include/mesos/csi/v1.hpp
index d4ebe42..9209111 100644
--- a/include/mesos/csi/v1.hpp
+++ b/include/mesos/csi/v1.hpp
@@ -39,6 +39,8 @@ namespace v1 {
using namespace ::csi::v1;
+constexpr char API_VERSION[] = "v1";
+
} // namespace v1 {
} // namespace csi {
} // namespace mesos {
diff --git a/src/examples/test_csi_plugin.cpp b/src/examples/test_csi_plugin.cpp
index 54753d9..b54d666 100644
--- a/src/examples/test_csi_plugin.cpp
+++ b/src/examples/test_csi_plugin.cpp
@@ -33,10 +33,12 @@
#include
#include
+#include
#include
#include
+#include
#include
#include
#include
@@ -56,6 +58,7 @@
#include
#include "csi/v0_utils.hpp"
+#include "csi/v1_utils.hpp"
#include "linux/fs.hpp"
@@ -106,6 +109,20 @@ class Flags : public virtual
mesos::internal::logging::Flags
public:
Flags()
{
+const hashset supportedApiVersions =
+ {mesos::csi::v0::API_VERSION, mesos::csi::v1::API_VERSION};
+
+add(::api_version,
+"api_version",
+"If set, the plugin would only serve CSI calls of the specified API\n"
+"version. Otherwise, the plugin serves all supported API versions by\n"
+"default. (Supported values: " + stringify(supportedApiVersions) + ")",
+[=](const Option& apiVersion) {
+ return apiVersion.isNone() ||
+supportedApiVersions.contains(apiVersion.get())
+ ? Option::none() : Error("Unsupported API version");
+});
+
add(::endpoint,
"endpoint",
"Path to the Unix domain socket the plugin should bind to.");
@@ -138,6 +155,7 @@ public:
"domain socket. (Example: 'unix:///path/to/socket')");
}
+ Option api_version;
string endpoint;
string work_dir;
Bytes available_capacity;
@@ -150,17 +168,22 @@ public:
class TestCSIPlugin
: public csi::v0::Identity::Service,
public csi::v0::Controller::Service,
-public csi::v0::Node::Service
+public csi::v0::Node::Service,
+public csi::v1::Identity::Service,
+public csi::v1::Controller::Service,
+public csi::v1::Node::Service
{
public:
TestCSIPlugin(
- const string& _workDir,
+ const Option& _apiVersion,
const string& _endpoint,
+ const string& _workDir,
const Bytes& _availableCapacity,
const hashmap& _createParameters,
const hashmap& _volumes)
-: workDir(_workDir),
+: apiVersion(_apiVersion),
endpoint(_endpoint),
+ workDir(_workDir),
availableCapacity(_availableCapacity),
createParameters(_createParameters.begin(), _createParameters.end())
{
@@ -208,6 +231,8 @@ public:
void run();
+ // CSI v0 RPCs.
+
Status GetPluginInfo(
ServerContext* context,
const csi::v0::GetPluginInfoRequest* request,
@@ -293,6 +318,93 @@ public:
const csi::v0::NodeGetCapabilitiesRequest* request,
csi::v0::NodeGetCapabilitiesResponse* response) override;
+ // CSI v1 RPCs.
+
+ Status GetPluginInfo(
+ ServerContext* context,
+ const csi::v1::GetPluginInfoRequest* request,
+ csi::v1::GetPluginInfoResponse* response) override;
+
+ Status GetPluginCapabilities(
+ ServerContext* context,
+ const csi::v1::GetPluginCapabilitiesRequest* request,
+ csi::v1::GetPluginCapabilitiesResponse* response) override;
+
+ Status Probe(
+ ServerContext* context,
+ const csi::v1::ProbeRequest* request,
+ csi::v1::ProbeResponse* response) override;
+
+ Status CreateVolume(
+ ServerContext* context,
+ const csi::v1::CreateVolumeRequest* request,
+ csi::v1::CreateVolumeResponse* response) override;
+
+ Status DeleteVolume(
+ ServerContext* context,
+ const csi::v1::DeleteVolumeRequest* request,
+ csi::v1::DeleteVolumeResponse* response) override;
+
+ Status
[mesos] 05/06: Refactored the test CSI plugin to make it easy to support CSI v1.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit eca223cf54fcd6a6d828548fb045ce6d150dfe72
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 18:48:24 2019 -0700
Refactored the test CSI plugin to make it easy to support CSI v1.
Review: https://reviews.apache.org/r/70403
---
src/examples/test_csi_plugin.cpp | 951 +--
1 file changed, 606 insertions(+), 345 deletions(-)
diff --git a/src/examples/test_csi_plugin.cpp b/src/examples/test_csi_plugin.cpp
index 4321f8f..54753d9 100644
--- a/src/examples/test_csi_plugin.cpp
+++ b/src/examples/test_csi_plugin.cpp
@@ -14,9 +14,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+#include
+#include
#include
#include
#include
+#include
+
+#include
+#include
#include
@@ -25,16 +31,24 @@
#include
+#include
#include
+#include
+#include
+
#include
#include
#include
#include
+#include
#include
#include
+#include
#include
#include
+#include
+#include
#include
#include
@@ -47,6 +61,7 @@
#include "logging/logging.hpp"
+namespace http = process::http;
namespace fs = mesos::internal::fs;
using std::cerr;
@@ -59,6 +74,10 @@ using std::string;
using std::unique_ptr;
using std::vector;
+using google::protobuf::Map;
+using google::protobuf::MapPair;
+using google::protobuf::RepeatedPtrField;
+
using grpc::AsyncGenericService;
using grpc::ByteBuffer;
using grpc::ClientContext;
@@ -73,6 +92,10 @@ using grpc::ServerContext;
using grpc::Status;
using grpc::WriteOptions;
+using mesos::csi::types::VolumeCapability;
+
+using process::grpc::StatusError;
+
constexpr char PLUGIN_NAME[] = "org.apache.mesos.csi.test";
constexpr char NODE_ID[] = "localhost";
constexpr Bytes DEFAULT_VOLUME_CAPACITY = Megabytes(64);
@@ -144,7 +167,7 @@ public:
// Construct the default mount volume capability.
defaultVolumeCapability.mutable_mount();
defaultVolumeCapability.mutable_access_mode()
- ->set_mode(csi::v0::VolumeCapability::AccessMode::SINGLE_NODE_WRITER);
+ ->set_mode(VolumeCapability::AccessMode::SINGLE_NODE_WRITER);
// Scan for preprovisioned volumes.
//
@@ -280,12 +303,69 @@ private:
string getVolumePath(const VolumeInfo& volumeInfo);
Try parseVolumePath(const string& path);
+ Try createVolume(
+ const string& name,
+ const Bytes& requiredBytes,
+ const Bytes& limitBytes,
+ const RepeatedPtrField& capabilities,
+ const Map parameters);
+
+ Try deleteVolume(const string& volumeId);
+
+ Try controllerPublishVolume(
+ const string& volumeId,
+ const string& nodeId,
+ const VolumeCapability& capability,
+ bool readonly,
+ const Map& volumeContext);
+
+ Try controllerUnpublishVolume(
+ const string& volumeId, const string& nodeId);
+
+ // Returns `StatusError` if the volume does not exist; returns
`Option`
+ // with an error set if the volume is not compatible with the given
arguments.
+ Try, StatusError> validateVolumeCapabilities(
+ const string& volumeId,
+ const Map& volumeContext,
+ const RepeatedPtrField& capabilities,
+ const Option>& parameters = None());
+
+ Try, StatusError> listVolumes(
+ const Option& maxEntries,
+ const Option& startingToken);
+
+ Try getCapacity(
+ const RepeatedPtrField& capabilities,
+ const Map& parameters);
+
+ Try nodeStageVolume(
+ const string& volumeId,
+ const Map& publishContext,
+ const string& stagingPath,
+ const VolumeCapability& capability,
+ const Map& volumeContext);
+
+ Try nodeUnstageVolume(
+ const string& volumeId, const string& stagingPath);
+
+ Try nodePublishVolume(
+ const string& volumeId,
+ const Map& publishContext,
+ const string& stagingPath,
+ const string& targetPath,
+ const VolumeCapability& capability,
+ bool readonly,
+ const Map& volumeContext);
+
+ Try nodeUnpublishVolume(
+ const string& volumeId, const string& targetPath);
+
const string workDir;
const string endpoint;
Bytes availableCapacity;
- csi::v0::VolumeCapability defaultVolumeCapability;
- google::protobuf::Map createParameters;
+ VolumeCapability defaultVolumeCapability;
+ Map createParameters;
hashmap volumes;
};
@@ -349,92 +429,26 @@ Status TestCSIPlugin::CreateVolume(
{
LOG(INFO) << request->GetDescriptor()->name() << " '" << *request << "'";
- // TODO(chhsiao): Validate required fields.
+ // TODO(chhsiao): Validate the request.
- if (request->name().empty()) {
-return Status(grpc::INVALID_ARGUMENT, "Volume name cannot be empty");
- }
-
- if (strings::contains(request->name(), stringify(os::PATH_SEPARATOR))) {
-return Status(
-grpc::INVALID_ARGUMENT,
-"Volume name cannot contain '" +
[mesos] 04/06: Implemented CSI v1 volume manager.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit a3c2632f87d1ccd5275dca073f87287fe09a370b
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 22:47:49 2019 -0700
Implemented CSI v1 volume manager.
The v1 Volume manager is basically copied from the v0 volume manager,
with the following changes:
* `validateVolume` will validate volumes against parameters.
* `publishVolume` will only create the parent directory of the target
path. The creation of the target path itself is the responsibility
of the plugin.
* The node ID is obtained through the `NodeGetInfo` call.
Review: https://reviews.apache.org/r/70402
---
src/CMakeLists.txt|1 +
src/Makefile.am |3 +
src/csi/v1_volume_manager.cpp | 1356 +
src/csi/v1_volume_manager.hpp | 116 +++
src/csi/v1_volume_manager_process.hpp | 218 ++
5 files changed, 1694 insertions(+)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index d36d0be..1d4f541 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -244,6 +244,7 @@ set(CSI_SRC
csi/v1.cpp
csi/v1_client.cpp
csi/v1_utils.cpp
+ csi/v1_volume_manager.cpp
csi/volume_manager.cpp)
set(DOCKER_SRC
diff --git a/src/Makefile.am b/src/Makefile.am
index 64898df..5f97523 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1600,6 +1600,9 @@ libcsi_la_SOURCES =
\
csi/v1_client.hpp\
csi/v1_utils.cpp \
csi/v1_utils.hpp \
+ csi/v1_volume_manager.cpp\
+ csi/v1_volume_manager.hpp\
+ csi/v1_volume_manager_process.hpp\
csi/volume_manager.cpp \
csi/volume_manager.hpp
diff --git a/src/csi/v1_volume_manager.cpp b/src/csi/v1_volume_manager.cpp
new file mode 100644
index 000..bd334f1
--- /dev/null
+++ b/src/csi/v1_volume_manager.cpp
@@ -0,0 +1,1356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "csi/v1_volume_manager.hpp"
+
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "csi/paths.hpp"
+#include "csi/v1_client.hpp"
+#include "csi/v1_utils.hpp"
+#include "csi/v1_volume_manager_process.hpp"
+
+#include "slave/state.hpp"
+
+namespace http = process::http;
+namespace slave = mesos::internal::slave;
+
+using std::list;
+using std::string;
+using std::vector;
+
+using google::protobuf::Map;
+
+using mesos::csi::state::VolumeState;
+
+using process::Break;
+using process::Continue;
+using process::ControlFlow;
+using process::Failure;
+using process::Future;
+using process::ProcessBase;
+
+using process::grpc::StatusError;
+
+using process::grpc::client::Runtime;
+
+namespace mesos{
+namespace csi {
+namespace v1 {
+
+VolumeManagerProcess::VolumeManagerProcess(
+const http::URL& agentUrl,
+const string& _rootDir,
+const CSIPluginInfo& _info,
+const hashset _services,
+const string& containerPrefix,
+const Option& authToken,
+const Runtime& _runtime,
+Metrics* _metrics)
+ : ProcessBase(process::ID::generate("csi-v1-volume-manager")),
+rootDir(_rootDir),
+info(_info),
+services(_services),
+runtime(_runtime),
+metrics(_metrics),
+serviceManager(new ServiceManager(
+agentUrl,
+rootDir,
+info,
+services,
+containerPrefix,
+authToken,
+runtime,
+metrics))
+{
+ // This should have been validated in `VolumeManager::create`.
+ CHECK(!services.empty())
+<< "Must specify at least one service for CSI plugin type '" << info.type()
+<< "'
[mesos] 06/06: Supported CSI v1 in the test CSI plugin.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit a2f0f8b6b048b3f4147762641345ceb16878f565
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 21:36:12 2019 -0700
Supported CSI v1 in the test CSI plugin.
A new `--api_version` optional flag is added to the test CSI plugin. By
default, it will serve both CSI v0 and v1 requests. When this flag is
specified, only one CSI version will be served.
Review: https://reviews.apache.org/r/70404
---
include/mesos/csi/v0.hpp | 2 +
include/mesos/csi/v1.hpp | 2 +
src/examples/test_csi_plugin.cpp | 543 +--
3 files changed, 531 insertions(+), 16 deletions(-)
diff --git a/include/mesos/csi/v0.hpp b/include/mesos/csi/v0.hpp
index 8be070b..eb303ff 100644
--- a/include/mesos/csi/v0.hpp
+++ b/include/mesos/csi/v0.hpp
@@ -39,6 +39,8 @@ namespace v0 {
using namespace ::csi::v0;
+constexpr char API_VERSION[] = "v0";
+
} // namespace v0 {
} // namespace csi {
} // namespace mesos {
diff --git a/include/mesos/csi/v1.hpp b/include/mesos/csi/v1.hpp
index d4ebe42..9209111 100644
--- a/include/mesos/csi/v1.hpp
+++ b/include/mesos/csi/v1.hpp
@@ -39,6 +39,8 @@ namespace v1 {
using namespace ::csi::v1;
+constexpr char API_VERSION[] = "v1";
+
} // namespace v1 {
} // namespace csi {
} // namespace mesos {
diff --git a/src/examples/test_csi_plugin.cpp b/src/examples/test_csi_plugin.cpp
index 54753d9..b54d666 100644
--- a/src/examples/test_csi_plugin.cpp
+++ b/src/examples/test_csi_plugin.cpp
@@ -33,10 +33,12 @@
#include
#include
+#include
#include
#include
+#include
#include
#include
#include
@@ -56,6 +58,7 @@
#include
#include "csi/v0_utils.hpp"
+#include "csi/v1_utils.hpp"
#include "linux/fs.hpp"
@@ -106,6 +109,20 @@ class Flags : public virtual
mesos::internal::logging::Flags
public:
Flags()
{
+const hashset supportedApiVersions =
+ {mesos::csi::v0::API_VERSION, mesos::csi::v1::API_VERSION};
+
+add(::api_version,
+"api_version",
+"If set, the plugin would only serve CSI calls of the specified API\n"
+"version. Otherwise, the plugin serves all supported API versions by\n"
+"default. (Supported values: " + stringify(supportedApiVersions) + ")",
+[=](const Option& apiVersion) {
+ return apiVersion.isNone() ||
+supportedApiVersions.contains(apiVersion.get())
+ ? Option::none() : Error("Unsupported API version");
+});
+
add(::endpoint,
"endpoint",
"Path to the Unix domain socket the plugin should bind to.");
@@ -138,6 +155,7 @@ public:
"domain socket. (Example: 'unix:///path/to/socket')");
}
+ Option api_version;
string endpoint;
string work_dir;
Bytes available_capacity;
@@ -150,17 +168,22 @@ public:
class TestCSIPlugin
: public csi::v0::Identity::Service,
public csi::v0::Controller::Service,
-public csi::v0::Node::Service
+public csi::v0::Node::Service,
+public csi::v1::Identity::Service,
+public csi::v1::Controller::Service,
+public csi::v1::Node::Service
{
public:
TestCSIPlugin(
- const string& _workDir,
+ const Option& _apiVersion,
const string& _endpoint,
+ const string& _workDir,
const Bytes& _availableCapacity,
const hashmap& _createParameters,
const hashmap& _volumes)
-: workDir(_workDir),
+: apiVersion(_apiVersion),
endpoint(_endpoint),
+ workDir(_workDir),
availableCapacity(_availableCapacity),
createParameters(_createParameters.begin(), _createParameters.end())
{
@@ -208,6 +231,8 @@ public:
void run();
+ // CSI v0 RPCs.
+
Status GetPluginInfo(
ServerContext* context,
const csi::v0::GetPluginInfoRequest* request,
@@ -293,6 +318,93 @@ public:
const csi::v0::NodeGetCapabilitiesRequest* request,
csi::v0::NodeGetCapabilitiesResponse* response) override;
+ // CSI v1 RPCs.
+
+ Status GetPluginInfo(
+ ServerContext* context,
+ const csi::v1::GetPluginInfoRequest* request,
+ csi::v1::GetPluginInfoResponse* response) override;
+
+ Status GetPluginCapabilities(
+ ServerContext* context,
+ const csi::v1::GetPluginCapabilitiesRequest* request,
+ csi::v1::GetPluginCapabilitiesResponse* response) override;
+
+ Status Probe(
+ ServerContext* context,
+ const csi::v1::ProbeRequest* request,
+ csi::v1::ProbeResponse* response) override;
+
+ Status CreateVolume(
+ ServerContext* context,
+ const csi::v1::CreateVolumeRequest* request,
+ csi::v1::CreateVolumeResponse* response) override;
+
+ Status DeleteVolume(
+ ServerContext* context,
+ const csi::v1::DeleteVolumeRequest* request,
+ csi::v1::DeleteVolumeResponse* response) override;
+
+ Status
[mesos] 03/06: Updated CSI volume state fields to match names in v1 spec.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 51d11d93fb89d6822d6914d6e546265c672a19fe
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 22:44:34 2019 -0700
Updated CSI volume state fields to match names in v1 spec.
We rename `volume_attributes` to `volume_context` and `publish_info` to
`publish_context` in `state.proto` to match the new CSI v1 standard to
avoid confusions. Since this protobuf message is only used for
checkpointing and the binary format does not change, there will be no
compatibility issue.
Review: https://reviews.apache.org/r/70401
---
src/csi/state.proto | 13 +++--
src/csi/v0_volume_manager.cpp | 18 +-
src/csi/v0_volume_manager_process.hpp | 6 +++---
3 files changed, 19 insertions(+), 18 deletions(-)
diff --git a/src/csi/state.proto b/src/csi/state.proto
index b7213ed..96ba420 100644
--- a/src/csi/state.proto
+++ b/src/csi/state.proto
@@ -50,14 +50,15 @@ message VolumeState {
map parameters = 6;
// Attributes of the volume to be used on the node. This field MUST match the
- // attributes of the `Volume` returned by `CreateVolume`. This is an OPTIONAL
- // field.
- map volume_attributes = 3;
+ // `volume_context` field of the `Volume` returned by `CreateVolume`. This is
+ // an OPTIONAL field.
+ map volume_context = 3;
// If the plugin has the `PUBLISH_UNPUBLISH_VOLUME` controller capability,
- // this field MUST be set to the value returned by `ControllerPublishVolume`.
- // Otherwise, this field MUST remain unset. This is an OPTIONAL field.
- map publish_info = 4;
+ // this field MUST be set to the `publish_context` field returned by
+ // `ControllerPublishVolume`. Otherwise, this field MUST remain unset. This
is
+ // an OPTIONAL field.
+ map publish_context = 4;
// This field is used to check if the node has been rebooted since the volume
// is made publishable on the node. It MUST be set to the boot ID of the node
diff --git a/src/csi/v0_volume_manager.cpp b/src/csi/v0_volume_manager.cpp
index 7de9000..02de17e 100644
--- a/src/csi/v0_volume_manager.cpp
+++ b/src/csi/v0_volume_manager.cpp
@@ -326,7 +326,7 @@ Future VolumeManagerProcess::createVolume(
volumeState.set_state(VolumeState::CREATED);
*volumeState.mutable_volume_capability() = capability;
*volumeState.mutable_parameters() = parameters;
- *volumeState.mutable_volume_attributes() =
response.volume().attributes();
+ *volumeState.mutable_volume_context() = response.volume().attributes();
volumes.put(volumeId, std::move(volumeState));
checkpointVolumeState(volumeId);
@@ -397,7 +397,7 @@ Future> VolumeManagerProcess::validateVolume(
volumeState.set_state(VolumeState::CREATED);
*volumeState.mutable_volume_capability() = capability;
*volumeState.mutable_parameters() = parameters;
- *volumeState.mutable_volume_attributes() = volumeInfo.context;
+ *volumeState.mutable_volume_context() = volumeInfo.context;
volumes.put(volumeInfo.id, std::move(volumeState));
checkpointVolumeState(volumeInfo.id);
@@ -820,7 +820,7 @@ Future VolumeManagerProcess::_attachVolume(const
string& volumeId)
*request.mutable_volume_capability() =
evolve(volumeState.volume_capability());
request.set_readonly(false);
- *request.mutable_volume_attributes() = volumeState.volume_attributes();
+ *request.mutable_volume_attributes() = volumeState.volume_context();
return call(
CONTROLLER_SERVICE, ::controllerPublishVolume, std::move(request))
@@ -829,7 +829,7 @@ Future VolumeManagerProcess::_attachVolume(const
string& volumeId)
CHECK(volumes.contains(volumeId));
VolumeState& volumeState = volumes.at(volumeId).state;
volumeState.set_state(VolumeState::NODE_READY);
- *volumeState.mutable_publish_info() = response.publish_info();
+ *volumeState.mutable_publish_context() = response.publish_info();
checkpointVolumeState(volumeId);
@@ -886,7 +886,7 @@ Future VolumeManagerProcess::_detachVolume(const
string& volumeId)
CHECK(volumes.contains(volumeId));
VolumeState& volumeState = volumes.at(volumeId).state;
volumeState.set_state(VolumeState::CREATED);
- volumeState.mutable_publish_info()->clear();
+ volumeState.mutable_publish_context()->clear();
checkpointVolumeState(volumeId);
@@ -943,12 +943,12 @@ Future
VolumeManagerProcess::_publishVolume(const string& volumeId)
NodePublishVolumeRequest request;
request.set_volume_id(volumeId);
- *request.mutable_publish_info() = volumeState.publish_info();
+ *request.mutable_publish_info() = volumeState.publish_context();
request.set_target_path(targetPath);
*request.mutable_volume_capability() =
evolve(volumeState.volume_capability());
[mesos] 03/06: Updated CSI volume state fields to match names in v1 spec.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 275059a566be1cc8daecb998659767cf7572f588
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 22:44:34 2019 -0700
Updated CSI volume state fields to match names in v1 spec.
We rename `volume_attributes` to `volume_context` and `publish_info` to
`publish_context` in `state.proto` to match the new CSI v1 standard to
avoid confusions. Since this protobuf message is only used for
checkpointing and the binary format does not change, there will be no
compatibility issue.
Review: https://reviews.apache.org/r/70401
---
src/csi/state.proto | 13 +++--
src/csi/v0_volume_manager.cpp | 18 +-
src/csi/v0_volume_manager_process.hpp | 6 +++---
3 files changed, 19 insertions(+), 18 deletions(-)
diff --git a/src/csi/state.proto b/src/csi/state.proto
index b7213ed..96ba420 100644
--- a/src/csi/state.proto
+++ b/src/csi/state.proto
@@ -50,14 +50,15 @@ message VolumeState {
map parameters = 6;
// Attributes of the volume to be used on the node. This field MUST match the
- // attributes of the `Volume` returned by `CreateVolume`. This is an OPTIONAL
- // field.
- map volume_attributes = 3;
+ // `volume_context` field of the `Volume` returned by `CreateVolume`. This is
+ // an OPTIONAL field.
+ map volume_context = 3;
// If the plugin has the `PUBLISH_UNPUBLISH_VOLUME` controller capability,
- // this field MUST be set to the value returned by `ControllerPublishVolume`.
- // Otherwise, this field MUST remain unset. This is an OPTIONAL field.
- map publish_info = 4;
+ // this field MUST be set to the `publish_context` field returned by
+ // `ControllerPublishVolume`. Otherwise, this field MUST remain unset. This
is
+ // an OPTIONAL field.
+ map publish_context = 4;
// This field is used to check if the node has been rebooted since the volume
// is made publishable on the node. It MUST be set to the boot ID of the node
diff --git a/src/csi/v0_volume_manager.cpp b/src/csi/v0_volume_manager.cpp
index 7de9000..02de17e 100644
--- a/src/csi/v0_volume_manager.cpp
+++ b/src/csi/v0_volume_manager.cpp
@@ -326,7 +326,7 @@ Future VolumeManagerProcess::createVolume(
volumeState.set_state(VolumeState::CREATED);
*volumeState.mutable_volume_capability() = capability;
*volumeState.mutable_parameters() = parameters;
- *volumeState.mutable_volume_attributes() =
response.volume().attributes();
+ *volumeState.mutable_volume_context() = response.volume().attributes();
volumes.put(volumeId, std::move(volumeState));
checkpointVolumeState(volumeId);
@@ -397,7 +397,7 @@ Future> VolumeManagerProcess::validateVolume(
volumeState.set_state(VolumeState::CREATED);
*volumeState.mutable_volume_capability() = capability;
*volumeState.mutable_parameters() = parameters;
- *volumeState.mutable_volume_attributes() = volumeInfo.context;
+ *volumeState.mutable_volume_context() = volumeInfo.context;
volumes.put(volumeInfo.id, std::move(volumeState));
checkpointVolumeState(volumeInfo.id);
@@ -820,7 +820,7 @@ Future VolumeManagerProcess::_attachVolume(const
string& volumeId)
*request.mutable_volume_capability() =
evolve(volumeState.volume_capability());
request.set_readonly(false);
- *request.mutable_volume_attributes() = volumeState.volume_attributes();
+ *request.mutable_volume_attributes() = volumeState.volume_context();
return call(
CONTROLLER_SERVICE, ::controllerPublishVolume, std::move(request))
@@ -829,7 +829,7 @@ Future VolumeManagerProcess::_attachVolume(const
string& volumeId)
CHECK(volumes.contains(volumeId));
VolumeState& volumeState = volumes.at(volumeId).state;
volumeState.set_state(VolumeState::NODE_READY);
- *volumeState.mutable_publish_info() = response.publish_info();
+ *volumeState.mutable_publish_context() = response.publish_info();
checkpointVolumeState(volumeId);
@@ -886,7 +886,7 @@ Future VolumeManagerProcess::_detachVolume(const
string& volumeId)
CHECK(volumes.contains(volumeId));
VolumeState& volumeState = volumes.at(volumeId).state;
volumeState.set_state(VolumeState::CREATED);
- volumeState.mutable_publish_info()->clear();
+ volumeState.mutable_publish_context()->clear();
checkpointVolumeState(volumeId);
@@ -943,12 +943,12 @@ Future
VolumeManagerProcess::_publishVolume(const string& volumeId)
NodePublishVolumeRequest request;
request.set_volume_id(volumeId);
- *request.mutable_publish_info() = volumeState.publish_info();
+ *request.mutable_publish_info() = volumeState.publish_context();
request.set_target_path(targetPath);
*request.mutable_volume_capability() =
evolve(volumeState.volume_capability());
[mesos] branch 1.8.x updated (2c7cee0 -> 2e86f64)
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a change to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 2c7cee0 Updated CHANGELOG for 1.8.0.
new b4cc665 Added helpers to evolve/devolve repeated CSI v0
`VolumeCapability`s.
new e22d602 Added capabilities and evolve/devolve helpers for CSI v1.
new 275059a Updated CSI volume state fields to match names in v1 spec.
new a3c2632 Implemented CSI v1 volume manager.
new eca223c Refactored the test CSI plugin to make it easy to support CSI
v1.
new 2e86f64 Supported CSI v1 in the test CSI plugin.
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
include/mesos/csi/v0.hpp |2 +
include/mesos/csi/v1.hpp |2 +
src/CMakeLists.txt |2 +
src/Makefile.am|5 +
src/csi/state.proto| 13 +-
src/csi/v0_utils.cpp | 42 +-
src/csi/v0_utils.hpp | 27 +-
src/csi/v0_volume_manager.cpp | 18 +-
src/csi/v0_volume_manager_process.hpp |6 +-
src/csi/{v0_utils.cpp => v1_utils.cpp} | 48 +-
src/csi/v1_utils.hpp | 218 +++
...v0_volume_manager.cpp => v1_volume_manager.cpp} | 128 +-
...v0_volume_manager.hpp => v1_volume_manager.hpp} | 10 +-
...r_process.hpp => v1_volume_manager_process.hpp} | 16 +-
src/examples/test_csi_plugin.cpp | 1540 +++-
src/tests/csi_utils_tests.cpp | 50 +-
16 files changed, 1648 insertions(+), 479 deletions(-)
copy src/csi/{v0_utils.cpp => v1_utils.cpp} (85%)
create mode 100644 src/csi/v1_utils.hpp
copy src/csi/{v0_volume_manager.cpp => v1_volume_manager.cpp} (90%)
copy src/csi/{v0_volume_manager.hpp => v1_volume_manager.hpp} (95%)
copy src/csi/{v0_volume_manager_process.hpp => v1_volume_manager_process.hpp}
(96%)
[mesos] 02/06: Added capabilities and evolve/devolve helpers for CSI v1.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit e22d60212d3859b69b64b8b7c4d063856df29665
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 16:27:48 2019 -0700
Added capabilities and evolve/devolve helpers for CSI v1.
This patch adds capability helper structures for CSI v1 services, and
evolve/devolve helpers to convert between CSI v1 protobufs and their
unversioned counterparts.
Review: https://reviews.apache.org/r/70400
---
src/CMakeLists.txt| 1 +
src/Makefile.am | 2 +
src/csi/v1_utils.cpp | 242 ++
src/csi/v1_utils.hpp | 218 +
src/tests/csi_utils_tests.cpp | 50 +++--
5 files changed, 507 insertions(+), 6 deletions(-)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 51c7a04..d36d0be 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -243,6 +243,7 @@ set(CSI_SRC
csi/v0_volume_manager.cpp
csi/v1.cpp
csi/v1_client.cpp
+ csi/v1_utils.cpp
csi/volume_manager.cpp)
set(DOCKER_SRC
diff --git a/src/Makefile.am b/src/Makefile.am
index 9c0180f..64898df 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1598,6 +1598,8 @@ libcsi_la_SOURCES =
\
csi/v1.cpp \
csi/v1_client.cpp\
csi/v1_client.hpp\
+ csi/v1_utils.cpp \
+ csi/v1_utils.hpp \
csi/volume_manager.cpp \
csi/volume_manager.hpp
diff --git a/src/csi/v1_utils.cpp b/src/csi/v1_utils.cpp
new file mode 100644
index 000..e74138b
--- /dev/null
+++ b/src/csi/v1_utils.cpp
@@ -0,0 +1,242 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "csi/v1_utils.hpp"
+
+using google::protobuf::RepeatedPtrField;
+
+namespace mesos {
+namespace csi {
+namespace v1 {
+
+// Helper for repeated field devolving to `T1` from `T2`.
+template
+RepeatedPtrField devolve(RepeatedPtrField from)
+{
+ RepeatedPtrField to;
+ foreach (const T2& value, from) {
+*to.Add() = devolve(value);
+ }
+
+ return to;
+}
+
+
+types::VolumeCapability::BlockVolume devolve(
+const VolumeCapability::BlockVolume& block)
+{
+ return types::VolumeCapability::BlockVolume();
+}
+
+
+types::VolumeCapability::MountVolume devolve(
+const VolumeCapability::MountVolume& mount)
+{
+ types::VolumeCapability::MountVolume result;
+ result.set_fs_type(mount.fs_type());
+ *result.mutable_mount_flags() = mount.mount_flags();
+ return result;
+}
+
+
+types::VolumeCapability::AccessMode devolve(
+const VolumeCapability::AccessMode& accessMode)
+{
+ types::VolumeCapability::AccessMode result;
+
+ switch (accessMode.mode()) {
+case VolumeCapability::AccessMode::UNKNOWN: {
+ result.set_mode(types::VolumeCapability::AccessMode::UNKNOWN);
+ break;
+}
+case VolumeCapability::AccessMode::SINGLE_NODE_WRITER: {
+ result.set_mode(types::VolumeCapability::AccessMode::SINGLE_NODE_WRITER);
+ break;
+}
+case VolumeCapability::AccessMode::SINGLE_NODE_READER_ONLY: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::SINGLE_NODE_READER_ONLY);
+ break;
+}
+case VolumeCapability::AccessMode::MULTI_NODE_READER_ONLY: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::MULTI_NODE_READER_ONLY);
+ break;
+}
+case VolumeCapability::AccessMode::MULTI_NODE_SINGLE_WRITER: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::MULTI_NODE_SINGLE_WRITER);
+ break;
+}
+case VolumeCapability::AccessMode::MULTI_NODE_MULTI_WRITER: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::MULTI_NODE_MULTI_WRITER);
+ break;
+}
+// NOTE: We avoid using a default clause for the following values in
+//
[mesos] branch master updated (a41d190 -> a2f0f8b)
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from a41d190 Parameterized some allocator benchmarks for DRF and random
sorters.
new f9cf1c5 Added helpers to evolve/devolve repeated CSI v0
`VolumeCapability`s.
new 645c6ed Added capabilities and evolve/devolve helpers for CSI v1.
new 51d11d9 Updated CSI volume state fields to match names in v1 spec.
new 8c33596 Implemented CSI v1 volume manager.
new 8e60bc7 Refactored the test CSI plugin to make it easy to support CSI
v1.
new a2f0f8b Supported CSI v1 in the test CSI plugin.
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
include/mesos/csi/v0.hpp |2 +
include/mesos/csi/v1.hpp |2 +
src/CMakeLists.txt |2 +
src/Makefile.am|5 +
src/csi/state.proto| 13 +-
src/csi/v0_utils.cpp | 42 +-
src/csi/v0_utils.hpp | 27 +-
src/csi/v0_volume_manager.cpp | 18 +-
src/csi/v0_volume_manager_process.hpp |6 +-
src/csi/{v0_utils.cpp => v1_utils.cpp} | 48 +-
src/csi/v1_utils.hpp | 218 +++
...v0_volume_manager.cpp => v1_volume_manager.cpp} | 128 +-
...v0_volume_manager.hpp => v1_volume_manager.hpp} | 10 +-
...r_process.hpp => v1_volume_manager_process.hpp} | 16 +-
src/examples/test_csi_plugin.cpp | 1540 +++-
src/tests/csi_utils_tests.cpp | 50 +-
16 files changed, 1648 insertions(+), 479 deletions(-)
copy src/csi/{v0_utils.cpp => v1_utils.cpp} (85%)
create mode 100644 src/csi/v1_utils.hpp
copy src/csi/{v0_volume_manager.cpp => v1_volume_manager.cpp} (90%)
copy src/csi/{v0_volume_manager.hpp => v1_volume_manager.hpp} (95%)
copy src/csi/{v0_volume_manager_process.hpp => v1_volume_manager_process.hpp}
(96%)
[mesos] 01/06: Added helpers to evolve/devolve repeated CSI v0 `VolumeCapability`s.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit b4cc665de5ab74cb527c6fa67b2bb40fd781db5c
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 15:37:35 2019 -0700
Added helpers to evolve/devolve repeated CSI v0 `VolumeCapability`s.
Review: https://reviews.apache.org/r/70399
---
src/csi/v0_utils.cpp | 42 +-
src/csi/v0_utils.hpp | 27 +--
2 files changed, 66 insertions(+), 3 deletions(-)
diff --git a/src/csi/v0_utils.cpp b/src/csi/v0_utils.cpp
index a95d240..248e417 100644
--- a/src/csi/v0_utils.cpp
+++ b/src/csi/v0_utils.cpp
@@ -16,12 +16,25 @@
#include "csi/v0_utils.hpp"
-#include
+using google::protobuf::RepeatedPtrField;
namespace mesos {
namespace csi {
namespace v0 {
+// Helper for repeated field devolving to `T1` from `T2`.
+template
+RepeatedPtrField devolve(RepeatedPtrField from)
+{
+ RepeatedPtrField to;
+ foreach (const T2& value, from) {
+*to.Add() = devolve(value);
+ }
+
+ return to;
+}
+
+
types::VolumeCapability::BlockVolume devolve(
const VolumeCapability::BlockVolume& block)
{
@@ -112,6 +125,26 @@ types::VolumeCapability devolve(const VolumeCapability&
capability)
}
+RepeatedPtrField devolve(
+const RepeatedPtrField& capabilities)
+{
+ return devolve(capabilities);
+}
+
+
+// Helper for repeated field evolving to `T1` from `T2`.
+template
+RepeatedPtrField evolve(RepeatedPtrField from)
+{
+ RepeatedPtrField to;
+ foreach (const T2& value, from) {
+*to.Add() = evolve(value);
+ }
+
+ return to;
+}
+
+
VolumeCapability::BlockVolume evolve(
const types::VolumeCapability::BlockVolume& block)
{
@@ -197,6 +230,13 @@ VolumeCapability evolve(const types::VolumeCapability&
capability)
return result;
}
+
+RepeatedPtrField devolve(
+const RepeatedPtrField& capabilities)
+{
+ return evolve(capabilities);
+}
+
} // namespace v0 {
} // namespace csi {
} // namespace mesos {
diff --git a/src/csi/v0_utils.hpp b/src/csi/v0_utils.hpp
index 46a5f13..468b416 100644
--- a/src/csi/v0_utils.hpp
+++ b/src/csi/v0_utils.hpp
@@ -17,6 +17,8 @@
#ifndef __CSI_V0_UTILS_HPP__
#define __CSI_V0_UTILS_HPP__
+#include
+
#include
#include
@@ -31,7 +33,8 @@ struct PluginCapabilities
{
PluginCapabilities() = default;
- template PluginCapabilities(const Iterable& capabilities)
+ template
+ PluginCapabilities(const Iterable& capabilities)
{
foreach (const auto& capability, capabilities) {
if (capability.has_service() &&
@@ -43,6 +46,10 @@ struct PluginCapabilities
case PluginCapability::Service::CONTROLLER_SERVICE:
controllerService = true;
break;
+
+ // NOTE: We avoid using a default clause for the following values in
+ // proto3's open enum to enable the compiler to detect missing enum
+ // cases for us. See: https://github.com/google/protobuf/issues/3917
case google::protobuf::kint32min:
case google::protobuf::kint32max:
UNREACHABLE();
@@ -81,6 +88,10 @@ struct ControllerCapabilities
case ControllerServiceCapability::RPC::GET_CAPACITY:
getCapacity = true;
break;
+
+ // NOTE: We avoid using a default clause for the following values in
+ // proto3's open enum to enable the compiler to detect missing enum
+ // cases for us. See: https://github.com/google/protobuf/issues/3917
case google::protobuf::kint32min:
case google::protobuf::kint32max:
UNREACHABLE();
@@ -100,7 +111,8 @@ struct NodeCapabilities
{
NodeCapabilities() = default;
- template NodeCapabilities(const Iterable& capabilities)
+ template
+ NodeCapabilities(const Iterable& capabilities)
{
foreach (const auto& capability, capabilities) {
if (capability.has_rpc() &&
@@ -111,6 +123,10 @@ struct NodeCapabilities
case NodeServiceCapability::RPC::STAGE_UNSTAGE_VOLUME:
stageUnstageVolume = true;
break;
+
+ // NOTE: We avoid using a default clause for the following values in
+ // proto3's open enum to enable the compiler to detect missing enum
+ // cases for us. See: https://github.com/google/protobuf/issues/3917
case google::protobuf::kint32min:
case google::protobuf::kint32max:
UNREACHABLE();
@@ -126,10 +142,17 @@ struct NodeCapabilities
// Helpers to devolve CSI v0 protobufs to their unversioned counterparts.
types::VolumeCapability devolve(const VolumeCapability& capability);
+google::protobuf::RepeatedPtrField devolve(
+const google::protobuf::RepeatedPtrField& capabilities);
+
// Helpers to evolve unversioned CSI protobufs to their v0 counterparts.
VolumeCapability evolve(const types::VolumeCapability& capability);
[mesos] 02/06: Added capabilities and evolve/devolve helpers for CSI v1.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 645c6ed89852404b0ee483f12f1ced354ced13fc
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 16:27:48 2019 -0700
Added capabilities and evolve/devolve helpers for CSI v1.
This patch adds capability helper structures for CSI v1 services, and
evolve/devolve helpers to convert between CSI v1 protobufs and their
unversioned counterparts.
Review: https://reviews.apache.org/r/70400
---
src/CMakeLists.txt| 1 +
src/Makefile.am | 2 +
src/csi/v1_utils.cpp | 242 ++
src/csi/v1_utils.hpp | 218 +
src/tests/csi_utils_tests.cpp | 50 +++--
5 files changed, 507 insertions(+), 6 deletions(-)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 51c7a04..d36d0be 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -243,6 +243,7 @@ set(CSI_SRC
csi/v0_volume_manager.cpp
csi/v1.cpp
csi/v1_client.cpp
+ csi/v1_utils.cpp
csi/volume_manager.cpp)
set(DOCKER_SRC
diff --git a/src/Makefile.am b/src/Makefile.am
index 9c0180f..64898df 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1598,6 +1598,8 @@ libcsi_la_SOURCES =
\
csi/v1.cpp \
csi/v1_client.cpp\
csi/v1_client.hpp\
+ csi/v1_utils.cpp \
+ csi/v1_utils.hpp \
csi/volume_manager.cpp \
csi/volume_manager.hpp
diff --git a/src/csi/v1_utils.cpp b/src/csi/v1_utils.cpp
new file mode 100644
index 000..e74138b
--- /dev/null
+++ b/src/csi/v1_utils.cpp
@@ -0,0 +1,242 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "csi/v1_utils.hpp"
+
+using google::protobuf::RepeatedPtrField;
+
+namespace mesos {
+namespace csi {
+namespace v1 {
+
+// Helper for repeated field devolving to `T1` from `T2`.
+template
+RepeatedPtrField devolve(RepeatedPtrField from)
+{
+ RepeatedPtrField to;
+ foreach (const T2& value, from) {
+*to.Add() = devolve(value);
+ }
+
+ return to;
+}
+
+
+types::VolumeCapability::BlockVolume devolve(
+const VolumeCapability::BlockVolume& block)
+{
+ return types::VolumeCapability::BlockVolume();
+}
+
+
+types::VolumeCapability::MountVolume devolve(
+const VolumeCapability::MountVolume& mount)
+{
+ types::VolumeCapability::MountVolume result;
+ result.set_fs_type(mount.fs_type());
+ *result.mutable_mount_flags() = mount.mount_flags();
+ return result;
+}
+
+
+types::VolumeCapability::AccessMode devolve(
+const VolumeCapability::AccessMode& accessMode)
+{
+ types::VolumeCapability::AccessMode result;
+
+ switch (accessMode.mode()) {
+case VolumeCapability::AccessMode::UNKNOWN: {
+ result.set_mode(types::VolumeCapability::AccessMode::UNKNOWN);
+ break;
+}
+case VolumeCapability::AccessMode::SINGLE_NODE_WRITER: {
+ result.set_mode(types::VolumeCapability::AccessMode::SINGLE_NODE_WRITER);
+ break;
+}
+case VolumeCapability::AccessMode::SINGLE_NODE_READER_ONLY: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::SINGLE_NODE_READER_ONLY);
+ break;
+}
+case VolumeCapability::AccessMode::MULTI_NODE_READER_ONLY: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::MULTI_NODE_READER_ONLY);
+ break;
+}
+case VolumeCapability::AccessMode::MULTI_NODE_SINGLE_WRITER: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::MULTI_NODE_SINGLE_WRITER);
+ break;
+}
+case VolumeCapability::AccessMode::MULTI_NODE_MULTI_WRITER: {
+ result.set_mode(
+ types::VolumeCapability::AccessMode::MULTI_NODE_MULTI_WRITER);
+ break;
+}
+// NOTE: We avoid using a default clause for the following values in
+//
[mesos] 04/06: Implemented CSI v1 volume manager.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 8c33596d7bdf490248064a7294e892497d2d334f
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 22:47:49 2019 -0700
Implemented CSI v1 volume manager.
The v1 Volume manager is basically copied from the v0 volume manager,
with the following changes:
* `validateVolume` will validate volumes against parameters.
* `publishVolume` will only create the parent directory of the target
path. The creation of the target path itself is the responsibility
of the plugin.
* The node ID is obtained through the `NodeGetInfo` call.
Review: https://reviews.apache.org/r/70402
---
src/CMakeLists.txt|1 +
src/Makefile.am |3 +
src/csi/v1_volume_manager.cpp | 1356 +
src/csi/v1_volume_manager.hpp | 116 +++
src/csi/v1_volume_manager_process.hpp | 218 ++
5 files changed, 1694 insertions(+)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index d36d0be..1d4f541 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -244,6 +244,7 @@ set(CSI_SRC
csi/v1.cpp
csi/v1_client.cpp
csi/v1_utils.cpp
+ csi/v1_volume_manager.cpp
csi/volume_manager.cpp)
set(DOCKER_SRC
diff --git a/src/Makefile.am b/src/Makefile.am
index 64898df..5f97523 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1600,6 +1600,9 @@ libcsi_la_SOURCES =
\
csi/v1_client.hpp\
csi/v1_utils.cpp \
csi/v1_utils.hpp \
+ csi/v1_volume_manager.cpp\
+ csi/v1_volume_manager.hpp\
+ csi/v1_volume_manager_process.hpp\
csi/volume_manager.cpp \
csi/volume_manager.hpp
diff --git a/src/csi/v1_volume_manager.cpp b/src/csi/v1_volume_manager.cpp
new file mode 100644
index 000..bd334f1
--- /dev/null
+++ b/src/csi/v1_volume_manager.cpp
@@ -0,0 +1,1356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "csi/v1_volume_manager.hpp"
+
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "csi/paths.hpp"
+#include "csi/v1_client.hpp"
+#include "csi/v1_utils.hpp"
+#include "csi/v1_volume_manager_process.hpp"
+
+#include "slave/state.hpp"
+
+namespace http = process::http;
+namespace slave = mesos::internal::slave;
+
+using std::list;
+using std::string;
+using std::vector;
+
+using google::protobuf::Map;
+
+using mesos::csi::state::VolumeState;
+
+using process::Break;
+using process::Continue;
+using process::ControlFlow;
+using process::Failure;
+using process::Future;
+using process::ProcessBase;
+
+using process::grpc::StatusError;
+
+using process::grpc::client::Runtime;
+
+namespace mesos{
+namespace csi {
+namespace v1 {
+
+VolumeManagerProcess::VolumeManagerProcess(
+const http::URL& agentUrl,
+const string& _rootDir,
+const CSIPluginInfo& _info,
+const hashset _services,
+const string& containerPrefix,
+const Option& authToken,
+const Runtime& _runtime,
+Metrics* _metrics)
+ : ProcessBase(process::ID::generate("csi-v1-volume-manager")),
+rootDir(_rootDir),
+info(_info),
+services(_services),
+runtime(_runtime),
+metrics(_metrics),
+serviceManager(new ServiceManager(
+agentUrl,
+rootDir,
+info,
+services,
+containerPrefix,
+authToken,
+runtime,
+metrics))
+{
+ // This should have been validated in `VolumeManager::create`.
+ CHECK(!services.empty())
+<< "Must specify at least one service for CSI plugin type '" << info.type()
+<< "'
[mesos] 01/06: Added helpers to evolve/devolve repeated CSI v0 `VolumeCapability`s.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit f9cf1c5706093e0c05b2555e5bd03da4011dd5db
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 15:37:35 2019 -0700
Added helpers to evolve/devolve repeated CSI v0 `VolumeCapability`s.
Review: https://reviews.apache.org/r/70399
---
src/csi/v0_utils.cpp | 42 +-
src/csi/v0_utils.hpp | 27 +--
2 files changed, 66 insertions(+), 3 deletions(-)
diff --git a/src/csi/v0_utils.cpp b/src/csi/v0_utils.cpp
index a95d240..248e417 100644
--- a/src/csi/v0_utils.cpp
+++ b/src/csi/v0_utils.cpp
@@ -16,12 +16,25 @@
#include "csi/v0_utils.hpp"
-#include
+using google::protobuf::RepeatedPtrField;
namespace mesos {
namespace csi {
namespace v0 {
+// Helper for repeated field devolving to `T1` from `T2`.
+template
+RepeatedPtrField devolve(RepeatedPtrField from)
+{
+ RepeatedPtrField to;
+ foreach (const T2& value, from) {
+*to.Add() = devolve(value);
+ }
+
+ return to;
+}
+
+
types::VolumeCapability::BlockVolume devolve(
const VolumeCapability::BlockVolume& block)
{
@@ -112,6 +125,26 @@ types::VolumeCapability devolve(const VolumeCapability&
capability)
}
+RepeatedPtrField devolve(
+const RepeatedPtrField& capabilities)
+{
+ return devolve(capabilities);
+}
+
+
+// Helper for repeated field evolving to `T1` from `T2`.
+template
+RepeatedPtrField evolve(RepeatedPtrField from)
+{
+ RepeatedPtrField to;
+ foreach (const T2& value, from) {
+*to.Add() = evolve(value);
+ }
+
+ return to;
+}
+
+
VolumeCapability::BlockVolume evolve(
const types::VolumeCapability::BlockVolume& block)
{
@@ -197,6 +230,13 @@ VolumeCapability evolve(const types::VolumeCapability&
capability)
return result;
}
+
+RepeatedPtrField devolve(
+const RepeatedPtrField& capabilities)
+{
+ return evolve(capabilities);
+}
+
} // namespace v0 {
} // namespace csi {
} // namespace mesos {
diff --git a/src/csi/v0_utils.hpp b/src/csi/v0_utils.hpp
index 46a5f13..468b416 100644
--- a/src/csi/v0_utils.hpp
+++ b/src/csi/v0_utils.hpp
@@ -17,6 +17,8 @@
#ifndef __CSI_V0_UTILS_HPP__
#define __CSI_V0_UTILS_HPP__
+#include
+
#include
#include
@@ -31,7 +33,8 @@ struct PluginCapabilities
{
PluginCapabilities() = default;
- template PluginCapabilities(const Iterable& capabilities)
+ template
+ PluginCapabilities(const Iterable& capabilities)
{
foreach (const auto& capability, capabilities) {
if (capability.has_service() &&
@@ -43,6 +46,10 @@ struct PluginCapabilities
case PluginCapability::Service::CONTROLLER_SERVICE:
controllerService = true;
break;
+
+ // NOTE: We avoid using a default clause for the following values in
+ // proto3's open enum to enable the compiler to detect missing enum
+ // cases for us. See: https://github.com/google/protobuf/issues/3917
case google::protobuf::kint32min:
case google::protobuf::kint32max:
UNREACHABLE();
@@ -81,6 +88,10 @@ struct ControllerCapabilities
case ControllerServiceCapability::RPC::GET_CAPACITY:
getCapacity = true;
break;
+
+ // NOTE: We avoid using a default clause for the following values in
+ // proto3's open enum to enable the compiler to detect missing enum
+ // cases for us. See: https://github.com/google/protobuf/issues/3917
case google::protobuf::kint32min:
case google::protobuf::kint32max:
UNREACHABLE();
@@ -100,7 +111,8 @@ struct NodeCapabilities
{
NodeCapabilities() = default;
- template NodeCapabilities(const Iterable& capabilities)
+ template
+ NodeCapabilities(const Iterable& capabilities)
{
foreach (const auto& capability, capabilities) {
if (capability.has_rpc() &&
@@ -111,6 +123,10 @@ struct NodeCapabilities
case NodeServiceCapability::RPC::STAGE_UNSTAGE_VOLUME:
stageUnstageVolume = true;
break;
+
+ // NOTE: We avoid using a default clause for the following values in
+ // proto3's open enum to enable the compiler to detect missing enum
+ // cases for us. See: https://github.com/google/protobuf/issues/3917
case google::protobuf::kint32min:
case google::protobuf::kint32max:
UNREACHABLE();
@@ -126,10 +142,17 @@ struct NodeCapabilities
// Helpers to devolve CSI v0 protobufs to their unversioned counterparts.
types::VolumeCapability devolve(const VolumeCapability& capability);
+google::protobuf::RepeatedPtrField devolve(
+const google::protobuf::RepeatedPtrField& capabilities);
+
// Helpers to evolve unversioned CSI protobufs to their v0 counterparts.
VolumeCapability evolve(const types::VolumeCapability& capability);
[mesos] 05/06: Refactored the test CSI plugin to make it easy to support CSI v1.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 8e60bc7a921a2d4b37e5bec32ad3a10ddec96e78
Author: Chun-Hung Hsiao
AuthorDate: Thu Apr 4 18:48:24 2019 -0700
Refactored the test CSI plugin to make it easy to support CSI v1.
Review: https://reviews.apache.org/r/70403
---
src/examples/test_csi_plugin.cpp | 951 +--
1 file changed, 606 insertions(+), 345 deletions(-)
diff --git a/src/examples/test_csi_plugin.cpp b/src/examples/test_csi_plugin.cpp
index 4321f8f..54753d9 100644
--- a/src/examples/test_csi_plugin.cpp
+++ b/src/examples/test_csi_plugin.cpp
@@ -14,9 +14,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+#include
+#include
#include
#include
#include
+#include
+
+#include
+#include
#include
@@ -25,16 +31,24 @@
#include
+#include
#include
+#include
+#include
+
#include
#include
#include
#include
+#include
#include
#include
+#include
#include
#include
+#include
+#include
#include
#include
@@ -47,6 +61,7 @@
#include "logging/logging.hpp"
+namespace http = process::http;
namespace fs = mesos::internal::fs;
using std::cerr;
@@ -59,6 +74,10 @@ using std::string;
using std::unique_ptr;
using std::vector;
+using google::protobuf::Map;
+using google::protobuf::MapPair;
+using google::protobuf::RepeatedPtrField;
+
using grpc::AsyncGenericService;
using grpc::ByteBuffer;
using grpc::ClientContext;
@@ -73,6 +92,10 @@ using grpc::ServerContext;
using grpc::Status;
using grpc::WriteOptions;
+using mesos::csi::types::VolumeCapability;
+
+using process::grpc::StatusError;
+
constexpr char PLUGIN_NAME[] = "org.apache.mesos.csi.test";
constexpr char NODE_ID[] = "localhost";
constexpr Bytes DEFAULT_VOLUME_CAPACITY = Megabytes(64);
@@ -144,7 +167,7 @@ public:
// Construct the default mount volume capability.
defaultVolumeCapability.mutable_mount();
defaultVolumeCapability.mutable_access_mode()
- ->set_mode(csi::v0::VolumeCapability::AccessMode::SINGLE_NODE_WRITER);
+ ->set_mode(VolumeCapability::AccessMode::SINGLE_NODE_WRITER);
// Scan for preprovisioned volumes.
//
@@ -280,12 +303,69 @@ private:
string getVolumePath(const VolumeInfo& volumeInfo);
Try parseVolumePath(const string& path);
+ Try createVolume(
+ const string& name,
+ const Bytes& requiredBytes,
+ const Bytes& limitBytes,
+ const RepeatedPtrField& capabilities,
+ const Map parameters);
+
+ Try deleteVolume(const string& volumeId);
+
+ Try controllerPublishVolume(
+ const string& volumeId,
+ const string& nodeId,
+ const VolumeCapability& capability,
+ bool readonly,
+ const Map& volumeContext);
+
+ Try controllerUnpublishVolume(
+ const string& volumeId, const string& nodeId);
+
+ // Returns `StatusError` if the volume does not exist; returns
`Option`
+ // with an error set if the volume is not compatible with the given
arguments.
+ Try, StatusError> validateVolumeCapabilities(
+ const string& volumeId,
+ const Map& volumeContext,
+ const RepeatedPtrField& capabilities,
+ const Option>& parameters = None());
+
+ Try, StatusError> listVolumes(
+ const Option& maxEntries,
+ const Option& startingToken);
+
+ Try getCapacity(
+ const RepeatedPtrField& capabilities,
+ const Map& parameters);
+
+ Try nodeStageVolume(
+ const string& volumeId,
+ const Map& publishContext,
+ const string& stagingPath,
+ const VolumeCapability& capability,
+ const Map& volumeContext);
+
+ Try nodeUnstageVolume(
+ const string& volumeId, const string& stagingPath);
+
+ Try nodePublishVolume(
+ const string& volumeId,
+ const Map& publishContext,
+ const string& stagingPath,
+ const string& targetPath,
+ const VolumeCapability& capability,
+ bool readonly,
+ const Map& volumeContext);
+
+ Try nodeUnpublishVolume(
+ const string& volumeId, const string& targetPath);
+
const string workDir;
const string endpoint;
Bytes availableCapacity;
- csi::v0::VolumeCapability defaultVolumeCapability;
- google::protobuf::Map createParameters;
+ VolumeCapability defaultVolumeCapability;
+ Map createParameters;
hashmap volumes;
};
@@ -349,92 +429,26 @@ Status TestCSIPlugin::CreateVolume(
{
LOG(INFO) << request->GetDescriptor()->name() << " '" << *request << "'";
- // TODO(chhsiao): Validate required fields.
+ // TODO(chhsiao): Validate the request.
- if (request->name().empty()) {
-return Status(grpc::INVALID_ARGUMENT, "Volume name cannot be empty");
- }
-
- if (strings::contains(request->name(), stringify(os::PATH_SEPARATOR))) {
-return Status(
-grpc::INVALID_ARGUMENT,
-"Volume name cannot contain '" +
