[mesos] 01/02: Supported file operations for command tasks.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit f24c54e85e08bc9c8b118cce29ad487661a0ffc6
Author: Qian Zhang
AuthorDate: Tue Jun 11 15:50:43 2019 -0700
Supported file operations for command tasks.
Review: https://reviews.apache.org/r/70826/
---
src/launcher/executor.cpp | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/launcher/executor.cpp b/src/launcher/executor.cpp
index fa4bcaa..38d8261 100644
--- a/src/launcher/executor.cpp
+++ b/src/launcher/executor.cpp
@@ -494,6 +494,9 @@ protected:
if (taskLaunchInfo.isSome()) {
launchInfo.mutable_mounts()->CopyFrom(taskLaunchInfo->mounts());
+ launchInfo.mutable_file_operations()->CopyFrom(
+ taskLaunchInfo->file_operations());
+
launchInfo.mutable_pre_exec_commands()->CopyFrom(
taskLaunchInfo->pre_exec_commands());
[mesos] branch master updated (46c7349 -> 1961e41)
This is an automated email from the ASF dual-hosted git repository. gilbert pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from 46c7349 Fixed test `QuotaRoleAllocateNonQuotaResource`. new f24c54e Supported file operations for command tasks. new 1961e41 Improved container-specific cgroups test by checking `cpu.shares`. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/launcher/executor.cpp | 3 +++ src/tests/containerizer/cgroups_isolator_tests.cpp | 16 2 files changed, 11 insertions(+), 8 deletions(-)
[mesos] 02/02: Improved container-specific cgroups test by checking `cpu.shares`.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 1961e41a61def2b7baca7563c0b7e1855880b55c
Author: Qian Zhang
AuthorDate: Tue Jun 11 15:50:47 2019 -0700
Improved container-specific cgroups test by checking `cpu.shares`.
This is to ensure the symbolic links (see below as an example) we
create for the container exist.
ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpu
Review: https://reviews.apache.org/r/70827/
---
src/tests/containerizer/cgroups_isolator_tests.cpp | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/tests/containerizer/cgroups_isolator_tests.cpp
b/src/tests/containerizer/cgroups_isolator_tests.cpp
index 957f72d..eef8eb2 100644
--- a/src/tests/containerizer/cgroups_isolator_tests.cpp
+++ b/src/tests/containerizer/cgroups_isolator_tests.cpp
@@ -2185,7 +2185,7 @@ TEST_F(CgroupsIsolatorTest,
ROOT_CGROUPS_NestedContainerSpecificCgroupsMount)
{
// Disable AuthN on the agent.
slave::Flags flags = CreateSlaveFlags();
- flags.isolation = "filesystem/linux,docker/runtime,cgroups/mem";
+ flags.isolation = "filesystem/linux,docker/runtime,cgroups/mem,cgroups/cpu";
flags.image_providers = "docker";
flags.authenticate_http_readwrite = false;
@@ -2234,9 +2234,8 @@ TEST_F(CgroupsIsolatorTest,
ROOT_CGROUPS_NestedContainerSpecificCgroupsMount)
const v1::Offer& offer = offers->offers(0);
- // Create a task to check if its memory (including both executor and task's
- // memory) is correctly set in its specific cgroup, e.g.:
- // `/sys/fs/cgroup/memory/memory.soft_limit_in_bytes`
+ // Create a task to check if its memory and CPU shares (including both
+ // executor's and task's) are correctly set in its specific cgroup.
//
// And we also verify the freezer cgroup is correctly mounted for this task
// by checking if the current shell PID is included in the freezer cgroup.
@@ -2244,6 +2243,7 @@ TEST_F(CgroupsIsolatorTest,
ROOT_CGROUPS_NestedContainerSpecificCgroupsMount)
offer.agent_id(),
v1::Resources::parse("cpus:0.1;mem:32;disk:32").get(),
"test `cat /sys/fs/cgroup/memory/memory.soft_limit_in_bytes` = 67108864 "
+ "&& test `cat /sys/fs/cgroup/cpu/cpu.shares` = 204"
"&& grep $$ /sys/fs/cgroup/freezer/cgroup.procs");
mesos::v1::Image image;
@@ -2299,7 +2299,7 @@ TEST_F(CgroupsIsolatorTest,
ROOT_CGROUPS_CommandTaskSpecificCgroupsMount)
Owned detector = master.get()->createDetector();
slave::Flags flags = CreateSlaveFlags();
- flags.isolation = "filesystem/linux,docker/runtime,cgroups/mem";
+ flags.isolation = "filesystem/linux,docker/runtime,cgroups/mem,cgroups/cpu";
flags.image_providers = "docker";
Try> slave = StartSlave(detector.get(), flags);
@@ -2321,9 +2321,8 @@ TEST_F(CgroupsIsolatorTest,
ROOT_CGROUPS_CommandTaskSpecificCgroupsMount)
AWAIT_READY(offers);
EXPECT_EQ(1u, offers->size());
- // Create a task to check if its memory (including both executor and task's
- // memory) is correctly set in its specific cgroup, e.g.:
- // `/sys/fs/cgroup/memory/memory.soft_limit_in_bytes`
+ // Create a task to check if its memory and CPU shares (including both
+ // executor's and task's) are correctly set in its specific cgroup.
//
// And we also verify the freezer cgroup is correctly mounted for this task
// by checking if the current shell PID is included in the freezer cgroup.
@@ -2331,6 +2330,7 @@ TEST_F(CgroupsIsolatorTest,
ROOT_CGROUPS_CommandTaskSpecificCgroupsMount)
offers->front().slave_id(),
Resources::parse("cpus:0.1;mem:32;disk:32").get(),
"test `cat /sys/fs/cgroup/memory/memory.soft_limit_in_bytes` = 67108864 "
+ "&& test `cat /sys/fs/cgroup/cpu/cpu.shares` = 204"
"&& grep $$ /sys/fs/cgroup/freezer/cgroup.procs");
Image image;
[mesos] branch master updated: Fixed test `QuotaRoleAllocateNonQuotaResource`.
This is an automated email from the ASF dual-hosted git repository.
mzhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 46c7349 Fixed test `QuotaRoleAllocateNonQuotaResource`.
46c7349 is described below
commit 46c7349c9f8acc387edc8c9737eee02385fcee04
Author: Meng Zhu
AuthorDate: Mon Jun 10 17:07:50 2019 -0700
Fixed test `QuotaRoleAllocateNonQuotaResource`.
The test was failing because:
After `agent3` is added, it misses a settle call where the allocation
of `agent3` is racy.
In addition, after MESOS-8456, the allocator now offers non-quota
resources on an agent (even that means "chopping") on top of a role's
satisfied guarantees instead of skipping the agent all-together.
This patch fixes the test by expecting the right amount of resources
to be allocated from `agent3`.
Review: https://reviews.apache.org/r/70824
---
src/tests/hierarchical_allocator_tests.cpp | 18 --
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/tests/hierarchical_allocator_tests.cpp
b/src/tests/hierarchical_allocator_tests.cpp
index bfea34e..7dc3dc1 100644
--- a/src/tests/hierarchical_allocator_tests.cpp
+++ b/src/tests/hierarchical_allocator_tests.cpp
@@ -4090,8 +4090,22 @@ TEST_F(HierarchicalAllocatorTest,
QuotaRoleAllocateNonQuotaResource)
agent3.resources(),
{});
- // No allocation will happen because QUOTA_ROLE_1's quota has been met.
- EXPECT_TRUE(allocations.get().isPending());
+ Clock::settle();
+
+ // `QUOTA_ROLE_1` quota has been reached. Only resources with the default
+ // quota (i.e. no limits) are allocated.
+
+ Resources nonQuotaResources =
+Resources(agent3.resources()).filter([&](const Resource& resource) {
+ return ResourceQuantities::fromScalarResources(
+ Resources(quota1.info.guarantee()))
+ .get(resource.name()) == Value::Scalar();
+});
+
+ expected = Allocation(
+ framework.id(), {{QUOTA_ROLE_1, {{agent3.id(), nonQuotaResources);
+
+ AWAIT_EXPECT_EQ(expected, allocations.get());
}
[mesos] branch master updated: Supported updating framework info in the scheduler driver.
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 4036da6 Supported updating framework info in the scheduler driver.
4036da6 is described below
commit 4036da6638427aa05f66f6780ce5d11049f95bb3
Author: Andrei Sekretenko
AuthorDate: Tue Jun 11 13:31:09 2019 -0400
Supported updating framework info in the scheduler driver.
This patch adds a method to the MesosSchedulerDriver which updates the
FrameworkInfo (by making an UPDATE_FRAMEWORK call to the master and also
updating the FrameworkInfo stored in the driver for the purposes of
(re-)subscribing).
Review: https://reviews.apache.org/r/70752/
---
include/mesos/scheduler.hpp | 13 ++
src/sched/sched.cpp | 111 ++--
2 files changed, 110 insertions(+), 14 deletions(-)
diff --git a/include/mesos/scheduler.hpp b/include/mesos/scheduler.hpp
index 2ea7cdf..c5e61d4 100644
--- a/include/mesos/scheduler.hpp
+++ b/include/mesos/scheduler.hpp
@@ -317,6 +317,17 @@ public:
// currently known.
virtual Status reconcileTasks(
const std::vector& statuses) = 0;
+
+ // Updates the FrameworkInfo with the provided value (except for the
+ // framework_id field, which should not be set by the caller).
+ // The driver implementation should send the supplied FrameworkInfo update
+ // to the master. Also, all the next re-registration attempts will be
+ // performed with the provided FrameworkInfo.
+ //
+ // NOTE: If the supplied info is invalid or fails authorization,
+ // the `error()` callback will be invoked asynchronously (after
+ // the master replies with a FrameworkErrorMessage).
+ virtual Status updateFramework(const FrameworkInfo& frameworkInfo) = 0;
};
@@ -454,6 +465,8 @@ public:
Status reconcileTasks(
const std::vector& statuses) override;
+ Status updateFramework(const FrameworkInfo& frameworkInfo) override;
+
protected:
// Used to detect (i.e., choose) the master.
std::shared_ptr detector;
diff --git a/src/sched/sched.cpp b/src/sched/sched.cpp
index e77a029..281236b 100644
--- a/src/sched/sched.cpp
+++ b/src/sched/sched.cpp
@@ -218,6 +218,7 @@ public:
latch(_latch),
failover(_framework.has_id() && !framework.id().value().empty()),
connected(false),
+ sendUpdateFrameworkOnConnect(false),
running(true),
detector(_detector),
flags(_flags),
@@ -748,6 +749,11 @@ protected:
connected = true;
failover = false;
+if (sendUpdateFrameworkOnConnect) {
+ sendUpdateFramework();
+}
+sendUpdateFrameworkOnConnect = false;
+
Stopwatch stopwatch;
if (FLAGS_v >= 1) {
stopwatch.start();
@@ -790,6 +796,12 @@ protected:
connected = true;
failover = false;
+if (sendUpdateFrameworkOnConnect) {
+ sendUpdateFramework();
+}
+sendUpdateFrameworkOnConnect = false;
+
+
Stopwatch stopwatch;
if (FLAGS_v >= 1) {
stopwatch.start();
@@ -1583,6 +1595,24 @@ protected:
send(master->pid(), call);
}
+ void updateFramework(const FrameworkInfo& framework_)
+ {
+CHECK(!framework_.has_id());
+
+// Update the FrameworkInfo used for re-registration
+FrameworkID frameworkId = framework.id();
+framework = framework_;
+*framework.mutable_id() = std::move(frameworkId);
+
+if (connected) {
+ sendUpdateFramework();
+} else {
+ VLOG(1) << "Postponing UPDATE_FRAMEWORK call:"
+ " not registered with master";
+ sendUpdateFrameworkOnConnect = true;
+}
+ }
+
private:
friend class mesos::MesosSchedulerDriver;
@@ -1626,6 +1656,23 @@ private:
return static_cast(eventCount());
}
+ void sendUpdateFramework()
+ {
+Call call;
+
+CHECK(framework.has_id());
+*call.mutable_framework_id() = framework.id();
+
+call.set_type(Call::UPDATE_FRAMEWORK);
+*call.mutable_update_framework()->mutable_framework_info() = framework;
+
+VLOG(1) << "Sending UPDATE_FRAMEWORK message";
+
+CHECK_SOME(master);
+send(master->pid(), call);
+ }
+
+
MesosSchedulerDriver* driver;
Scheduler* scheduler;
FrameworkInfo framework;
@@ -1638,6 +1685,10 @@ private:
bool connected; // Flag to indicate if framework is registered.
+ // Flag to indicate that an UPDATE_FRAMEWORK with a current FrameworkInfo
+ // should be sent after successful (re)connection attempt.
+ bool sendUpdateFrameworkOnConnect;
+
// TODO(vinod): Instead of 'bool' use 'Status'.
// We set 'running' to false in SchedulerDriver::stop() and
// SchedulerDriver::abort() to prevent any further messages from
@@ -1686,6 +1737,25 @@ private:
} // namespace mesos {
+void fillMissingFrameworkInfoFields(FrameworkInfo* framework)
+{
+ if (framework->user().empty()) {
+Result user =
[mesos] branch master updated: Fixed compilation error on Mac OS.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new f2bedda Fixed compilation error on Mac OS.
f2bedda is described below
commit f2beddaf04b82c7bf89b4abc6a8f76ab7e41705b
Author: Andrei Budnik
AuthorDate: Fri Jun 7 18:02:01 2019 +0200
Fixed compilation error on Mac OS.
This patch adds missing switch case to fix compilation error introduced
in `bc5a57122635`.
Review: https://reviews.apache.org/r/70811
---
src/slave/containerizer/mesos/launch.cpp | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/slave/containerizer/mesos/launch.cpp
b/src/slave/containerizer/mesos/launch.cpp
index a69a688..b29ec55 100644
--- a/src/slave/containerizer/mesos/launch.cpp
+++ b/src/slave/containerizer/mesos/launch.cpp
@@ -496,8 +496,8 @@ static Try executeFileOperation(const
ContainerFileOperation& op)
return Nothing();
}
-#ifdef __linux__
case ContainerFileOperation::MOUNT: {
+#ifdef __linux__
Try result = mountContainerFilesystem(op.mount());
if (result.isError()) {
return Error(
@@ -506,8 +506,10 @@ static Try executeFileOperation(const
ContainerFileOperation& op)
}
return Nothing();
-}
+#else
+ return Error("Container mount is not supported on non-Linux systems");
#endif // __linux__
+}
case ContainerFileOperation::RENAME: {
Try result =
[mesos] branch master updated: Updated Jenkinsfile for packaging again to activate logrotate.
This is an automated email from the ASF dual-hosted git repository.
tillt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 9ee6bdd Updated Jenkinsfile for packaging again to activate logrotate.
9ee6bdd is described below
commit 9ee6bddc8b28eeef3d4cedda959aa1cedf9fbd47
Author: Till Toenshoff
AuthorDate: Tue Jun 11 15:34:53 2019 +0200
Updated Jenkinsfile for packaging again to activate logrotate.
---
support/jenkins/Jenkinsfile-packaging-centos | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/support/jenkins/Jenkinsfile-packaging-centos
b/support/jenkins/Jenkinsfile-packaging-centos
index f468b24..aa8f37f 100644
--- a/support/jenkins/Jenkinsfile-packaging-centos
+++ b/support/jenkins/Jenkinsfile-packaging-centos
@@ -1,8 +1,6 @@
#!groovy
-options {
- buildDiscarder(logRotator(daysToKeepStr: '30'))
-}
+properties [[$class: 'BuildDiscarderProperty', strategy: [$class:
'LogRotator', daysToKeepStr: '30']], [$class: 'ScannerJobProperty', doNotScan:
false]]
if (env.NODE_LABELS == null) {
env.NODE_LABELS = "(ubuntu)&&(!ubuntu-us1)&&(!ubuntu-eu2)&&(!qnode3)&&(!H23)"
[mesos] branch master updated: Added logrotate option to force Jenkins into removing older artefacts.
This is an automated email from the ASF dual-hosted git repository.
tillt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 47cd9be Added logrotate option to force Jenkins into removing older
artefacts.
47cd9be is described below
commit 47cd9be3187a73a62f3ff659699facb92a6e7123
Author: Till Toenshoff
AuthorDate: Tue Jun 11 14:36:21 2019 +0200
Added logrotate option to force Jenkins into removing older artefacts.
Review: https://reviews.apache.org/r/70829/
---
support/jenkins/Jenkinsfile-packaging-centos | 4
1 file changed, 4 insertions(+)
diff --git a/support/jenkins/Jenkinsfile-packaging-centos
b/support/jenkins/Jenkinsfile-packaging-centos
index de688f4..f468b24 100644
--- a/support/jenkins/Jenkinsfile-packaging-centos
+++ b/support/jenkins/Jenkinsfile-packaging-centos
@@ -1,5 +1,9 @@
#!groovy
+options {
+ buildDiscarder(logRotator(daysToKeepStr: '30'))
+}
+
if (env.NODE_LABELS == null) {
env.NODE_LABELS = "(ubuntu)&&(!ubuntu-us1)&&(!ubuntu-eu2)&&(!qnode3)&&(!H23)"
}
