[mesos] branch master updated: Fixed the flaky ExamplesTest.DynamicReservationFramework.
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 6bc1c80 Fixed the flaky ExamplesTest.DynamicReservationFramework.
6bc1c80 is described below
commit 6bc1c809dfcbcb62ca09e09ea9acb628e7a36738
Author: Benjamin Mahler
AuthorDate: Fri Apr 19 16:14:03 2019 -0400
Fixed the flaky ExamplesTest.DynamicReservationFramework.
The test failed in MESOS-5804 due to the following race:
1. Framework launches task T, moves from RESERVED to
TASK_RUNNING state.
2. Allocation cycle triggers and will send the unreserved
resources to the framework.
3. Before the offer gets to the framework, task T finishes and
framework moves from TASK_RUNNING to RESERVED.
4. In the RESERVED state, the framework expects the reservation
in the offer. But, it's coming in a later offer, and the one
that arrives is for the unreserved resources since it was
generated while the task was still running.
Tne fix applied here for this specific race is to use a 2 week
filter rather than a 0 second filter. That would ensure that the
unreserved resources do not get re-offered to the framework on
their own. However, this fix does not work until MESOS-9616 is
resolved.
Review: https://reviews.apache.org/r/70508
---
src/examples/dynamic_reservation_framework.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/examples/dynamic_reservation_framework.cpp
b/src/examples/dynamic_reservation_framework.cpp
index f9c7dfe..094d805 100644
--- a/src/examples/dynamic_reservation_framework.cpp
+++ b/src/examples/dynamic_reservation_framework.cpp
@@ -115,7 +115,7 @@ public:
const State state = states[offer.slave_id()];
Filters filters;
- filters.set_refuse_seconds(0);
+ filters.set_refuse_seconds(Weeks(2).secs());
switch (state) {
case State::INIT: {
[mesos] 02/02: Added MESOS-9616 to the 1.5.4 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch 1.5.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 13b2888ead47cc7efbdd20c267089e673546a380 Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:41:32 2019 -0700 Added MESOS-9616 to the 1.5.4 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index fd85213..c38fa85 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ Release Notes - Mesos - Version 1.5.4 (WIP) ** Bug * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9619] - Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer * [MESOS-9707] - Calling link::lo() may cause runtime error
[mesos] 01/02: Do not implicitly decline speculatively converted resources.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 6c0d92b5826205481be133e8f054f184b2cbb4cc
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 22 15:46:36 2019 -0700
Do not implicitly decline speculatively converted resources.
Currently if a framework accepts an offer with a `RESERVE` operation
without a task consuming the reserved resources, the resources will be
implicitly declined. This is counter to what one would expect (that only
the remaining resources in the offer will be declined):
Offer `cpus:10` -> `ACCEPT` with `RESERVE cpus(role):1`
*Actual* implicit decline: `cpus:9;cpus(role):1`
*Expected* implicit decline: `cpus:9`
The same issue is present with other transformational operations (i.e.,
`UNRESERVE`, `CREATE` and `DESTROY`).
This patch fixes this issue by only implicitly declining the "remaining"
untransformed resources, computed as follows:
Offered = `cpus:10`
Remaining = `cpus:9;cpus(role):1`
Implicitly declined = remaining - (remaining - offered)
= `cpus:9;cpus(role):1` - `cpus:(role):1`
= `cpus:9`
Review: https://reviews.apache.org/r/70132
---
src/master/master.cpp | 37 +++--
1 file changed, 31 insertions(+), 6 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 995ff55..42f88b6 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -5456,13 +5456,38 @@ void Master::_accept(
conversions);
}
- if (!_offeredResources.empty()) {
-// Tell the allocator about the unused (e.g., refused) resources.
+ // We now need to compute the amounts of remaining (1) speculatively
converted
+ // resources to recover without a filter and (2) resources that are
implicitly
+ // declined with the filter:
+ //
+ // Speculatively converted resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources
+ //
+ // (The last equality holds because resource subtraction yields no
negatives.)
+ //
+ // Implicitly declined resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - speculatively converted resources
+ // = `_offeredResources` - speculatively converted resources
+ Resources speculativelyConverted = _offeredResources - offeredResources;
+ Resources implicitlyDeclined = _offeredResources - speculativelyConverted;
+
+ // Tell the allocator about the net speculatively converted resources. These
+ // resources should not be implicitly declined.
+ if (!speculativelyConverted.empty()) {
allocator->recoverResources(
-frameworkId,
-slaveId,
-_offeredResources,
-accept.filters());
+frameworkId, slaveId, speculativelyConverted, None());
+ }
+
+ // Tell the allocator about the implicitly declined resources.
+ if (!implicitlyDeclined.empty()) {
+allocator->recoverResources(
+frameworkId, slaveId, implicitlyDeclined, accept.filters());
}
}
[mesos] branch master updated: Added MESOS-9616 to the 1.5.4 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git The following commit(s) were added to refs/heads/master by this push: new 3a0c2fa Added MESOS-9616 to the 1.5.4 CHANGELOG. 3a0c2fa is described below commit 3a0c2fa2ae338eeab292e7c0d3dae55b66f5886d Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:41:32 2019 -0700 Added MESOS-9616 to the 1.5.4 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index f2bf363..2ee079b 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1352,6 +1352,7 @@ Release Notes - Mesos - Version 1.5.4 (WIP) ** Bug * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9619] - Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer * [MESOS-9707] - Calling link::lo() may cause runtime error
[mesos] 01/02: Do not implicitly decline speculatively converted resources.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 899fac19fd0082ce96a0c15bf00ac9d9d0453932
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 22 15:46:36 2019 -0700
Do not implicitly decline speculatively converted resources.
Currently if a framework accepts an offer with a `RESERVE` operation
without a task consuming the reserved resources, the resources will be
implicitly declined. This is counter to what one would expect (that only
the remaining resources in the offer will be declined):
Offer `cpus:10` -> `ACCEPT` with `RESERVE cpus(role):1`
*Actual* implicit decline: `cpus:9;cpus(role):1`
*Expected* implicit decline: `cpus:9`
The same issue is present with other transformational operations (i.e.,
`UNRESERVE`, `CREATE` and `DESTROY`).
This patch fixes this issue by only implicitly declining the "remaining"
untransformed resources, computed as follows:
Offered = `cpus:10`
Remaining = `cpus:9;cpus(role):1`
Implicitly declined = remaining - (remaining - offered)
= `cpus:9;cpus(role):1` - `cpus:(role):1`
= `cpus:9`
Review: https://reviews.apache.org/r/70132
---
src/master/master.cpp | 44
1 file changed, 36 insertions(+), 8 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 28a1593..66e8e92 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -5785,16 +5785,44 @@ void Master::_accept(
conversions);
}
+ // We now need to compute the amounts of remaining (1) speculatively
converted
+ // resources to recover without a filter and (2) resources that are
implicitly
+ // declined with the filter:
+ //
+ // Speculatively converted resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources
+ //
+ // (The last equality holds because resource subtraction yields no
negatives.)
+ //
+ // Implicitly declined resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - speculatively converted resources
+ // = `_offeredResources` - speculatively converted resources
+ //
+ // TODO(zhitao): Right now `GROW_VOLUME` and `SHRINK_VOLUME` are implemented
+ // as speculative operations. Since the plan is to make them non-speculative
+ // in the future, their results are not in `_offeredResources`, so we add
them
+ // back here. Remove this once the operations become non-speculative.
+ Resources speculativelyConverted =
+_offeredResources + resizedResources - offeredResources;
+ Resources implicitlyDeclined = _offeredResources - speculativelyConverted;
+
+ // Tell the allocator about the net speculatively converted resources. These
+ // resources should not be implicitly declined.
+ if (!speculativelyConverted.empty()) {
+allocator->recoverResources(
+frameworkId, slaveId, speculativelyConverted, None());
+ }
- // TODO(zhitao): Remove `resizedResources` once `GROW_VOLUME` and
- // `SHRINK_VOLUME` become non-speculative.
- if (!_offeredResources.empty() || !resizedResources.empty()) {
-// Tell the allocator about the unused (e.g., refused) resources.
+ // Tell the allocator about the implicitly declined resources.
+ if (!implicitlyDeclined.empty()) {
allocator->recoverResources(
-frameworkId,
-slaveId,
-_offeredResources + resizedResources,
-accept.filters());
+frameworkId, slaveId, implicitlyDeclined, accept.filters());
}
}
[mesos] branch 1.6.x updated (13fdaa4 -> d8e3909)
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a change to branch 1.6.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 13fdaa4 Added MESOS-9695 to the 1.6.3 CHANGELOG. new 899fac1 Do not implicitly decline speculatively converted resources. new d8e3909 Added MESOS-9616 to the 1.6.3 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/master/master.cpp | 44 2 files changed, 37 insertions(+), 8 deletions(-)
[mesos] 02/02: Added MESOS-9616 to the 1.6.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch 1.6.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit d8e39097a92ac304ab241d0c51e9a65ff3fdba0e Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:40:52 2019 -0700 Added MESOS-9616 to the 1.6.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 55b74d1..3f84ffb 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -7,6 +7,7 @@ Release Notes - Mesos - Version 1.6.3 (WIP) * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. * [MESOS-9536] - Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`. * [MESOS-9564] - Logrotate container logger lets tasks execute arbitrary commands in the Mesos agent's namespace. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9619] - Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9692] - Quota may be under allocated for disk resources. * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer
[mesos] branch master updated: Added MESOS-9616 to the 1.6.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git The following commit(s) were added to refs/heads/master by this push: new 4fff6dc Added MESOS-9616 to the 1.6.3 CHANGELOG. 4fff6dc is described below commit 4fff6dc1e1ad858ec872d7694e529ab85c514ba5 Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:40:52 2019 -0700 Added MESOS-9616 to the 1.6.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 366fe81..f2bf363 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -886,6 +886,7 @@ Release Notes - Mesos - Version 1.6.3 (WIP) * [MESOS-9529] - `/proc` should be remounted even if a nested container set `share_pid_namespace` to true. * [MESOS-9536] - Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`. * [MESOS-9564] - Logrotate container logger lets tasks execute arbitrary commands in the Mesos agent's namespace. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9619] - Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9692] - Quota may be under allocated for disk resources. * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer
[mesos] 02/02: Added MESOS-9616 to the 1.7.3 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch 1.7.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit b82ad53dfcd222619ce1ab53caa81201facd59ec Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:40:30 2019 -0700 Added MESOS-9616 to the 1.7.3 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 369a2c8..5767b1f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -12,6 +12,7 @@ Release Notes - Mesos - Version 1.7.3 (WIP) * [MESOS-9568] - SLRP does not clean up mount directories for destroyed MOUNT disks. * [MESOS-9607] - Removing a resource provider with consumers breaks resource publishing. * [MESOS-9610] - Fetcher vulnerability - escaping from sandbox. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9619] - Mesos Master Crashes with Launch Group when using Port Resources * [MESOS-9661] - Agent crashes when SLRP recovers dropped operations. * [MESOS-9692] - Quota may be under allocated for disk resources.
[mesos] 01/02: Do not implicitly decline speculatively converted resources.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit f05ac51eafd35d3ec0fb8a6492d0a0bb2622e375
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 22 15:46:36 2019 -0700
Do not implicitly decline speculatively converted resources.
Currently if a framework accepts an offer with a `RESERVE` operation
without a task consuming the reserved resources, the resources will be
implicitly declined. This is counter to what one would expect (that only
the remaining resources in the offer will be declined):
Offer `cpus:10` -> `ACCEPT` with `RESERVE cpus(role):1`
*Actual* implicit decline: `cpus:9;cpus(role):1`
*Expected* implicit decline: `cpus:9`
The same issue is present with other transformational operations (i.e.,
`UNRESERVE`, `CREATE` and `DESTROY`).
This patch fixes this issue by only implicitly declining the "remaining"
untransformed resources, computed as follows:
Offered = `cpus:10`
Remaining = `cpus:9;cpus(role):1`
Implicitly declined = remaining - (remaining - offered)
= `cpus:9;cpus(role):1` - `cpus:(role):1`
= `cpus:9`
Review: https://reviews.apache.org/r/70132
---
src/master/master.cpp | 44
1 file changed, 36 insertions(+), 8 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index ed072d3..479d56c 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -5866,16 +5866,44 @@ void Master::_accept(
conversions);
}
+ // We now need to compute the amounts of remaining (1) speculatively
converted
+ // resources to recover without a filter and (2) resources that are
implicitly
+ // declined with the filter:
+ //
+ // Speculatively converted resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources
+ //
+ // (The last equality holds because resource subtraction yields no
negatives.)
+ //
+ // Implicitly declined resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - speculatively converted resources
+ // = `_offeredResources` - speculatively converted resources
+ //
+ // TODO(zhitao): Right now `GROW_VOLUME` and `SHRINK_VOLUME` are implemented
+ // as speculative operations. Since the plan is to make them non-speculative
+ // in the future, their results are not in `_offeredResources`, so we add
them
+ // back here. Remove this once the operations become non-speculative.
+ Resources speculativelyConverted =
+_offeredResources + resizedResources - offeredResources;
+ Resources implicitlyDeclined = _offeredResources - speculativelyConverted;
+
+ // Tell the allocator about the net speculatively converted resources. These
+ // resources should not be implicitly declined.
+ if (!speculativelyConverted.empty()) {
+allocator->recoverResources(
+frameworkId, slaveId, speculativelyConverted, None());
+ }
- // TODO(zhitao): Remove `resizedResources` once `GROW_VOLUME` and
- // `SHRINK_VOLUME` become non-speculative.
- if (!_offeredResources.empty() || !resizedResources.empty()) {
-// Tell the allocator about the unused (e.g., refused) resources.
+ // Tell the allocator about the implicitly declined resources.
+ if (!implicitlyDeclined.empty()) {
allocator->recoverResources(
-frameworkId,
-slaveId,
-_offeredResources + resizedResources,
-accept.filters());
+frameworkId, slaveId, implicitlyDeclined, accept.filters());
}
}
[mesos] branch 1.7.x updated (80c9fd7 -> b82ad53)
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a change to branch 1.7.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 80c9fd7 Added MESOS-9695 to the 1.7.3 CHANGELOG. new f05ac51 Do not implicitly decline speculatively converted resources. new b82ad53 Added MESOS-9616 to the 1.7.3 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/master/master.cpp | 44 2 files changed, 37 insertions(+), 8 deletions(-)
[mesos] 01/02: Do not implicitly decline speculatively converted resources.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 7e43f1fcfab8983f102ba79419a81f62fba677d8
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 22 15:46:36 2019 -0700
Do not implicitly decline speculatively converted resources.
Currently if a framework accepts an offer with a `RESERVE` operation
without a task consuming the reserved resources, the resources will be
implicitly declined. This is counter to what one would expect (that only
the remaining resources in the offer will be declined):
Offer `cpus:10` -> `ACCEPT` with `RESERVE cpus(role):1`
*Actual* implicit decline: `cpus:9;cpus(role):1`
*Expected* implicit decline: `cpus:9`
The same issue is present with other transformational operations (i.e.,
`UNRESERVE`, `CREATE` and `DESTROY`).
This patch fixes this issue by only implicitly declining the "remaining"
untransformed resources, computed as follows:
Offered = `cpus:10`
Remaining = `cpus:9;cpus(role):1`
Implicitly declined = remaining - (remaining - offered)
= `cpus:9;cpus(role):1` - `cpus:(role):1`
= `cpus:9`
Review: https://reviews.apache.org/r/70132
---
src/master/master.cpp | 49 +++
src/tests/slave_tests.cpp | 3 ++-
2 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index ad54ae2..555136e 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -5959,17 +5959,50 @@ void Master::_accept(
conversions);
}
+ // We now need to compute the amounts of remaining (1) speculatively
converted
+ // resources to recover without a filter and (2) resources that are
implicitly
+ // declined with the filter:
+ //
+ // Speculatively converted resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources
+ //
+ // (The last equality holds because resource subtraction yields no
negatives.)
+ //
+ // Implicitly declined resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - speculatively converted resources
+ // = `_offeredResources` - speculatively converted resources
+ //
+ // TODO(zhitao): Right now `GROW_VOLUME` and `SHRINK_VOLUME` are implemented
+ // as speculative operations. Since the plan is to make them non-speculative
+ // in the future, their results are not in `_offeredResources`, so we add
them
+ // back here. Remove this once the operations become non-speculative.
+ Resources speculativelyConverted =
+_offeredResources + resizedResources - offeredResources;
+ Resources implicitlyDeclined = _offeredResources - speculativelyConverted;
+
+ // Prevent any allocations from occurring during resource recovery below.
+ allocator->pause();
- // TODO(zhitao): Remove `resizedResources` once `GROW_VOLUME` and
- // `SHRINK_VOLUME` become non-speculative.
- if (!_offeredResources.empty() || !resizedResources.empty()) {
-// Tell the allocator about the unused (e.g., refused) resources.
+ // Tell the allocator about the net speculatively converted resources. These
+ // resources should not be implicitly declined.
+ if (!speculativelyConverted.empty()) {
allocator->recoverResources(
-frameworkId,
-slaveId,
-_offeredResources + resizedResources,
-accept.filters());
+frameworkId, slaveId, speculativelyConverted, None());
+ }
+
+ // Tell the allocator about the implicitly declined resources.
+ if (!implicitlyDeclined.empty()) {
+allocator->recoverResources(
+frameworkId, slaveId, implicitlyDeclined, accept.filters());
}
+
+ allocator->resume();
}
diff --git a/src/tests/slave_tests.cpp b/src/tests/slave_tests.cpp
index 019dbd7..50882a5 100644
--- a/src/tests/slave_tests.cpp
+++ b/src/tests/slave_tests.cpp
@@ -6495,7 +6495,8 @@ TEST_F(SlaveTest, UpdateOperationStatusRetry)
Future offers;
EXPECT_CALL(*scheduler, offers(_, _))
-.WillOnce(FutureArg<1>());
+.WillOnce(FutureArg<1>())
+.WillRepeatedly(Return()); // Ignore subsequent offers.
ContentType contentType = ContentType::PROTOBUF;
[mesos] branch 1.8.x updated (a684f07 -> 400f8e7)
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a change to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from a684f07 Allowed compiling Seccomp isolator on older kernel versions. new 7e43f1f Do not implicitly decline speculatively converted resources. new 400f8e7 Added MESOS-9616 to the 1.8.1 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/master/master.cpp | 49 +++ src/tests/slave_tests.cpp | 3 ++- 3 files changed, 44 insertions(+), 9 deletions(-)
[mesos] 02/02: Added MESOS-9616 to the 1.8.1 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch 1.8.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit 400f8e716d07d2f9d54000dabd37186b07667796 Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:39:50 2019 -0700 Added MESOS-9616 to the 1.8.1 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index c99523c..ed2862c 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ Release Notes - Mesos - Version 1.8.1 (WIP) ** Bug * [MESOS-9536] - Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer Release Notes - Mesos - Version 1.8.0
[mesos] branch master updated (fc8847d -> 8d28725)
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from fc8847d Added MESOS-9695 to the 1.4.4 CHANGELOG. new de7c969 Do not implicitly decline speculatively converted resources. new 8d28725 Added MESOS-9616 to the 1.8.1 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/master/master.cpp | 49 +++ src/tests/slave_tests.cpp | 3 ++- 3 files changed, 44 insertions(+), 9 deletions(-)
[mesos] 01/02: Do not implicitly decline speculatively converted resources.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit de7c969d061d0d6a815cc38dc59bcb4e8dfc2ff7
Author: Chun-Hung Hsiao
AuthorDate: Mon Apr 22 15:46:36 2019 -0700
Do not implicitly decline speculatively converted resources.
Currently if a framework accepts an offer with a `RESERVE` operation
without a task consuming the reserved resources, the resources will be
implicitly declined. This is counter to what one would expect (that only
the remaining resources in the offer will be declined):
Offer `cpus:10` -> `ACCEPT` with `RESERVE cpus(role):1`
*Actual* implicit decline: `cpus:9;cpus(role):1`
*Expected* implicit decline: `cpus:9`
The same issue is present with other transformational operations (i.e.,
`UNRESERVE`, `CREATE` and `DESTROY`).
This patch fixes this issue by only implicitly declining the "remaining"
untransformed resources, computed as follows:
Offered = `cpus:10`
Remaining = `cpus:9;cpus(role):1`
Implicitly declined = remaining - (remaining - offered)
= `cpus:9;cpus(role):1` - `cpus:(role):1`
= `cpus:9`
Review: https://reviews.apache.org/r/70132
---
src/master/master.cpp | 49 +++
src/tests/slave_tests.cpp | 3 ++-
2 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 9f0a976..a8ee629 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -5938,17 +5938,50 @@ void Master::_accept(
conversions);
}
+ // We now need to compute the amounts of remaining (1) speculatively
converted
+ // resources to recover without a filter and (2) resources that are
implicitly
+ // declined with the filter:
+ //
+ // Speculatively converted resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources not consumed by any operation
+ // = `_offeredResources` - offered resources
+ //
+ // (The last equality holds because resource subtraction yields no
negatives.)
+ //
+ // Implicitly declined resources
+ // = (offered resources).apply(speculative operations)
+ // - resources consumed by non-speculative operations
+ // - speculatively converted resources
+ // = `_offeredResources` - speculatively converted resources
+ //
+ // TODO(zhitao): Right now `GROW_VOLUME` and `SHRINK_VOLUME` are implemented
+ // as speculative operations. Since the plan is to make them non-speculative
+ // in the future, their results are not in `_offeredResources`, so we add
them
+ // back here. Remove this once the operations become non-speculative.
+ Resources speculativelyConverted =
+_offeredResources + resizedResources - offeredResources;
+ Resources implicitlyDeclined = _offeredResources - speculativelyConverted;
+
+ // Prevent any allocations from occurring during resource recovery below.
+ allocator->pause();
- // TODO(zhitao): Remove `resizedResources` once `GROW_VOLUME` and
- // `SHRINK_VOLUME` become non-speculative.
- if (!_offeredResources.empty() || !resizedResources.empty()) {
-// Tell the allocator about the unused (e.g., refused) resources.
+ // Tell the allocator about the net speculatively converted resources. These
+ // resources should not be implicitly declined.
+ if (!speculativelyConverted.empty()) {
allocator->recoverResources(
-frameworkId,
-slaveId,
-_offeredResources + resizedResources,
-accept.filters());
+frameworkId, slaveId, speculativelyConverted, None());
+ }
+
+ // Tell the allocator about the implicitly declined resources.
+ if (!implicitlyDeclined.empty()) {
+allocator->recoverResources(
+frameworkId, slaveId, implicitlyDeclined, accept.filters());
}
+
+ allocator->resume();
}
diff --git a/src/tests/slave_tests.cpp b/src/tests/slave_tests.cpp
index 019dbd7..50882a5 100644
--- a/src/tests/slave_tests.cpp
+++ b/src/tests/slave_tests.cpp
@@ -6495,7 +6495,8 @@ TEST_F(SlaveTest, UpdateOperationStatusRetry)
Future offers;
EXPECT_CALL(*scheduler, offers(_, _))
-.WillOnce(FutureArg<1>());
+.WillOnce(FutureArg<1>())
+.WillRepeatedly(Return()); // Ignore subsequent offers.
ContentType contentType = ContentType::PROTOBUF;
[mesos] 02/02: Added MESOS-9616 to the 1.8.1 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 8d28725b7035308acdb9807d9c967d3c2bd34777 Author: Chun-Hung Hsiao AuthorDate: Wed May 1 13:39:50 2019 -0700 Added MESOS-9616 to the 1.8.1 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index f3f6f92..56f3ef6 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ Release Notes - Mesos - Version 1.8.1 (WIP) ** Bug * [MESOS-9536] - Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`. + * [MESOS-9616] - `Filters.refuse_seconds` declines resources not in offers. * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer Release Notes - Mesos - Version 1.8.0
[mesos] branch master updated: Added MESOS-9695 to the 1.4.4 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git The following commit(s) were added to refs/heads/master by this push: new fc8847d Added MESOS-9695 to the 1.4.4 CHANGELOG. fc8847d is described below commit fc8847d0002a6c937e59341aace12e739102449d Author: Andrei Budnik AuthorDate: Wed May 1 12:25:30 2019 +0200 Added MESOS-9695 to the 1.4.4 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 1a4d782..f3f6f92 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1979,6 +1979,7 @@ Release Notes - Mesos - Version 1.4.4 (WIP) ** Bug * [MESOS-9507] - Agent could not recover due to empty docker volume checkpointed files. + * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer ** Improvement: * [MESOS-9159] - Support Foreign URLs in docker registry puller.
[mesos] branch 1.4.x updated (1a76202 -> f05058d)
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a change to branch 1.4.x in repository https://gitbox.apache.org/repos/asf/mesos.git. from 1a76202 Added MESOS-9159 and MESOS-9675 to the 1.4.4 CHANGELOG. new ca65f99 Removed the duplicate pid check in Docker containerizer. new f05058d Added MESOS-9695 to the 1.4.4 CHANGELOG. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG | 1 + src/slave/containerizer/docker.cpp | 27 ++- 2 files changed, 7 insertions(+), 21 deletions(-)
[mesos] 01/02: Removed the duplicate pid check in Docker containerizer.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch 1.4.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit ca65f991727494b9b78e64a19306231ac004289f
Author: Qian Zhang
AuthorDate: Tue Apr 30 13:23:26 2019 +0200
Removed the duplicate pid check in Docker containerizer.
Review: https://reviews.apache.org/r/70561/
---
src/slave/containerizer/docker.cpp | 27 ++-
1 file changed, 6 insertions(+), 21 deletions(-)
diff --git a/src/slave/containerizer/docker.cpp
b/src/slave/containerizer/docker.cpp
index 31a47f0..97cd75e 100644
--- a/src/slave/containerizer/docker.cpp
+++ b/src/slave/containerizer/docker.cpp
@@ -925,10 +925,6 @@ Future DockerContainerizerProcess::_recover(
}
}
-// Collection of pids that we've started reaping in order to
-// detect very unlikely duplicate scenario (see below).
-hashmap pids;
-
foreachvalue (const FrameworkState& framework, state->frameworks) {
foreachvalue (const ExecutorState& executor, framework.executors) {
if (executor.info.isNone()) {
@@ -1007,9 +1003,12 @@ Future DockerContainerizerProcess::_recover(
// Only reap the executor process if the executor can be connected
// otherwise just set `container->status` to `None()`. This is to
-// avoid reaping an irrelevant process, e.g., after the agent host is
-// rebooted, the executor pid happens to be reused by another process.
-// See MESOS-8125 for details.
+// avoid reaping an irrelevant process, e.g., agent process is stopped
+// for a long time, and during this time executor terminates and its
+// pid happens to be reused by another irrelevant process. When agent
+// is restarted, it still considers this executor not complete (i.e.,
+// `run->completed` is false), so we would reap the irrelevant process
+// if we do not check whether that process can be connected.
// Note that if both the pid and the port of the executor are reused
// by another process or two processes respectively after the agent
// host reboots we will still reap an irrelevant process, but that
@@ -1045,20 +1044,6 @@ Future DockerContainerizerProcess::_recover(
container->status.future().get()
.onAny(defer(self(), ::reaped, containerId));
-if (pids.containsValue(pid)) {
- // This should (almost) never occur. There is the
- // possibility that a new executor is launched with the same
- // pid as one that just exited (highly unlikely) and the
- // slave dies after the new executor is launched but before
- // it hears about the termination of the earlier executor
- // (also unlikely).
- return Failure(
- "Detected duplicate pid " + stringify(pid) +
- " for container " + stringify(containerId));
-}
-
-pids.put(containerId, pid);
-
const string sandboxDirectory = paths::getExecutorRunPath(
flags.work_dir,
state->id,
[mesos] 02/02: Added MESOS-9695 to the 1.4.4 CHANGELOG.
This is an automated email from the ASF dual-hosted git repository. abudnik pushed a commit to branch 1.4.x in repository https://gitbox.apache.org/repos/asf/mesos.git commit f05058dad7219950fd9bdc2748ab9c9a79d6e7f1 Author: Andrei Budnik AuthorDate: Wed May 1 12:25:30 2019 +0200 Added MESOS-9695 to the 1.4.4 CHANGELOG. --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 0f32baa..0ce1715 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ Release Notes - Mesos - Version 1.4.4 (WIP) ** Bug * [MESOS-9507] - Agent could not recover due to empty docker volume checkpointed files. + * [MESOS-9695] - Remove the duplicate pid check in Docker containerizer ** Improvement: * [MESOS-9159] - Support Foreign URLs in docker registry puller.
[mesos] branch 1.8.x updated: Allowed compiling Seccomp isolator on older kernel versions.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch 1.8.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/1.8.x by this push:
new a684f07 Allowed compiling Seccomp isolator on older kernel versions.
a684f07 is described below
commit a684f073db683d83965fccbc7e5a308e35a7da9c
Author: Andrei Budnik
AuthorDate: Wed May 1 11:52:49 2019 +0200
Allowed compiling Seccomp isolator on older kernel versions.
This patch removes dependency on `linux/seccomp.h` header, which
may be missing on some Linux distributions.
Review: https://reviews.apache.org/r/70567/
---
configure.ac | 8
src/slave/containerizer/mesos/isolators/linux/seccomp.cpp | 10 --
2 files changed, 8 insertions(+), 10 deletions(-)
diff --git a/configure.ac b/configure.ac
index a367c28..8f3a36e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1627,14 +1627,6 @@ The Seccomp isolator is only supported on Linux.
---
])])
- AC_CHECK_HEADERS([linux/seccomp.h], [],
- [AC_MSG_ERROR([Cannot find seccomp system headers
-Please install the Linux kernel headers and make sure that you have
-Linux kernel 3.5+ installed.
- ])])
-
# Check if libseccomp prefix path was supplied and if so, add it to
# CPPFLAGS while extending it by /include and to LDFLAGS while
# extending it by /lib.
diff --git a/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
b/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
index 5624c24..bb35c6e 100644
--- a/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
+++ b/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
@@ -14,8 +14,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-#include
-
#include
#include
@@ -34,6 +32,14 @@ using mesos::slave::ContainerConfig;
using mesos::slave::ContainerLaunchInfo;
using mesos::slave::Isolator;
+// NOTE: The definition below was taken from the Linux Kernel sources.
+//
+// TODO(abudnik): This definition should be removed in favor of using
+// `linux/seccomp.h` once we drop support for kernels older than 3.5.
+#if !defined(SECCOMP_MODE_FILTER)
+#define SECCOMP_MODE_FILTER 2
+#endif
+
namespace mesos {
namespace internal {
namespace slave {
[mesos] branch master updated: Allowed compiling Seccomp isolator on older kernel versions.
This is an automated email from the ASF dual-hosted git repository.
abudnik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 1954bea Allowed compiling Seccomp isolator on older kernel versions.
1954bea is described below
commit 1954beae2133fa55efe53038a020de4309b7ae91
Author: Andrei Budnik
AuthorDate: Wed May 1 11:52:49 2019 +0200
Allowed compiling Seccomp isolator on older kernel versions.
This patch removes dependency on `linux/seccomp.h` header, which
may be missing on some Linux distributions.
Review: https://reviews.apache.org/r/70567/
---
configure.ac | 8
src/slave/containerizer/mesos/isolators/linux/seccomp.cpp | 10 --
2 files changed, 8 insertions(+), 10 deletions(-)
diff --git a/configure.ac b/configure.ac
index e5afde4..b4bad57 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1627,14 +1627,6 @@ The Seccomp isolator is only supported on Linux.
---
])])
- AC_CHECK_HEADERS([linux/seccomp.h], [],
- [AC_MSG_ERROR([Cannot find seccomp system headers
-Please install the Linux kernel headers and make sure that you have
-Linux kernel 3.5+ installed.
- ])])
-
# Check if libseccomp prefix path was supplied and if so, add it to
# CPPFLAGS while extending it by /include and to LDFLAGS while
# extending it by /lib.
diff --git a/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
b/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
index 5624c24..bb35c6e 100644
--- a/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
+++ b/src/slave/containerizer/mesos/isolators/linux/seccomp.cpp
@@ -14,8 +14,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-#include
-
#include
#include
@@ -34,6 +32,14 @@ using mesos::slave::ContainerConfig;
using mesos::slave::ContainerLaunchInfo;
using mesos::slave::Isolator;
+// NOTE: The definition below was taken from the Linux Kernel sources.
+//
+// TODO(abudnik): This definition should be removed in favor of using
+// `linux/seccomp.h` once we drop support for kernels older than 3.5.
+#if !defined(SECCOMP_MODE_FILTER)
+#define SECCOMP_MODE_FILTER 2
+#endif
+
namespace mesos {
namespace internal {
namespace slave {
