mesos git commit: Added to MESOS-2537 to CHANGELOG for 1.0.4.
Repository: mesos Updated Branches: refs/heads/master ff9ed0c83 -> 3ded707ca Added to MESOS-2537 to CHANGELOG for 1.0.4. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/3ded707c Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/3ded707c Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/3ded707c Branch: refs/heads/master Commit: 3ded707cab2c1037fd1a699b075895feceb3ae4a Parents: ff9ed0c Author: Kapil Arya Authored: Mon Apr 17 17:26:17 2017 -0400 Committer: Kapil Arya Committed: Mon Apr 17 17:26:17 2017 -0400 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/3ded707c/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index a0fca05..8cd6635 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -930,6 +930,7 @@ Release Notes - Mesos - Version 1.0.4 (WIP) All Issues: ** Bug +* [MESOS-2537] - AC_ARG_ENABLED checks are broken * [MESOS-6606] - Reject optimized builds with libcxx before 3.9 * [MESOS-7008] - Quota not recovered from registry in empty cluster. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content.
mesos git commit: Added to MESOS-2537 to CHANGELOG for 1.0.4.
Repository: mesos Updated Branches: refs/heads/1.0.x 18b6245ed -> 84118f177 Added to MESOS-2537 to CHANGELOG for 1.0.4. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/84118f17 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/84118f17 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/84118f17 Branch: refs/heads/1.0.x Commit: 84118f17701bfd32bd97fa19b4816ba44f5def41 Parents: 18b6245 Author: Kapil Arya Authored: Mon Apr 17 17:26:17 2017 -0400 Committer: Kapil Arya Committed: Mon Apr 17 17:27:18 2017 -0400 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/84118f17/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 6b1bb6d..0ff4a4f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ Release Notes - Mesos - Version 1.0.4 (WIP) All Issues: ** Bug +* [MESOS-2537] - AC_ARG_ENABLED checks are broken * [MESOS-6606] - Reject optimized builds with libcxx before 3.9 * [MESOS-7008] - Quota not recovered from registry in empty cluster. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content.
mesos git commit: Updated Mesos version to 1.0.4.
Repository: mesos
Updated Branches:
refs/heads/1.0.x 84118f177 -> 71e41f166
Updated Mesos version to 1.0.4.
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/71e41f16
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/71e41f16
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/71e41f16
Branch: refs/heads/1.0.x
Commit: 71e41f166f671c988e36c1bf04728ec3589eb509
Parents: 84118f1
Author: Vinod Kone
Authored: Mon Apr 17 15:15:18 2017 -0700
Committer: Vinod Kone
Committed: Mon Apr 17 15:15:18 2017 -0700
--
CMakeLists.txt | 2 +-
configure.ac | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/71e41f16/CMakeLists.txt
--
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 1d64708..e9a33ef 100755
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -21,7 +21,7 @@ cmake_minimum_required(VERSION 2.8)
project(Mesos)
set(MESOS_MAJOR_VERSION 1)
set(MESOS_MINOR_VERSION 0)
-set(MESOS_PATCH_VERSION 3)
+set(MESOS_PATCH_VERSION 4)
set(PACKAGE_VERSION
${MESOS_MAJOR_VERSION}.${MESOS_MINOR_VERSION}.${MESOS_PATCH_VERSION})
http://git-wip-us.apache.org/repos/asf/mesos/blob/71e41f16/configure.ac
--
diff --git a/configure.ac b/configure.ac
index ddbf68b..b1d6468 100644
--- a/configure.ac
+++ b/configure.ac
@@ -18,7 +18,7 @@
# Process this file with autoconf to produce a configure script.
AC_PREREQ([2.61])
-AC_INIT([mesos], [1.0.3])
+AC_INIT([mesos], [1.0.4])
# Have autoconf setup some variables related to the system.
AC_CANONICAL_HOST
[mesos] Git Push Summary
Repository: mesos Updated Tags: refs/tags/1.0.4-rc1 [created] 71e41f166
svn commit: r19183 - in /dev/mesos/1.0.4-rc1: ./ mesos-1.0.4.tar.gz mesos-1.0.4.tar.gz.asc mesos-1.0.4.tar.gz.md5
Author: vinodkone Date: Mon Apr 17 22:42:55 2017 New Revision: 19183 Log: Adding mesos-1.0.4-rc1. Added: dev/mesos/1.0.4-rc1/ dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz (with props) dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.asc dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.md5 Added: dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz == Binary file - no diff available. Propchange: dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz -- svn:mime-type = application/octet-stream Added: dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.asc == --- dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.asc (added) +++ dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.asc Mon Apr 17 22:42:55 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1 + +iQEcBAABAgAGBQJY9UTeAAoJEBf21SiiwsdKr8AH/R3E8cIrolbiHQlng9QIEoGp +JMnz6Bg98ugdWZTdz38g3TSfTSWQpGJ3rYUGtWw4uAjjOUvJKlDcAax0TntMb+Aa +GHmsCH5Ef8T1tww4c9Menphmjt3iVBb9mkkrO/Cq8UwCYrhkYbrkV5cxCwdFVw9J +bq6tIyT240ztCg2DfeWFAxA9mjK3cyEXgbHaQlFNqAgNbrS7ttx0RcJfMEskxWg2 +wX139eY59spOlL/4ojJWoUgxDNjYrksr+s2Ky+FXrsvei5Jwx3Z396arMK0SJFvl +prxQH/kLZF14vQAISejsVYOxE0K+KehGtfsm0b5eHwAJFU7bhsVIlueFrvZALSU= +=Uxx2 +-END PGP SIGNATURE- Added: dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.md5 == --- dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.md5 (added) +++ dev/mesos/1.0.4-rc1/mesos-1.0.4.tar.gz.md5 Mon Apr 17 22:42:55 2017 @@ -0,0 +1 @@ +mesos-1.0.4.tar.gz: 4A 07 04 74 46 DA CF B4 65 96 21 81 3E DC B3 14
mesos git commit: Fixed a regression hiding previously exposed master and agent flags.
Repository: mesos Updated Branches: refs/heads/1.2.x 990e58498 -> 78cf56e9e Fixed a regression hiding previously exposed master and agent flags. In f441eb9 we in a number of places changed how 'Flag's were added to 'Flags' by moving from ad-hoc invocations of 'FlagsBase::add' on particular instances to proper 'Flags' member variables. This was needed to ensure 'Flags' instances could always safely be copied. For that we introduced local derived 'Flags' classes to support localized parsing needs. At the same time, this implementation strategy led to these these local variables not being accessible through instances of the original class anymore (this was inevitable when making 'Flags' classes properly copyable), which e.g., causes a regression in the flags displayed in a master's '/flags' endpoint. This commit moves the flags into the respective base class removing the local classes so that all passed flags are exposed to users. Review: https://reviews.apache.org/r/58214/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/78cf56e9 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/78cf56e9 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/78cf56e9 Branch: refs/heads/1.2.x Commit: 78cf56e9e5d42d9055013251f01c152f6fb9882f Parents: 990e584 Author: Benjamin Bannier Authored: Mon Apr 17 16:38:17 2017 -0700 Committer: Michael Park Committed: Mon Apr 17 16:38:17 2017 -0700 -- src/master/flags.cpp | 35 src/master/flags.hpp | 16 + src/master/main.cpp | 59 +-- src/slave/flags.cpp | 34 +++ src/slave/flags.hpp | 15 src/slave/main.cpp | 59 +-- 6 files changed, 102 insertions(+), 116 deletions(-) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/78cf56e9/src/master/flags.cpp -- diff --git a/src/master/flags.cpp b/src/master/flags.cpp index d25cfdd..3a89304 100644 --- a/src/master/flags.cpp +++ b/src/master/flags.cpp @@ -592,4 +592,39 @@ mesos::internal::master::Flags::Flags() "information about all connected agents. See also the\n" "`registry_max_agent_age` flag.", DEFAULT_REGISTRY_MAX_AGENT_COUNT); + + add(&Flags::ip, + "ip", + "IP address to listen on. This cannot be used in conjunction\n" + "with `--ip_discovery_command`."); + + add(&Flags::port, "port", "Port to listen on.", MasterInfo().port()); + + add(&Flags::advertise_ip, + "advertise_ip", + "IP address advertised to reach this Mesos master.\n" + "The master does not bind using this IP address.\n" + "However, this IP address may be used to access this master."); + + add(&Flags::advertise_port, + "advertise_port", + "Port advertised to reach Mesos master (along with\n" + "`advertise_ip`). The master does not bind to this port.\n" + "However, this port (along with `advertise_ip`) may be used to\n" + "access this master."); + + add(&Flags::zk, + "zk", + "ZooKeeper URL (used for leader election amongst masters)\n" + "May be one of:\n" + " `zk://host1:port1,host2:port2,.../path`\n" + " `zk://username:password@host1:port1,host2:port2,.../path`\n" + " `file:///path/to/file` (where file contains one of the above)\n" + "NOTE: Not required if master is run in standalone mode (non-HA)."); + + add(&Flags::ip_discovery_command, + "ip_discovery_command", + "Optional IP discovery binary: if set, it is expected to emit\n" + "the IP address which the master will try to bind to.\n" + "Cannot be used in conjunction with `--ip`."); } http://git-wip-us.apache.org/repos/asf/mesos/blob/78cf56e9/src/master/flags.hpp -- diff --git a/src/master/flags.hpp b/src/master/flags.hpp index 41a0edf..9336a50 100644 --- a/src/master/flags.hpp +++ b/src/master/flags.hpp @@ -17,6 +17,8 @@ #ifndef __MASTER_FLAGS_HPP__ #define __MASTER_FLAGS_HPP__ +#include + #include #include @@ -94,6 +96,20 @@ public: Duration registry_max_agent_age; size_t registry_max_agent_count; + // The following flags are executable specific (e.g., since we only + // have one instance of libprocess per execution, we only want to + // advertise the IP and port option once, here). + + Option ip; + uint16_t port; + Option advertise_ip; + Option advertise_port; + Option zk; + + // Optional IP discover script that will set the Master IP. + // If set, its output is expected to be a valid parseable IP string. + Option ip_discovery_command; + #ifdef WITH_NETWORK_ISOLATOR Option max_executors_per_agent; #endif // WITH_N
[2/2] mesos git commit: Fixed the image signature check for Nexus Registry.
Fixed the image signature check for Nexus Registry.
Currently, the signature field of the docker v2 image manifest is
not used yet. The check of at least one image signature is too
strict because some registry (e.g., Nexus Registry) does not sign
the image manifest. We should release the signature check for now.
Review: https://reviews.apache.org/r/58479/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/643dafde
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/643dafde
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/643dafde
Branch: refs/heads/master
Commit: 643dafdec76bb176270fe686ec2400242ed0fe36
Parents: 265754f
Author: Gilbert Song
Authored: Tue Apr 18 07:57:30 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 07:57:30 2017 +0800
--
src/docker/spec.cpp | 4
1 file changed, 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/643dafde/src/docker/spec.cpp
--
diff --git a/src/docker/spec.cpp b/src/docker/spec.cpp
index 88029c2..6b5588e 100644
--- a/src/docker/spec.cpp
+++ b/src/docker/spec.cpp
@@ -332,10 +332,6 @@ Option validate(const ImageManifest& manifest)
return Error("'history' field size must be at least one");
}
- if (manifest.signatures_size() <= 0) {
-return Error("'signatures' field size must be at least one");
- }
-
// Verify that blobSum and v1Compatibility numbers are equal.
if (manifest.fslayers_size() != manifest.history_size()) {
return Error("The size of 'fsLayers' should be equal "
[1/2] mesos git commit: Fixed alicloud unit test for limited timeout.
Repository: mesos
Updated Branches:
refs/heads/master 3ded707ca -> 643dafdec
Fixed alicloud unit test for limited timeout.
Alicloud server is in Asia. We parameterized the unit test
`ROOT_INTERNET_CURL_SimpleCommand` to test different registry with
unifed containerizer. Longer `executor_registration_timeout` should
be given to make sure the image download is not impacted by network
legacy.
Review: https://reviews.apache.org/r/58465/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/265754fa
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/265754fa
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/265754fa
Branch: refs/heads/master
Commit: 265754fae99baeffbafaf2383d396754ba6ca3ff
Parents: 3ded707
Author: Gilbert Song
Authored: Tue Apr 18 07:57:23 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 07:57:23 2017 +0800
--
src/tests/containerizer/provisioner_docker_tests.cpp | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/265754fa/src/tests/containerizer/provisioner_docker_tests.cpp
--
diff --git a/src/tests/containerizer/provisioner_docker_tests.cpp
b/src/tests/containerizer/provisioner_docker_tests.cpp
index b0a4d21..d1224d8 100644
--- a/src/tests/containerizer/provisioner_docker_tests.cpp
+++ b/src/tests/containerizer/provisioner_docker_tests.cpp
@@ -16,6 +16,7 @@
#include
+#include
#include
#include
#include
@@ -474,6 +475,10 @@ TEST_P(ProvisionerDockerPullerTest,
ROOT_INTERNET_CURL_SimpleCommand)
flags.isolation = "docker/runtime,filesystem/linux";
flags.image_providers = "docker";
+ // Image pulling time may be long, depending on the location of
+ // the registry server.
+ flags.executor_registration_timeout = Minutes(3);
+
Owned detector = master.get()->createDetector();
Try> slave = StartSlave(detector.get(), flags);
ASSERT_SOME(slave);
@@ -527,7 +532,7 @@ TEST_P(ProvisionerDockerPullerTest,
ROOT_INTERNET_CURL_SimpleCommand)
driver.launchTasks(offer.id(), {task});
- AWAIT_READY_FOR(statusRunning, Seconds(60));
+ AWAIT_READY_FOR(statusRunning, Minutes(3));
EXPECT_EQ(task.task_id(), statusRunning->task_id());
EXPECT_EQ(TASK_RUNNING, statusRunning->state());
mesos git commit: Revert "Allowed whitelist additional devices in cgroups devices subsystem."
Repository: mesos
Updated Branches:
refs/heads/master 643dafdec -> 3398c95b0
Revert "Allowed whitelist additional devices in cgroups devices subsystem."
This reverts commit ff9ed0c831c347204d065c5f39e5c8bb86f38514.
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/3398c95b
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/3398c95b
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/3398c95b
Branch: refs/heads/master
Commit: 3398c95b0cbdf37a7ad8078fdbdb79e020e305ca
Parents: 643dafd
Author: Haosdent Huang
Authored: Tue Apr 18 10:09:23 2017 +0800
Committer: Haosdent Huang
Committed: Tue Apr 18 10:09:23 2017 +0800
--
docs/configuration.md | 21
.../isolators/cgroups/subsystems/devices.cpp| 111 +++
.../isolators/cgroups/subsystems/devices.hpp| 9 +-
src/slave/flags.cpp | 16 ---
src/slave/flags.hpp | 1 -
5 files changed, 18 insertions(+), 140 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/3398c95b/docs/configuration.md
--
diff --git a/docs/configuration.md b/docs/configuration.md
index 3c9aabc..159f946 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1153,27 +1153,6 @@ effect only when the
--cgroups_net_cls_primary_handle is set.
---cgroups_whitelist_devices
-
-
-JSON array representing the devices that will be additionally
-whitelisted by cgroups devices subsystem. This will take effect
-only when cgroups/devices is set in --isolation flag.
-
-Example:
-[
- {
-"path": "/path/to/device",
-"read_access": true,
-"write_access": false,
-"mknod_access": false
- }
-]
-
-
-
-
-
--cgroups_root=VALUE
http://git-wip-us.apache.org/repos/asf/mesos/blob/3398c95b/src/slave/containerizer/mesos/isolators/cgroups/subsystems/devices.cpp
--
diff --git
a/src/slave/containerizer/mesos/isolators/cgroups/subsystems/devices.cpp
b/src/slave/containerizer/mesos/isolators/cgroups/subsystems/devices.cpp
index d96e716..9b5cf83 100644
--- a/src/slave/containerizer/mesos/isolators/cgroups/subsystems/devices.cpp
+++ b/src/slave/containerizer/mesos/isolators/cgroups/subsystems/devices.cpp
@@ -14,13 +14,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-#include
-
#include
#include
#include
-#include
+
+#include "linux/cgroups.hpp"
#include "slave/containerizer/mesos/isolators/cgroups/subsystems/devices.hpp"
@@ -31,7 +30,6 @@ using process::Future;
using process::Owned;
using std::string;
-using std::vector;
namespace mesos {
namespace internal {
@@ -65,98 +63,15 @@ Try> DevicesSubsystem::create(
const Flags& flags,
const string& hierarchy)
{
- vector whitelistDeviceEntries;
-
- foreach (const char* _entry, DEFAULT_WHITELIST_ENTRIES) {
-Try entry =
- cgroups::devices::Entry::parse(_entry);
-
-CHECK_SOME(entry);
-whitelistDeviceEntries.push_back(entry.get());
- }
-
- if (flags.cgroups_whitelist_devices.isSome()) {
-foreach (const JSON::Value& value,
- flags.cgroups_whitelist_devices.get().values) {
- if (!value.is()) {
-return Error(
-"Failed to parse whitelist devices '" +
-stringify(flags.cgroups_whitelist_devices.get()) +
-"' in flag --cgroups_whitelist_devices");
- }
-
- JSON::Object object = value.as();
-
- Result path = object.at("path");
- if (!path.isSome()) {
-return Error("Malformed whitelist device entry '" +
- stringify(object) + "'");
- }
-
- Result _readAccess =
-object.at("read_access");
-
- Result _writeAccess =
-object.at("write_access");
-
- Result _mknodAccess =
-object.at("mknod_access");
-
- bool readAccess = (_readAccess.isSome() && _readAccess->value);
- bool writeAccess = (_readAccess.isSome() && _readAccess->value);
- bool mknodAccess = (_readAccess.isSome() && _readAccess->value);
-
- if (!(readAccess || writeAccess || mknodAccess)) {
-return Error("Could not whitelist device '" + path->value
- + "' without any access privileges");
- }
-
- Try device = os::stat::rdev(path->value);
- if (device.isError()) {
-return Error("Failed to obtain device ID for '" + path->value +
- "': " + device.error());
- }
-
- Try mode = os::stat::mode(path->value);
- if (mode.isError()) {
-return Error("Failed to obtain device mode for '" + path->value +
- "': " + mode.err
[2/2] mesos git commit: Added MESOS-7210 to 1.1.2 CHANGELOG.
Added MESOS-7210 to 1.1.2 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/ea5056aa Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/ea5056aa Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/ea5056aa Branch: refs/heads/1.1.x Commit: ea5056aaa3044de309dc6b76d2b21937cf87c4f5 Parents: c32bd1c Author: Haosdent Huang Authored: Tue Apr 18 10:27:31 2017 +0800 Committer: Haosdent Huang Committed: Tue Apr 18 10:27:31 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/ea5056aa/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index af54db8..ede244f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -6,6 +6,7 @@ All Issues: ** Bug * [MESOS-2537] - AC_ARG_ENABLED checks are broken. * [MESOS-7197] - Requesting tiny amount of CPU crashes master. + * [MESOS-7210] - HTTP health check doesn't work when mesos runs with --docker_mesos_image. * [MESOS-7237] - Enabling cgroups_limit_swap can lead to "invalid argument" error. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters.
[1/2] mesos git commit: Fixed health check bug when running agents with `docker_mesos_image`.
Repository: mesos
Updated Branches:
refs/heads/1.1.x 85e2da519 -> ea5056aaa
Fixed health check bug when running agents with `docker_mesos_image`.
When running Mesos agents in docker with the `docker_mesos_image` flag,
HTTP health check would fail because the `mesos-docker-executor` could
not find the pid of the task and don't have permissions to enter the
namespaces of the task. This patch updated the options used to run
`mesos-docker-executor` in a separate docker container and ensure
`mesos-docker-executor` got the appropriate permissions to enter the
namespaces of the tasks.
Review: https://reviews.apache.org/r/58200/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/c32bd1ce
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/c32bd1ce
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/c32bd1ce
Branch: refs/heads/1.1.x
Commit: c32bd1ce77b2c04568b7bc8e1d0462c3a28efba5
Parents: 85e2da5
Author: Deshi Xiao
Authored: Mon Apr 17 02:00:47 2017 +0800
Committer: Haosdent Huang
Committed: Tue Apr 18 10:23:54 2017 +0800
--
src/slave/containerizer/docker.cpp | 18 ++
1 file changed, 18 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/c32bd1ce/src/slave/containerizer/docker.cpp
--
diff --git a/src/slave/containerizer/docker.cpp
b/src/slave/containerizer/docker.cpp
index 750f1b6..33b8f67 100644
--- a/src/slave/containerizer/docker.cpp
+++ b/src/slave/containerizer/docker.cpp
@@ -328,6 +328,24 @@ DockerContainerizerProcess::Container::create(
ContainerInfo::DockerInfo dockerInfo;
dockerInfo.set_image(flags.docker_mesos_image.get());
+// `--pid=host` is required for `mesos-docker-executor` to find
+// the pid of the task in `/proc` when running
+// `mesos-docker-executor` in a separate docker container.
+Parameter* pidParameter = dockerInfo.add_parameters();
+pidParameter ->set_key("pid");
+pidParameter->set_value("host");
+
+// `--cap-add=SYS_ADMIN` and `--cap-add=SYS_PTRACE` are required
+// for `mesos-docker-executor` to enter the namespaces of the task
+// during health checking when running `mesos-docker-executor` in a
+// separate docker container.
+Parameter* capAddParameter = dockerInfo.add_parameters();
+capAddParameter->set_key("cap-add");
+capAddParameter->set_value("SYS_ADMIN");
+capAddParameter = dockerInfo.add_parameters();
+capAddParameter->set_key("cap-add");
+capAddParameter->set_value("SYS_PTRACE");
+
newContainerInfo.mutable_docker()->CopyFrom(dockerInfo);
// NOTE: We do not set the optional `taskEnvironment` here as
[1/2] mesos git commit: Fixed health check bug when running agents with `docker_mesos_image`.
Repository: mesos
Updated Branches:
refs/heads/1.2.x 78cf56e9e -> 6855b50a4
Fixed health check bug when running agents with `docker_mesos_image`.
When running Mesos agents in docker with the `docker_mesos_image` flag,
HTTP health check would fail because the `mesos-docker-executor` could
not find the pid of the task and don't have permissions to enter the
namespaces of the task. This patch updated the options used to run
`mesos-docker-executor` in a separate docker container and ensure
`mesos-docker-executor` got the appropriate permissions to enter the
namespaces of the tasks.
Review: https://reviews.apache.org/r/58200/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0ea4e632
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0ea4e632
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/0ea4e632
Branch: refs/heads/1.2.x
Commit: 0ea4e632246405561af832b04ed59e2c1e2343e2
Parents: 78cf56e
Author: Deshi Xiao
Authored: Mon Apr 17 02:00:47 2017 +0800
Committer: Haosdent Huang
Committed: Tue Apr 18 10:28:58 2017 +0800
--
src/slave/containerizer/docker.cpp | 18 ++
1 file changed, 18 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/0ea4e632/src/slave/containerizer/docker.cpp
--
diff --git a/src/slave/containerizer/docker.cpp
b/src/slave/containerizer/docker.cpp
index 029df97..ff7ab9b 100644
--- a/src/slave/containerizer/docker.cpp
+++ b/src/slave/containerizer/docker.cpp
@@ -353,6 +353,24 @@ DockerContainerizerProcess::Container::create(
ContainerInfo::DockerInfo dockerInfo;
dockerInfo.set_image(flags.docker_mesos_image.get());
+// `--pid=host` is required for `mesos-docker-executor` to find
+// the pid of the task in `/proc` when running
+// `mesos-docker-executor` in a separate docker container.
+Parameter* pidParameter = dockerInfo.add_parameters();
+pidParameter ->set_key("pid");
+pidParameter->set_value("host");
+
+// `--cap-add=SYS_ADMIN` and `--cap-add=SYS_PTRACE` are required
+// for `mesos-docker-executor` to enter the namespaces of the task
+// during health checking when running `mesos-docker-executor` in a
+// separate docker container.
+Parameter* capAddParameter = dockerInfo.add_parameters();
+capAddParameter->set_key("cap-add");
+capAddParameter->set_value("SYS_ADMIN");
+capAddParameter = dockerInfo.add_parameters();
+capAddParameter->set_key("cap-add");
+capAddParameter->set_value("SYS_PTRACE");
+
newContainerInfo.mutable_docker()->CopyFrom(dockerInfo);
// NOTE: We do not set the optional `taskEnvironment` here as
[2/2] mesos git commit: Added MESOS-7210 to 1.2.1 CHANGELOG.
Added MESOS-7210 to 1.2.1 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/6855b50a Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/6855b50a Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/6855b50a Branch: refs/heads/1.2.x Commit: 6855b50a49ec855146cb9bab59f58e0614278d7f Parents: 0ea4e63 Author: Haosdent Huang Authored: Tue Apr 18 10:29:46 2017 +0800 Committer: Haosdent Huang Committed: Tue Apr 18 10:29:46 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/6855b50a/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 608356a..f0daa41 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -7,6 +7,7 @@ All Issues: * [MESOS-6951] - Docker containerizer: mangled environment when env value contains LF byte. * [MESOS-7197] - Requesting tiny amount of CPU crashes master. * [MESOS-7208] - Persistent volume ownership is set to root when task is running with non-root user + * [MESOS-7210] - HTTP health check doesn't work when mesos runs with --docker_mesos_image. * [MESOS-7237] - Enabling cgroups_limit_swap can lead to "invalid argument" error. * [MESOS-7261] - maintenance.html is missing during packaging. * [MESOS-7263] - User supplied task environment variables cause warnings in sandbox stdout.
[2/2] mesos git commit: Added MESOS-7350 to 1.1.2 CHANGELOG.
Added MESOS-7350 to 1.1.2 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/15873bac Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/15873bac Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/15873bac Branch: refs/heads/master Commit: 15873bac33b89de6444f02a619d19c5827894c4d Parents: 6eb5232 Author: Jie Yu Authored: Tue Apr 18 11:21:24 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 11:21:24 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/15873bac/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 9a748f4..84c87fd 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -487,6 +487,7 @@ All Issues: * [MESOS-7237] - Enabling cgroups_limit_swap can lead to "invalid argument" error. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. + * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. Release Notes - Mesos - Version 1.1.1
[1/2] mesos git commit: Added MESOS-7350 to 1.2.1 CHANGELOG.
Repository: mesos Updated Branches: refs/heads/master 3398c95b0 -> 15873bac3 Added MESOS-7350 to 1.2.1 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/6eb52329 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/6eb52329 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/6eb52329 Branch: refs/heads/master Commit: 6eb5232900511a49450bca70ed98e8da91a50e16 Parents: 3398c95 Author: Jie Yu Authored: Tue Apr 18 11:15:19 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 11:15:19 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/6eb52329/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 8cd6635..9a748f4 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -39,6 +39,7 @@ All Issues: * [MESOS-7265] - Containerizer startup may cause sensitive data to leak into sandbox logs. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. + * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. Release Notes - Mesos - Version 1.2.0
[2/2] mesos git commit: Added MESOS-7350 to 1.2.1 CHANGELOG.
Added MESOS-7350 to 1.2.1 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/cb947417 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/cb947417 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/cb947417 Branch: refs/heads/1.2.x Commit: cb94741787602faedaba52a894ddece2c6264b4f Parents: d6a586b Author: Jie Yu Authored: Tue Apr 18 11:15:19 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 11:22:26 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/cb947417/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index f0daa41..2601e93 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -15,6 +15,7 @@ All Issues: * [MESOS-7265] - Containerizer startup may cause sensitive data to leak into sandbox logs. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. + * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. Release Notes - Mesos - Version 1.2.0
[1/2] mesos git commit: Fixed the image signature check for Nexus Registry.
Repository: mesos
Updated Branches:
refs/heads/1.2.x 6855b50a4 -> cb9474178
Fixed the image signature check for Nexus Registry.
Currently, the signature field of the docker v2 image manifest is
not used yet. The check of at least one image signature is too
strict because some registry (e.g., Nexus Registry) does not sign
the image manifest. We should release the signature check for now.
Review: https://reviews.apache.org/r/58479/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/d6a586be
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/d6a586be
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/d6a586be
Branch: refs/heads/1.2.x
Commit: d6a586be10cd4c2fad73603526ba1502214ae41b
Parents: 6855b50
Author: Gilbert Song
Authored: Tue Apr 18 07:57:30 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 11:17:43 2017 +0800
--
src/docker/spec.cpp | 4
1 file changed, 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/d6a586be/src/docker/spec.cpp
--
diff --git a/src/docker/spec.cpp b/src/docker/spec.cpp
index 88029c2..6b5588e 100644
--- a/src/docker/spec.cpp
+++ b/src/docker/spec.cpp
@@ -332,10 +332,6 @@ Option validate(const ImageManifest& manifest)
return Error("'history' field size must be at least one");
}
- if (manifest.signatures_size() <= 0) {
-return Error("'signatures' field size must be at least one");
- }
-
// Verify that blobSum and v1Compatibility numbers are equal.
if (manifest.fslayers_size() != manifest.history_size()) {
return Error("The size of 'fsLayers' should be equal "
[1/2] mesos git commit: Fixed the image signature check for Nexus Registry.
Repository: mesos
Updated Branches:
refs/heads/1.1.x ea5056aaa -> f7cfae90e
Fixed the image signature check for Nexus Registry.
Currently, the signature field of the docker v2 image manifest is
not used yet. The check of at least one image signature is too
strict because some registry (e.g., Nexus Registry) does not sign
the image manifest. We should release the signature check for now.
Review: https://reviews.apache.org/r/58479/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/e5c3997c
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/e5c3997c
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/e5c3997c
Branch: refs/heads/1.1.x
Commit: e5c3997c35c790d4caf928dbcf91aaab7af5ed1c
Parents: ea5056a
Author: Gilbert Song
Authored: Tue Apr 18 07:57:30 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 11:16:16 2017 +0800
--
src/docker/spec.cpp | 4
1 file changed, 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/e5c3997c/src/docker/spec.cpp
--
diff --git a/src/docker/spec.cpp b/src/docker/spec.cpp
index 2f2c32e..e022a33 100644
--- a/src/docker/spec.cpp
+++ b/src/docker/spec.cpp
@@ -330,10 +330,6 @@ Option validate(const ImageManifest& manifest)
return Error("'history' field size must be at least one");
}
- if (manifest.signatures_size() <= 0) {
-return Error("'signatures' field size must be at least one");
- }
-
// Verify that blobSum and v1Compatibility numbers are equal.
if (manifest.fslayers_size() != manifest.history_size()) {
return Error("The size of 'fsLayers' should be equal "
[2/2] mesos git commit: Added MESOS-7350 to 1.1.2 CHANGELOG.
Added MESOS-7350 to 1.1.2 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/f7cfae90 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/f7cfae90 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/f7cfae90 Branch: refs/heads/1.1.x Commit: f7cfae90ef42f9456a9c5c012333851096bfe01f Parents: e5c3997 Author: Jie Yu Authored: Tue Apr 18 11:21:24 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 11:23:09 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/f7cfae90/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index ede244f..9200af6 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -10,6 +10,7 @@ All Issues: * [MESOS-7237] - Enabling cgroups_limit_swap can lead to "invalid argument" error. * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. + * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. Release Notes - Mesos - Version 1.1.1
[3/4] mesos git commit: Unit test for file/symlink/directory overwriting in provisioners.
Unit test for file/symlink/directory overwriting in provisioners.
The test is based on the following image:
https://hub.docker.com/r/chhsiao/overwrite/
Review: https://reviews.apache.org/r/58443/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/6a3b4248
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/6a3b4248
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/6a3b4248
Branch: refs/heads/master
Commit: 6a3b4248924a7473d64b7da789897193c6927c7f
Parents: 3c8deed
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:19:09 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:19:09 2017 +0800
--
.../containerizer/provisioner_docker_tests.cpp | 117 +++
1 file changed, 117 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/6a3b4248/src/tests/containerizer/provisioner_docker_tests.cpp
--
diff --git a/src/tests/containerizer/provisioner_docker_tests.cpp
b/src/tests/containerizer/provisioner_docker_tests.cpp
index d1224d8..2620de7 100644
--- a/src/tests/containerizer/provisioner_docker_tests.cpp
+++ b/src/tests/containerizer/provisioner_docker_tests.cpp
@@ -731,6 +731,123 @@ TEST_P(ProvisionerDockerWhiteoutTest,
ROOT_INTERNET_CURL_Whiteout)
}
+class ProvisionerDockerOverwriteTest
+ : public MesosTest,
+public WithParamInterface
+{
+public:
+ // Returns the supported backends.
+ static vector parameters()
+ {
+vector backends = {COPY_BACKEND};
+
+Try aufsSupported = fs::supported("aufs");
+if (aufsSupported.isSome() && aufsSupported.get()) {
+ backends.push_back(AUFS_BACKEND);
+}
+
+Try overlayfsSupported = fs::supported("overlayfs");
+if (overlayfsSupported.isSome() && overlayfsSupported.get()) {
+ backends.push_back(OVERLAY_BACKEND);
+}
+
+return backends;
+ }
+};
+
+
+INSTANTIATE_TEST_CASE_P(
+BackendFlag,
+ProvisionerDockerOverwriteTest,
+::testing::ValuesIn(ProvisionerDockerOverwriteTest::parameters()));
+
+
+// This test verifies that the provisioner correctly overwrites a
+// directory in underlying layers with a with a regular file or symbolic
+// link of the same name in an upper layer, and vice versa.
+TEST_P(ProvisionerDockerOverwriteTest, ROOT_INTERNET_CURL_Overwrite)
+{
+ Try> master = StartMaster();
+ ASSERT_SOME(master);
+
+ slave::Flags flags = CreateSlaveFlags();
+ flags.isolation = "docker/runtime,filesystem/linux";
+ flags.image_providers = "docker";
+ flags.image_provisioner_backend = GetParam();
+
+ Owned detector = master.get()->createDetector();
+ Try> slave = StartSlave(detector.get(), flags);
+ ASSERT_SOME(slave);
+
+ MockScheduler sched;
+ MesosSchedulerDriver driver(
+ &sched, DEFAULT_FRAMEWORK_INFO, master.get()->pid, DEFAULT_CREDENTIAL);
+
+ EXPECT_CALL(sched, registered(&driver, _, _));
+
+ Future> offers;
+ EXPECT_CALL(sched, resourceOffers(&driver, _))
+.WillOnce(FutureArg<1>(&offers))
+.WillRepeatedly(Return()); // Ignore subsequent offers.
+
+ driver.start();
+
+ AWAIT_READY(offers);
+ ASSERT_EQ(1u, offers->size());
+
+ const Offer& offer = offers.get()[0];
+
+ // We are using the docker image 'chhsiao/overwrite' to verify that:
+ // 1. The '/merged' directory is merged.
+ // 2. All '/replaced*' files/directories are correctly overwritten.
+ // 3. The '/bar' symlink and '/baz' file are correctly overwritten.
+ // See more details in the following link:
+ // https://hub.docker.com/r/chhsiao/overwrite/
+ CommandInfo command = createCommandInfo(
+ "test -f /replaced1 &&"
+ "test -L /replaced2 &&"
+ "test -f /replaced2/m1 &&"
+ "test -f /replaced2/m2 &&"
+ "! test -e /replaced2/r2 &&"
+ "test -d /replaced3 &&"
+ "test -d /replaced4 &&"
+ "! test -e /replaced4/m1 &&"
+ "test -f /foo &&"
+ "! test -L /bar &&"
+ "test -L /baz");
+
+ TaskInfo task = createTask(
+ offer.slave_id(),
+ Resources::parse("cpus:1;mem:128").get(),
+ command);
+
+ Image image = createDockerImage("chhsiao/overwrite");
+
+ ContainerInfo* container = task.mutable_container();
+ container->set_type(ContainerInfo::MESOS);
+ container->mutable_mesos()->mutable_image()->CopyFrom(image);
+
+ Future statusRunning;
+ Future statusFinished;
+ EXPECT_CALL(sched, statusUpdate(&driver, _))
+.WillOnce(FutureArg<1>(&statusRunning))
+.WillOnce(FutureArg<1>(&statusFinished));
+
+ driver.launchTasks(offer.id(), {task});
+
+ AWAIT_READY_FOR(statusRunning, Seconds(60));
+ EXPECT_EQ(task.task_id(), statusRunning->task_id());
+ EXPECT_EQ(TASK_RUNNING, statusRunning->state());
+
+ AWAIT_READY(statusFinished);
+ EXPECT_EQ(task.task_id(), statusFinished->task_id());
+ EXPECT_EQ(TASK_FINISHED, statusFinished->state()
[4/4] mesos git commit: Renaming for Provisioner Tests.
Renaming for Provisioner Tests.
Renamed ProvisionerDockerPullerTest to ProvisionerDockeTest.
Renamed ProvisionerDockerWhiteoutTest to
ProvisionerDockerBackendTest.ROOT_INTERNET_CURL_Whiteout.
Renamed ProvisionerDockerOverwriteTest to
ProvisionerDockerBackendTest.ROOT_INTERNET_CURL_Overwrite.
Review: https://reviews.apache.org/r/58488/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/45811356
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/45811356
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/45811356
Branch: refs/heads/master
Commit: 45811356c03d50cb759ddfb914ea016727f7fc25
Parents: 6a3b424
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:19:28 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:19:28 2017 +0800
--
.../containerizer/provisioner_docker_tests.cpp | 57 +---
1 file changed, 13 insertions(+), 44 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/45811356/src/tests/containerizer/provisioner_docker_tests.cpp
--
diff --git a/src/tests/containerizer/provisioner_docker_tests.cpp
b/src/tests/containerizer/provisioner_docker_tests.cpp
index 2620de7..c26e1f9 100644
--- a/src/tests/containerizer/provisioner_docker_tests.cpp
+++ b/src/tests/containerizer/provisioner_docker_tests.cpp
@@ -369,14 +369,14 @@ TEST_F(ProvisionerDockerLocalStoreTest,
PullingSameImageSimutanuously)
#ifdef __linux__
-class ProvisionerDockerPullerTest
+class ProvisionerDockerTest
: public MesosTest,
public WithParamInterface {};
// This test verifies that local docker image can be pulled and
// provisioned correctly, and shell command should be executed.
-TEST_F(ProvisionerDockerPullerTest, ROOT_LocalPullerSimpleCommand)
+TEST_F(ProvisionerDockerTest, ROOT_LocalPullerSimpleCommand)
{
Try> master = StartMaster();
ASSERT_SOME(master);
@@ -456,7 +456,7 @@ TEST_F(ProvisionerDockerPullerTest,
ROOT_LocalPullerSimpleCommand)
// puller normalize docker official images if necessary.
INSTANTIATE_TEST_CASE_P(
ImageAlpine,
-ProvisionerDockerPullerTest,
+ProvisionerDockerTest,
::testing::ValuesIn(vector({
"alpine", // Verifies the normalization of the Docker repository name.
"library/alpine",
@@ -466,7 +466,7 @@ INSTANTIATE_TEST_CASE_P(
// TODO(jieyu): This is a ROOT test because of MESOS-4757. Remove the
// ROOT restriction after MESOS-4757 is resolved.
-TEST_P(ProvisionerDockerPullerTest, ROOT_INTERNET_CURL_SimpleCommand)
+TEST_P(ProvisionerDockerTest, ROOT_INTERNET_CURL_SimpleCommand)
{
Try> master = StartMaster();
ASSERT_SOME(master);
@@ -548,7 +548,7 @@ TEST_P(ProvisionerDockerPullerTest,
ROOT_INTERNET_CURL_SimpleCommand)
// This test verifies that the scratch based docker image (that
// only contain a single binary and its dependencies) can be
// launched correctly.
-TEST_F(ProvisionerDockerPullerTest, ROOT_INTERNET_CURL_ScratchImage)
+TEST_F(ProvisionerDockerTest, ROOT_INTERNET_CURL_ScratchImage)
{
Try> master = StartMaster();
ASSERT_SOME(master);
@@ -619,7 +619,7 @@ TEST_F(ProvisionerDockerPullerTest,
ROOT_INTERNET_CURL_ScratchImage)
}
-class ProvisionerDockerWhiteoutTest
+class ProvisionerDockerBackendTest
: public MesosTest,
public WithParamInterface
{
@@ -646,13 +646,13 @@ public:
INSTANTIATE_TEST_CASE_P(
BackendFlag,
-ProvisionerDockerWhiteoutTest,
-::testing::ValuesIn(ProvisionerDockerWhiteoutTest::parameters()));
+ProvisionerDockerBackendTest,
+::testing::ValuesIn(ProvisionerDockerBackendTest::parameters()));
// This test verifies that a docker image containing whiteout files
// will be processed correctly by copy, aufs and overlay backends.
-TEST_P(ProvisionerDockerWhiteoutTest, ROOT_INTERNET_CURL_Whiteout)
+TEST_P(ProvisionerDockerBackendTest, ROOT_INTERNET_CURL_Whiteout)
{
Try> master = StartMaster();
ASSERT_SOME(master);
@@ -731,41 +731,10 @@ TEST_P(ProvisionerDockerWhiteoutTest,
ROOT_INTERNET_CURL_Whiteout)
}
-class ProvisionerDockerOverwriteTest
- : public MesosTest,
-public WithParamInterface
-{
-public:
- // Returns the supported backends.
- static vector parameters()
- {
-vector backends = {COPY_BACKEND};
-
-Try aufsSupported = fs::supported("aufs");
-if (aufsSupported.isSome() && aufsSupported.get()) {
- backends.push_back(AUFS_BACKEND);
-}
-
-Try overlayfsSupported = fs::supported("overlayfs");
-if (overlayfsSupported.isSome() && overlayfsSupported.get()) {
- backends.push_back(OVERLAY_BACKEND);
-}
-
-return backends;
- }
-};
-
-
-INSTANTIATE_TEST_CASE_P(
-BackendFlag,
-ProvisionerDockerOverwriteTest,
-::testing::ValuesIn(ProvisionerDockerOverwriteTest::parameters()));
-
-
//
[1/4] mesos git commit: Overwriting Directories with Files in Copy Provisioner.
Repository: mesos
Updated Branches:
refs/heads/master 15873bac3 -> 45811356c
Overwriting Directories with Files in Copy Provisioner.
When a layer overwrites a directory with a regular file or symbolic
link (or vice versa), the old dir/file need to be removed before
copying the layer into the rootfs. This is processed together with
whiteout: The copy provisioner find all files to remove, including
files marked as whiteout and the files described above, and remove
them before the copy process.
Review: https://reviews.apache.org/r/58408/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/bc12a583
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/bc12a583
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/bc12a583
Branch: refs/heads/master
Commit: bc12a5835590178112ec0d46bbbcb014ed246f3b
Parents: 15873ba
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:18:09 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:18:21 2017 +0800
--
.../mesos/provisioner/backends/copy.cpp | 119 +++
1 file changed, 71 insertions(+), 48 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/bc12a583/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
--
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index 584cc65..68178cb 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -147,60 +147,83 @@ Future CopyBackendProcess::_provision(
vector whiteouts;
for (FTSENT *node = ::fts_read(tree);
node != nullptr; node = ::fts_read(tree)) {
-if (node->fts_info != FTS_F) {
+string ftsPath = string(node->fts_path);
+
+if (node->fts_info == FTS_DNR ||
+node->fts_info == FTS_ERR ||
+node->fts_info == FTS_NS) {
+ return Failure(
+ "Failed to read '" + ftsPath + "': " +
os::strerror(node->fts_errno));
+}
+
+// Skip the postorder visit of a directory.
+// See the manpage of fts_read in the following link:
+// http://man7.org/linux/man-pages/man3/fts_read.3.html
+if (node->fts_info == FTS_DP) {
continue;
}
-if (!strings::startsWith(node->fts_name, docker::spec::WHITEOUT_PREFIX)) {
+if (ftsPath == layer) {
continue;
}
-string ftsPath = string(node->fts_path);
-Path whiteout = Path(ftsPath.substr(layer.length() + 1));
-
-// Keep the relative paths of the whiteout files, we will
-// remove them from rootfs after layer is copied to rootfs.
-whiteouts.push_back(whiteout.string());
-
-if (node->fts_name == string(docker::spec::WHITEOUT_OPAQUE_PREFIX)) {
- const string path = path::join(rootfs, Path(whiteout).dirname());
-
- // Remove the entries under the directory labeled
- // as opaque whiteout from rootfs.
- Try rmdir = os::rmdir(path, true, false);
- if (rmdir.isError()) {
-::fts_close(tree);
-return Failure(
-"Failed to remove the entries under the directory labeled as"
-" opaque whiteout '" + path + "': " + rmdir.error());
+string layerPath = ftsPath.substr(layer.length() + 1);
+string rootfsPath = path::join(rootfs, layerPath);
+Option removePath;
+
+// Handle whiteout files.
+if (node->fts_info == FTS_F &&
+strings::startsWith(node->fts_name, docker::spec::WHITEOUT_PREFIX)) {
+ Path whiteout = Path(layerPath);
+
+ // Keep the absolute paths of the whiteout files, we will
+ // remove them from rootfs after layer is copied to rootfs.
+ whiteouts.push_back(rootfsPath);
+
+ if (node->fts_name == string(docker::spec::WHITEOUT_OPAQUE_PREFIX)) {
+removePath = path::join(rootfs, whiteout.dirname());
+ } else {
+removePath = path::join(
+rootfs,
+whiteout.dirname(),
+whiteout.basename().substr(strlen(docker::spec::WHITEOUT_PREFIX)));
}
-} else {
- const string path = path::join(
- rootfs,
- whiteout.dirname(),
- whiteout.basename().substr(strlen(docker::spec::WHITEOUT_PREFIX)));
-
- // The file/directory labeled as whiteout may have already been
- // removed with the code above due to its parent directory labeled
- // as opaque whiteout, so here we need to check if it still exists
- // before trying to remove it.
- if (os::exists(path)) {
-if (os::stat::isdir(path)) {
- Try rmdir = os::rmdir(path);
- if (rmdir.isError()) {
-::fts_close(tree);
-return Failure(
-"Failed to remove the directory labeled as whiteo
[2/4] mesos git commit: Overwriting Symbolic Links with Files in Copy Provisioner.
Overwriting Symbolic Links with Files in Copy Provisioner.
When a layer overwrites a symbolic link with a regular file, the link
must be removed first, otherwise 'cp' would follow the link and
overwrite the target instead of the link itself.
Review: https://reviews.apache.org/r/58463/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/3c8deedc
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/3c8deedc
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/3c8deedc
Branch: refs/heads/master
Commit: 3c8deedc9a1bce617965c3442713ebdc6691d1ae
Parents: bc12a58
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:18:45 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:18:45 2017 +0800
--
.../mesos/provisioner/backends/copy.cpp | 28 +---
1 file changed, 19 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/3c8deedc/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
--
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index 68178cb..69faa03 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -190,15 +190,25 @@ Future CopyBackendProcess::_provision(
}
}
-// Handle overwriting between directories and non-directories.
-// Note: If a symbolic link is overwritten by a directory, the
-// symbolic link must be removed before the directory is traversed
-// so the following case won't cause a security issue:
-// ROOTFS: /bad@ -> /usr
-// LAYER: /bad/bin/.wh.wh.opq
-bool ftsIsDir = node->fts_info == FTS_D || node->fts_info == FTS_DC;
-if (os::exists(rootfsPath) && os::stat::isdir(rootfsPath) != ftsIsDir) {
- removePath = rootfsPath;
+if (os::exists(rootfsPath)) {
+ bool ftsIsDir = node->fts_info == FTS_D || node->fts_info == FTS_DC;
+ if (os::stat::isdir(rootfsPath) != ftsIsDir) {
+// Handle overwriting between a directory and a non-directory.
+// Note: If a symlink is overwritten by a directory, the symlink
+// must be removed before the directory is traversed so the
+// following case won't cause a security issue:
+// ROOTFS: /bad@ -> /usr
+// LAYER: /bad/bin/.wh.wh.opq
+removePath = rootfsPath;
+ } else if (os::stat::islink(rootfsPath)) {
+// Handle overwriting a symlink with a regular file.
+// Note: The symlink must be removed, or 'cp' would follow the
+// link and overwrite the target instead of the link itself,
+// which would cause a security issue in the following case:
+// ROOTFS: /bad@ -> /usr/bin/python
+// LAYER: /bad is a malicious executable
+removePath = rootfsPath;
+ }
}
// The file/directory referred to by removePath may be empty or have
[2/2] mesos git commit: Added MESOS-5028 to 1.1.2 CHANGELOG.
Added MESOS-5028 to 1.1.2 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/85e324e2 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/85e324e2 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/85e324e2 Branch: refs/heads/master Commit: 85e324e2a7ae0fb7cc2fa10630ea1c8993dfdc3c Parents: 5301339 Author: Jie Yu Authored: Tue Apr 18 14:29:56 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 14:29:56 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/85e324e2/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 93817a1..d9a0985 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -489,6 +489,7 @@ All Issues: * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. + * [MESOS-5028] - Copy provisioner cannot replace directory with symlink. Release Notes - Mesos - Version 1.1.1
[1/3] mesos git commit: Overwriting Symbolic Links with Files in Copy Provisioner.
Repository: mesos
Updated Branches:
refs/heads/1.2.x cb9474178 -> ccfb4cc81
Overwriting Symbolic Links with Files in Copy Provisioner.
When a layer overwrites a symbolic link with a regular file, the link
must be removed first, otherwise 'cp' would follow the link and
overwrite the target instead of the link itself.
Review: https://reviews.apache.org/r/58463/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0c0b3cb1
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0c0b3cb1
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/0c0b3cb1
Branch: refs/heads/1.2.x
Commit: 0c0b3cb1eb8c2e2d7b44eb64f41020eb5446d563
Parents: 994df2b
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:18:45 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:28:04 2017 +0800
--
.../mesos/provisioner/backends/copy.cpp | 28 +---
1 file changed, 19 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/0c0b3cb1/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
--
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index a5cc38d..a54da48 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -188,15 +188,25 @@ Future CopyBackendProcess::_provision(
}
}
-// Handle overwriting between directories and non-directories.
-// Note: If a symbolic link is overwritten by a directory, the
-// symbolic link must be removed before the directory is traversed
-// so the following case won't cause a security issue:
-// ROOTFS: /bad@ -> /usr
-// LAYER: /bad/bin/.wh.wh.opq
-bool ftsIsDir = node->fts_info == FTS_D || node->fts_info == FTS_DC;
-if (os::exists(rootfsPath) && os::stat::isdir(rootfsPath) != ftsIsDir) {
- removePath = rootfsPath;
+if (os::exists(rootfsPath)) {
+ bool ftsIsDir = node->fts_info == FTS_D || node->fts_info == FTS_DC;
+ if (os::stat::isdir(rootfsPath) != ftsIsDir) {
+// Handle overwriting between a directory and a non-directory.
+// Note: If a symlink is overwritten by a directory, the symlink
+// must be removed before the directory is traversed so the
+// following case won't cause a security issue:
+// ROOTFS: /bad@ -> /usr
+// LAYER: /bad/bin/.wh.wh.opq
+removePath = rootfsPath;
+ } else if (os::stat::islink(rootfsPath)) {
+// Handle overwriting a symlink with a regular file.
+// Note: The symlink must be removed, or 'cp' would follow the
+// link and overwrite the target instead of the link itself,
+// which would cause a security issue in the following case:
+// ROOTFS: /bad@ -> /usr/bin/python
+// LAYER: /bad is a malicious executable
+removePath = rootfsPath;
+ }
}
// The file/directory referred to by removePath may be empty or have
[1/2] mesos git commit: Added MESOS-5028 to 1.2.1 CHANGELOG.
Repository: mesos Updated Branches: refs/heads/master 45811356c -> 85e324e2a Added MESOS-5028 to 1.2.1 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5301339e Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/5301339e Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/5301339e Branch: refs/heads/master Commit: 5301339e9974304399ff98a20e997a1378eed8ca Parents: 4581135 Author: Jie Yu Authored: Tue Apr 18 14:29:21 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 14:29:21 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/5301339e/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 84c87fd..93817a1 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -40,6 +40,7 @@ All Issues: * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. + * [MESOS-5028] - Copy provisioner cannot replace directory with symlink. Release Notes - Mesos - Version 1.2.0
[2/3] mesos git commit: Overwriting Directories with Files in Copy Provisioner.
Overwriting Directories with Files in Copy Provisioner.
When a layer overwrites a directory with a regular file or symbolic
link (or vice versa), the old dir/file need to be removed before
copying the layer into the rootfs. This is processed together with
whiteout: The copy provisioner find all files to remove, including
files marked as whiteout and the files described above, and remove
them before the copy process.
Review: https://reviews.apache.org/r/58408/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/994df2bb
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/994df2bb
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/994df2bb
Branch: refs/heads/1.2.x
Commit: 994df2bbfd9ff40aff796594948a625abd56b804
Parents: cb94741
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:18:09 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:28:04 2017 +0800
--
.../mesos/provisioner/backends/copy.cpp | 119 +++
1 file changed, 71 insertions(+), 48 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/994df2bb/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
--
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index 0ce3e1e..a5cc38d 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -145,60 +145,83 @@ Future CopyBackendProcess::_provision(
vector whiteouts;
for (FTSENT *node = ::fts_read(tree);
node != nullptr; node = ::fts_read(tree)) {
-if (node->fts_info != FTS_F) {
+string ftsPath = string(node->fts_path);
+
+if (node->fts_info == FTS_DNR ||
+node->fts_info == FTS_ERR ||
+node->fts_info == FTS_NS) {
+ return Failure(
+ "Failed to read '" + ftsPath + "': " +
os::strerror(node->fts_errno));
+}
+
+// Skip the postorder visit of a directory.
+// See the manpage of fts_read in the following link:
+// http://man7.org/linux/man-pages/man3/fts_read.3.html
+if (node->fts_info == FTS_DP) {
continue;
}
-if (!strings::startsWith(node->fts_name, docker::spec::WHITEOUT_PREFIX)) {
+if (ftsPath == layer) {
continue;
}
-string ftsPath = string(node->fts_path);
-Path whiteout = Path(ftsPath.substr(layer.length() + 1));
-
-// Keep the relative paths of the whiteout files, we will
-// remove them from rootfs after layer is copied to rootfs.
-whiteouts.push_back(whiteout.string());
-
-if (node->fts_name == string(docker::spec::WHITEOUT_OPAQUE_PREFIX)) {
- const string path = path::join(rootfs, Path(whiteout).dirname());
-
- // Remove the entries under the directory labeled
- // as opaque whiteout from rootfs.
- Try rmdir = os::rmdir(path, true, false);
- if (rmdir.isError()) {
-::fts_close(tree);
-return Failure(
-"Failed to remove the entries under the directory labeled as"
-" opaque whiteout '" + path + "': " + rmdir.error());
+string layerPath = ftsPath.substr(layer.length() + 1);
+string rootfsPath = path::join(rootfs, layerPath);
+Option removePath;
+
+// Handle whiteout files.
+if (node->fts_info == FTS_F &&
+strings::startsWith(node->fts_name, docker::spec::WHITEOUT_PREFIX)) {
+ Path whiteout = Path(layerPath);
+
+ // Keep the absolute paths of the whiteout files, we will
+ // remove them from rootfs after layer is copied to rootfs.
+ whiteouts.push_back(rootfsPath);
+
+ if (node->fts_name == string(docker::spec::WHITEOUT_OPAQUE_PREFIX)) {
+removePath = path::join(rootfs, whiteout.dirname());
+ } else {
+removePath = path::join(
+rootfs,
+whiteout.dirname(),
+whiteout.basename().substr(strlen(docker::spec::WHITEOUT_PREFIX)));
}
-} else {
- const string path = path::join(
- rootfs,
- whiteout.dirname(),
- whiteout.basename().substr(strlen(docker::spec::WHITEOUT_PREFIX)));
-
- // The file/directory labeled as whiteout may have already been
- // removed with the code above due to its parent directory labeled
- // as opaque whiteout, so here we need to check if it still exists
- // before trying to remove it.
- if (os::exists(path)) {
-if (os::stat::isdir(path)) {
- Try rmdir = os::rmdir(path);
- if (rmdir.isError()) {
-::fts_close(tree);
-return Failure(
-"Failed to remove the directory labeled as whiteout '" +
-path + "': " + rmdir.error());
- }
-} el
[3/3] mesos git commit: Added MESOS-5028 to 1.1.2 CHANGELOG.
Added MESOS-5028 to 1.1.2 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/45753a28 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/45753a28 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/45753a28 Branch: refs/heads/1.1.x Commit: 45753a28c17c86462a7b22bb3db165a41eb90490 Parents: 6959040 Author: Jie Yu Authored: Tue Apr 18 14:29:56 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 14:30:57 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/45753a28/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 9200af6..5237af1 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -11,6 +11,7 @@ All Issues: * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. + * [MESOS-5028] - Copy provisioner cannot replace directory with symlink. Release Notes - Mesos - Version 1.1.1
[3/3] mesos git commit: Added MESOS-5028 to 1.2.1 CHANGELOG.
Added MESOS-5028 to 1.2.1 CHANGELOG. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/ccfb4cc8 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/ccfb4cc8 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/ccfb4cc8 Branch: refs/heads/1.2.x Commit: ccfb4cc814b998f76af245d7ccacb6d27d0e022e Parents: 0c0b3cb Author: Jie Yu Authored: Tue Apr 18 14:29:21 2017 +0800 Committer: Jie Yu Committed: Tue Apr 18 14:30:34 2017 +0800 -- CHANGELOG | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/mesos/blob/ccfb4cc8/CHANGELOG -- diff --git a/CHANGELOG b/CHANGELOG index 2601e93..07220e0 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -16,6 +16,7 @@ All Issues: * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire persistent volume content. * [MESOS-7383] - Docker executor logs possibly sensitive parameters. * [MESOS-7350] - Failed to pull image from Nexus Registry due to signature missing. + * [MESOS-5028] - Copy provisioner cannot replace directory with symlink. Release Notes - Mesos - Version 1.2.0
[1/3] mesos git commit: Overwriting Symbolic Links with Files in Copy Provisioner.
Repository: mesos
Updated Branches:
refs/heads/1.1.x f7cfae90e -> 45753a28c
Overwriting Symbolic Links with Files in Copy Provisioner.
When a layer overwrites a symbolic link with a regular file, the link
must be removed first, otherwise 'cp' would follow the link and
overwrite the target instead of the link itself.
Review: https://reviews.apache.org/r/58463/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/6959040f
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/6959040f
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/6959040f
Branch: refs/heads/1.1.x
Commit: 6959040f04210410f867fdb07eddc8bd54d2415e
Parents: 27d79c5
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:18:45 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:28:37 2017 +0800
--
.../mesos/provisioner/backends/copy.cpp | 28 +---
1 file changed, 19 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/6959040f/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
--
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index a5cc38d..a54da48 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -188,15 +188,25 @@ Future CopyBackendProcess::_provision(
}
}
-// Handle overwriting between directories and non-directories.
-// Note: If a symbolic link is overwritten by a directory, the
-// symbolic link must be removed before the directory is traversed
-// so the following case won't cause a security issue:
-// ROOTFS: /bad@ -> /usr
-// LAYER: /bad/bin/.wh.wh.opq
-bool ftsIsDir = node->fts_info == FTS_D || node->fts_info == FTS_DC;
-if (os::exists(rootfsPath) && os::stat::isdir(rootfsPath) != ftsIsDir) {
- removePath = rootfsPath;
+if (os::exists(rootfsPath)) {
+ bool ftsIsDir = node->fts_info == FTS_D || node->fts_info == FTS_DC;
+ if (os::stat::isdir(rootfsPath) != ftsIsDir) {
+// Handle overwriting between a directory and a non-directory.
+// Note: If a symlink is overwritten by a directory, the symlink
+// must be removed before the directory is traversed so the
+// following case won't cause a security issue:
+// ROOTFS: /bad@ -> /usr
+// LAYER: /bad/bin/.wh.wh.opq
+removePath = rootfsPath;
+ } else if (os::stat::islink(rootfsPath)) {
+// Handle overwriting a symlink with a regular file.
+// Note: The symlink must be removed, or 'cp' would follow the
+// link and overwrite the target instead of the link itself,
+// which would cause a security issue in the following case:
+// ROOTFS: /bad@ -> /usr/bin/python
+// LAYER: /bad is a malicious executable
+removePath = rootfsPath;
+ }
}
// The file/directory referred to by removePath may be empty or have
[2/3] mesos git commit: Overwriting Directories with Files in Copy Provisioner.
Overwriting Directories with Files in Copy Provisioner.
When a layer overwrites a directory with a regular file or symbolic
link (or vice versa), the old dir/file need to be removed before
copying the layer into the rootfs. This is processed together with
whiteout: The copy provisioner find all files to remove, including
files marked as whiteout and the files described above, and remove
them before the copy process.
Review: https://reviews.apache.org/r/58408/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/27d79c5b
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/27d79c5b
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/27d79c5b
Branch: refs/heads/1.1.x
Commit: 27d79c5b8a4dc93e73fc373d2d04005661b57a27
Parents: f7cfae9
Author: Chun-Hung Hsiao
Authored: Tue Apr 18 14:18:09 2017 +0800
Committer: Jie Yu
Committed: Tue Apr 18 14:28:37 2017 +0800
--
.../mesos/provisioner/backends/copy.cpp | 119 +++
1 file changed, 71 insertions(+), 48 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/mesos/blob/27d79c5b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
--
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index 0ce3e1e..a5cc38d 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -145,60 +145,83 @@ Future CopyBackendProcess::_provision(
vector whiteouts;
for (FTSENT *node = ::fts_read(tree);
node != nullptr; node = ::fts_read(tree)) {
-if (node->fts_info != FTS_F) {
+string ftsPath = string(node->fts_path);
+
+if (node->fts_info == FTS_DNR ||
+node->fts_info == FTS_ERR ||
+node->fts_info == FTS_NS) {
+ return Failure(
+ "Failed to read '" + ftsPath + "': " +
os::strerror(node->fts_errno));
+}
+
+// Skip the postorder visit of a directory.
+// See the manpage of fts_read in the following link:
+// http://man7.org/linux/man-pages/man3/fts_read.3.html
+if (node->fts_info == FTS_DP) {
continue;
}
-if (!strings::startsWith(node->fts_name, docker::spec::WHITEOUT_PREFIX)) {
+if (ftsPath == layer) {
continue;
}
-string ftsPath = string(node->fts_path);
-Path whiteout = Path(ftsPath.substr(layer.length() + 1));
-
-// Keep the relative paths of the whiteout files, we will
-// remove them from rootfs after layer is copied to rootfs.
-whiteouts.push_back(whiteout.string());
-
-if (node->fts_name == string(docker::spec::WHITEOUT_OPAQUE_PREFIX)) {
- const string path = path::join(rootfs, Path(whiteout).dirname());
-
- // Remove the entries under the directory labeled
- // as opaque whiteout from rootfs.
- Try rmdir = os::rmdir(path, true, false);
- if (rmdir.isError()) {
-::fts_close(tree);
-return Failure(
-"Failed to remove the entries under the directory labeled as"
-" opaque whiteout '" + path + "': " + rmdir.error());
+string layerPath = ftsPath.substr(layer.length() + 1);
+string rootfsPath = path::join(rootfs, layerPath);
+Option removePath;
+
+// Handle whiteout files.
+if (node->fts_info == FTS_F &&
+strings::startsWith(node->fts_name, docker::spec::WHITEOUT_PREFIX)) {
+ Path whiteout = Path(layerPath);
+
+ // Keep the absolute paths of the whiteout files, we will
+ // remove them from rootfs after layer is copied to rootfs.
+ whiteouts.push_back(rootfsPath);
+
+ if (node->fts_name == string(docker::spec::WHITEOUT_OPAQUE_PREFIX)) {
+removePath = path::join(rootfs, whiteout.dirname());
+ } else {
+removePath = path::join(
+rootfs,
+whiteout.dirname(),
+whiteout.basename().substr(strlen(docker::spec::WHITEOUT_PREFIX)));
}
-} else {
- const string path = path::join(
- rootfs,
- whiteout.dirname(),
- whiteout.basename().substr(strlen(docker::spec::WHITEOUT_PREFIX)));
-
- // The file/directory labeled as whiteout may have already been
- // removed with the code above due to its parent directory labeled
- // as opaque whiteout, so here we need to check if it still exists
- // before trying to remove it.
- if (os::exists(path)) {
-if (os::stat::isdir(path)) {
- Try rmdir = os::rmdir(path);
- if (rmdir.isError()) {
-::fts_close(tree);
-return Failure(
-"Failed to remove the directory labeled as whiteout '" +
-path + "': " + rmdir.error());
- }
-} el
