Re: Review Request 72041: Updated default executor to call the `LaunchContainer` agent API.

2020-03-12 Thread Mesos Reviewbot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72041/#review219930
---



Bad patch!

Reviews applied: [72161, 72162, 71858, 71884, 71885, 71886, 71943, 71944, 
71950, 71951, 71952, 71953, 71955, 71956, 72210, 71983, 72022, 72027, 72211, 
72040, 72041]

Failed command: ['bash', '-c', "set -o pipefail; export OS='ubuntu:16.04' 
BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose 
--disable-libtool-wrappers --disable-parallel-test-execution' 
ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/jenkins/buildbot.sh 2>&1 | 
tee build_72041"]

Error:
..
CKAGE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 
-DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 
-DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 
-DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 
-DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 
-DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 
-DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 -DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 
-DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 
-DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. 
-I../../../../3rdparty/stout  -I../../../../3rdparty/stout/include 
-I../boost-1.65.0 -I../elfio-3.2 -I../glog-0.4.0/src 
-I../googletest-release-1.8.0/googlemock/include 
-I../googletest-release-1.8.0/googletest/include 
-I../libarchive-3.3.2/libarchive -D__STDC_FORMAT_MACROS -I../picojson-1.3.0 
-I../protobuf-3.5.0/src -I../rapidjson-1.1.0
 /include  -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0 -Wall -Wsign-compare -Wformat-security 
-fstack-protector-strong -fPIC -fPIE -g1 -O0 -Wno-unused-local-typedefs 
-std=c++11 -c -o stout_tests-ip_tests.o `test -f 'tests/ip_tests.cpp' || echo 
'../../../../3rdparty/stout/'`tests/ip_tests.cpp
g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" 
-DPACKAGE_VERSION=\"1.10.0\" -DPACKAGE_STRING=\"mesos\ 1.10.0\" 
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" 
-DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 
-DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 
-DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 
-DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. -I../../../../3rdparty/stout  
-I../../../../3rdparty/stout/include -I../boost-1.65.0 -I../elfio-3.2 
-I../glog-0.4.0/src -I../googletest-release-1.8.0/googlemock/include 
-I../googletest-rel
 ease-1.8.0/googletest/include -I../libarchive-3.3.2/libarchive 
-D__STDC_FORMAT_MACROS -I../picojson-1.3.0 -I../protobuf-3.5.0/src 
-I../rapidjson-1.1.0/include  -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0   -Wall -Wsign-compare -Wformat-security 
-fstack-protector-strong -fPIC -fPIE -g1 -O0 -Wno-unused-local-typedefs 
-std=c++11 -c -o stout_tests-json_tests.o `test -f 'tests/json_tests.cpp' || 
echo '../../../../3rdparty/stout/'`tests/json_tests.cpp
g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" 
-DPACKAGE_VERSION=\"1.10.0\" -DPACKAGE_STRING=\"mesos\ 1.10.0\" 
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" 
-DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 
-DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 
-DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 
-DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. -I../../../../3rdparty/stout  
-I../../../../3rdparty/stout/include -I../boost-1.65.0 -I../elfio-3.2 
-I../glog-0.4.0/src -I../googletest-release-1.8.0/googlemock/include 
-I../googletest-rel
 ease-1.8.0/googletest/include -I../libarchive-3.3.2/libarchive 
-D__STDC_FORMAT_MACROS -I../picojson-1.3.0 -I../protobuf-3.5.0/src 
-I../rapidjson-1.1.0/include  -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0   -Wall -Wsign-compare -Wformat-security 
-fstack-protector-strong -fPIC -fPIE -g1 -O0 -Wno-unused-local-typedefs 
-std=c++11 -c -o stout_tests-jsonify_t

Re: Review Request 71956: Added a test `ROOT_CGROUPS_CFS_CommandTaskLimits`.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71956/
---

(Updated March 13, 2020, 9:21 a.m.)


Review request for mesos, Andrei Budnik and Greg Mann.


Changes
---

Rebased.


Bugs: MESOS-10047
https://issues.apache.org/jira/browse/MESOS-10047


Repository: mesos


Description
---

Added a test `ROOT_CGROUPS_CFS_CommandTaskLimits`.


Diffs (updated)
-

  src/tests/containerizer/cgroups_isolator_tests.cpp 
f72e6cdab417368e63349915114aeed586e0ef0e 


Diff: https://reviews.apache.org/r/71956/diff/6/

Changes: https://reviews.apache.org/r/71956/diff/5-6/


Testing
---

sudo make check


Thanks,

Qian Zhang



Re: Review Request 72233: Fixed undefined behavior in ActionObject::reserve.

2020-03-12 Thread Mesos Reviewbot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72233/#review219929
---



Bad patch!

Reviews applied: [72233]

Failed command: ['bash', '-c', "set -o pipefail; export OS='ubuntu:16.04' 
BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose 
--disable-libtool-wrappers --disable-parallel-test-execution' 
ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/jenkins/buildbot.sh 2>&1 | 
tee build_72233"]

Error:
..
atus update OPERATION_FINISHED (Status UUID: 
8bd91c02-2b7a-4d76-8a5b-4ac211bee585) for operation UUID 
98d3bdb4-9d59-433b-adac-3668a25ed339 on agent 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e-S0
I0313 00:06:49.156639  3082 http_connection.hpp:131] Sending 
UPDATE_OPERATION_STATUS call to 
http://172.17.0.2:43769/slave(1237)/api/v1/resource_provider
I0313 00:06:49.158118  3096 process.cpp:3671] Handling HTTP event for process 
'slave(1237)' with path: '/slave(1237)/api/v1/resource_provider'
I0313 00:06:49.162569  3088 hierarchical.cpp:1853] Performed allocation for 1 
agents in 1.764905ms
I0313 00:06:49.163259  3092 master.cpp:9485] Sending offers [ 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e-O3 ] to framework 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e- (default) at 
scheduler-7415ea59-81b7-4f70-a422-878b18b00742@172.17.0.2:43769
I0313 00:06:49.163908  3092 sched.cpp:934] Scheduler::resourceOffers took 
82296ns
I0313 00:06:49.167528  3085 process.cpp:3671] Handling HTTP event for process 
'master' with path: '/master/api/v1'
I0313 00:06:49.170282  3083 http.cpp:1405] HTTP POST for /master/api/v1 from 
172.17.0.2:59554
I0313 00:06:49.170552  3083 http.cpp:277] Processing call CREATE_VOLUMES
I0313 00:06:49.171536  3083 master.cpp:3705] Authorizing principal 
'test-principal' to perform action CREATE_VOLUME on object 
{"value":"storage/default-role","resource":{"provider_id":{"value":"09cf7286-d072-4e9f-b244-fac5e5f263af"},"name":"disk","type":"SCALAR","scalar":{"value":2048.0},"reservations":[{"type":"DYNAMIC","role":"storage"},{"type":"DYNAMIC","role":"storage/default-role","principal":"test-principal"}],"disk":{"persistence":{"id":"a19b626b-4ba9-4163-8cb8-4af013820fd3","principal":"test-principal"},"volume":{"mode":"RW","container_path":"volume"},"source":{"type":"MOUNT","mount":{"root":"./csi/org.apache.mesos.csi.test/local/mounts"},"vendor":"org.apache.mesos.csi.test.local","id":"/tmp/CSIVersion_StorageLocalResourceProviderTest_OperatorOperationsWithResourceProviderResources_v1_u1NRe6/2GB-0e93e415-ba68-4b66-bf0d-7bc7611ba6aa","profile":"test"
I0313 00:06:49.173388  3091 sched.cpp:960] Rescinded offer 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e-O3
I0313 00:06:49.173463  3091 sched.cpp:971] Scheduler::offerRescinded took 
23066ns
I0313 00:06:49.173961  3092 hierarchical.cpp:1576] Recovered ports(allocated: 
storage/default-role):[31000-32000]; disk(allocated: 
storage/default-role)(reservations: 
[(DYNAMIC,storage),(DYNAMIC,storage/default-role,test-principal)])[MOUNT(org.apache.mesos.csi.test.local,/tmp/CSIVersion_StorageLocalResourceProviderTest_OperatorOperationsWithResourceProviderResources_v1_u1NRe6/2GB-0e93e415-ba68-4b66-bf0d-7bc7611ba6aa,test)]:2048;
 cpus(allocated: storage/default-role):2; mem(allocated: 
storage/default-role):1024; disk(allocated: storage/default-role):1024 (total: 
cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations: 
[(DYNAMIC,storage),(DYNAMIC,storage/default-role,test-principal)])[MOUNT(org.apache.mesos.csi.test.local,/tmp/CSIVersion_StorageLocalResourceProviderTest_OperatorOperationsWithResourceProviderResources_v1_u1NRe6/2GB-0e93e415-ba68-4b66-bf0d-7bc7611ba6aa,test)]:2048,
 offered or allocated: {}) on agent a330d48b-9410-478c-8b3a-a92f5a5a0d6e-S0 
from framework a330d
 48b-9410-478c-8b3a-a92f5a5a0d6e-
I0313 00:06:49.174172  3096 master.cpp:11784] Removing offer 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e-O3
I0313 00:06:49.175734  3092 hierarchical.cpp:1625] Framework 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e- filtered agent 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e-S0 for 5secs
I0313 00:06:49.178416  3085 master.cpp:11649] Sending operation '' (uuid: 
17a3a443-f24e-4549-ba5a-31cb147963f5) to agent 
a330d48b-9410-478c-8b3a-a92f5a5a0d6e-S0 at slave(1237)@172.17.0.2:43769 
(b49bdfa63993)
I0313 00:06:49.179114  3087 slave.cpp:4466] Ignoring new checkpointed resources 
and operations identical to the current version
I0313 00:06:49.182605  3090 provider.cpp:498] Received APPLY_OPERATION event
I0313 00:06:49.182662  3090 provider.cpp:1351] Received CREATE operation '' 
(uuid: 17a3a443-f24e-4549-ba5a-31cb147963f5)
I0313 00:06:49.185658  3086 master.cpp:5533] Processing REVIVE call for 
framework a330d48b-9410-478c-8b3a-a92f5a5a0d6e- (default) at 
scheduler-7415ea59-81b7-4f70-a422-878b18b00742@172.17.0.2:43769
I0313 00:06:49.186375  3086 hierarchical.cpp:1721] Unsuppressed offers and 
cleared filters for role

Review Request 72233: Fixed undefined behavior in ActionObject::reserve.

2020-03-12 Thread Benjamin Mahler

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72233/
---

Review request for mesos and Andrei Sekretenko.


Repository: mesos


Description
---

Found by Andrei Sekretenko using -fsanitize=address, the following
expression has an intermediate temporary!

  const RepeatedPtrField&
ancestorReservations =
RepeatedPtrField(ancestor).begin()->reservations();

Therefore, the intermediate temporary will not have its lifetime
extended and this is undefined behavior which leads to a crash
in the windows CI.


Diffs
-

  src/master/authorization.cpp 6dfa59ad621d9aa0fd1dbbdbe4fa8965e413774f 


Diff: https://reviews.apache.org/r/72233/diff/1/


Testing
---

Ran through windows CI.


Thanks,

Benjamin Mahler



Re: Review Request 72222: Added tests for agent validation of shared cgroups.

2020-03-12 Thread Mesos Reviewbot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7/#review219928
---



Bad patch!

Reviews applied: [72161, 72162, 71858, 71884, 71885, 71886, 71943, 71944, 
71950, 71951, 71952, 71953, 71955, 72216, 72217, 72221, 7]

Failed command: ['bash', '-c', "set -o pipefail; export OS='ubuntu:16.04' 
BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose 
--disable-libtool-wrappers --disable-parallel-test-execution' 
ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/jenkins/buildbot.sh 2>&1 | 
tee build_7"]

Error:
..
GE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 
-DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 
-DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 
-DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 
-DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 
-DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 
-DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 -DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 
-DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 
-DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. 
-I../../../../3rdparty/stout  -I../../../../3rdparty/stout/include 
-I../boost-1.65.0 -I../elfio-3.2 -I../glog-0.4.0/src 
-I../googletest-release-1.8.0/googlemock/include 
-I../googletest-release-1.8.0/googletest/include 
-I../libarchive-3.3.2/libarchive -D__STDC_FORMAT_MACROS -I../picojson-1.3.0 
-I../protobuf-3.5.0/src -I../rapidjson-1.1.0/in
 clude  -I/usr/include/subversion-1 -I/usr/include/apr-1 -I/usr/include/apr-1.0 
   -Wall -Wsign-compare -Wformat-security -fstack-protector-strong -fPIC 
-fPIE -g1 -O0 -Wno-unused-local-typedefs -std=c++11 -c -o 
stout_tests-ip_tests.o `test -f 'tests/ip_tests.cpp' || echo 
'../../../../3rdparty/stout/'`tests/ip_tests.cpp
g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" 
-DPACKAGE_VERSION=\"1.10.0\" -DPACKAGE_STRING=\"mesos\ 1.10.0\" 
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" 
-DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 
-DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 
-DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 
-DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. -I../../../../3rdparty/stout  
-I../../../../3rdparty/stout/include -I../boost-1.65.0 -I../elfio-3.2 
-I../glog-0.4.0/src -I../googletest-release-1.8.0/googlemock/include 
-I../googletest-rel
 ease-1.8.0/googletest/include -I../libarchive-3.3.2/libarchive 
-D__STDC_FORMAT_MACROS -I../picojson-1.3.0 -I../protobuf-3.5.0/src 
-I../rapidjson-1.1.0/include  -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0   -Wall -Wsign-compare -Wformat-security 
-fstack-protector-strong -fPIC -fPIE -g1 -O0 -Wno-unused-local-typedefs 
-std=c++11 -c -o stout_tests-json_tests.o `test -f 'tests/json_tests.cpp' || 
echo '../../../../3rdparty/stout/'`tests/json_tests.cpp
g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" 
-DPACKAGE_VERSION=\"1.10.0\" -DPACKAGE_STRING=\"mesos\ 1.10.0\" 
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" 
-DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 
-DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 
-DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 
-DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. -I../../../../3rdparty/stout  
-I../../../../3rdparty/stout/include -I../boost-1.65.0 -I../elfio-3.2 
-I../glog-0.4.0/src -I../googletest-release-1.8.0/googlemock/include 
-I../googletest-rel
 ease-1.8.0/googletest/include -I../libarchive-3.3.2/libarchive 
-D__STDC_FORMAT_MACROS -I../picojson-1.3.0 -I../protobuf-3.5.0/src 
-I../rapidjson-1.1.0/include  -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0   -Wall -Wsign-compare -Wformat-security 
-fstack-protector-strong -fPIC -fPIE -g1 -O0 -Wno-unused-local-typedefs 
-std=c++11 -c -o stout_tests-jsonify_tests.o `test -f 'tests/jsonify_tests.

Re: Review Request 72222: Added tests for agent validation of shared cgroups.

2020-03-12 Thread Greg Mann

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7/
---

(Updated March 12, 2020, 4:13 p.m.)


Review request for mesos, Andrei Budnik and Qian Zhang.


Bugs: MESOS-10045
https://issues.apache.org/jira/browse/MESOS-10045


Repository: mesos


Description
---

This patch adds validation for shared cgroups when specified
via the agent APIs. In doing so, a new validation test is
added for the agent's LaunchContainer API, since this was
previously missing.


Diffs (updated)
-

  src/tests/slave_validation_tests.cpp 25019ccdc34db35770b0f69301b94e38636059b7 


Diff: https://reviews.apache.org/r/7/diff/2/

Changes: https://reviews.apache.org/r/7/diff/1-2/


Testing
---

`make check`


Thanks,

Greg Mann



Re: Review Request 72221: Added agent validation for shared cgroups.

2020-03-12 Thread Greg Mann

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72221/
---

(Updated March 12, 2020, 4:12 p.m.)


Review request for mesos, Andrei Budnik and Qian Zhang.


Bugs: MESOS-10045
https://issues.apache.org/jira/browse/MESOS-10045


Repository: mesos


Description
---

Added agent validation for shared cgroups.


Diffs (updated)
-

  src/slave/validation.cpp 99b17c965d4cc522e9106078a9e6f34fae396b8a 


Diff: https://reviews.apache.org/r/72221/diff/2/

Changes: https://reviews.apache.org/r/72221/diff/1-2/


Testing
---


Thanks,

Greg Mann



Re: Review Request 72122: Fixed `cgroups::create` for nested cgroups.

2020-03-12 Thread Andrei Budnik


> On Март 12, 2020, 8:50 д.п., Qian Zhang wrote:
> > src/linux/cgroups.cpp
> > Line 665 (original), 677 (patched)
> > 
> >
> > Do we need `path::join("/", ` like the previous code?

No, we don't need to prepend `"/"` to the path because every string in the 
`missingCgroup` is guaranteed to represent an absolute path. These paths are 
constructed by using `path::join`, which automatically adds `"/"` between 
arguments: 

```
currentCgroup = path::join(currentCgroup, *it);
```

Previously, `cgroup` variable could refer to a relative path, so it was 
necessary to prepend `"/"`.


- Andrei


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72122/#review219916
---


On Фев. 12, 2020, 3:48 п.п., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72122/
> ---
> 
> (Updated Фев. 12, 2020, 3:48 п.п.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch modifies `cgroups::create` function to call
> `cloneCpusetCpusMems` for all absent nested cgroups along
> the path to a cgroup that is accepted as an argument of this function.
> For instance, if `cgroups::create` is called to create three
> non-existent cgroups recursively for the path `/a/b/c`, then
> `cloneCpusetCpusMems` is called to clone both `cpuset.cpus` and
> `cpuset.mems` for `/a` from its parent, then `/a/b` from `/a`,
> and so on down the path.
> 
> 
> Diffs
> -
> 
>   src/linux/cgroups.cpp 73646c9eb39948192acedb67e3d2fb13acb14b30 
> 
> 
> Diff: https://reviews.apache.org/r/72122/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 72190: Added a test `LaunchNestedShareCgroups`.

2020-03-12 Thread Mesos Reviewbot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72190/#review219925
---



Bad patch!

Reviews applied: [72121, 72122, 71965, 71966, 72189, 72190]

Failed command: ['bash', '-c', "set -o pipefail; export OS='ubuntu:16.04' 
BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose 
--disable-libtool-wrappers --disable-parallel-test-execution' 
ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/jenkins/buildbot.sh 2>&1 | 
tee build_72190"]

Error:
..
e_manager_process.hpp:528] Forwarding operation status update 
OPERATION_FINISHED (Status UUID: 24ee1ed8-568b-42e9-935d-62319a45d1ec) for 
operation UUID 2500967f-b94e-45e0-8394-ca0b87c00886 on agent 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-S0
I0312 15:30:43.718674  3106 http_connection.hpp:131] Sending 
UPDATE_OPERATION_STATUS call to 
http://172.17.0.2:35181/slave(1237)/api/v1/resource_provider
I0312 15:30:43.719671  3112 process.cpp:3671] Handling HTTP event for process 
'slave(1237)' with path: '/slave(1237)/api/v1/resource_provider'
I0312 15:30:43.722822  3102 hierarchical.cpp:1853] Performed allocation for 1 
agents in 1.149137ms
I0312 15:30:43.723472  3110 master.cpp:9485] Sending offers [ 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-O3 ] to framework 
4bb2a1df-7aad-4ad9-83a7-35b437e64572- (default) at 
scheduler-74557b8a-2606-4667-924a-4afa5bd4dbfb@172.17.0.2:35181
I0312 15:30:43.724125  3110 sched.cpp:934] Scheduler::resourceOffers took 
83602ns
I0312 15:30:43.727680  3105 process.cpp:3671] Handling HTTP event for process 
'master' with path: '/master/api/v1'
I0312 15:30:43.733299  3106 http.cpp:1405] HTTP POST for /master/api/v1 from 
172.17.0.2:57454
I0312 15:30:43.733705  3106 http.cpp:277] Processing call CREATE_VOLUMES
I0312 15:30:43.734825  3106 master.cpp:3705] Authorizing principal 
'test-principal' to perform action CREATE_VOLUME on object 
{"value":"storage/default-role","resource":{"provider_id":{"value":"c02fab3f-71bc-466f-8c45-d1b0f4b1cfee"},"name":"disk","type":"SCALAR","scalar":{"value":2048.0},"reservations":[{"type":"DYNAMIC","role":"storage"},{"type":"DYNAMIC","role":"storage/default-role","principal":"test-principal"}],"disk":{"persistence":{"id":"1453291a-08ac-4289-a7bc-f8a3bebc6f87","principal":"test-principal"},"volume":{"mode":"RW","container_path":"volume"},"source":{"type":"MOUNT","mount":{"root":"./csi/org.apache.mesos.csi.test/local/mounts"},"vendor":"org.apache.mesos.csi.test.local","id":"/tmp/CSIVersion_StorageLocalResourceProviderTest_OperatorOperationsWithResourceProviderResources_v1_iptpzV/2GB-cacaf8a8-e9dd-4688-a28e-09d620d7c945","profile":"test"
I0312 15:30:43.737004  3108 sched.cpp:960] Rescinded offer 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-O3
I0312 15:30:43.737107  3108 sched.cpp:971] Scheduler::offerRescinded took 
36985ns
I0312 15:30:43.737731  3101 master.cpp:11784] Removing offer 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-O3
I0312 15:30:43.737730  3110 hierarchical.cpp:1576] Recovered ports(allocated: 
storage/default-role):[31000-32000]; disk(allocated: 
storage/default-role)(reservations: 
[(DYNAMIC,storage),(DYNAMIC,storage/default-role,test-principal)])[MOUNT(org.apache.mesos.csi.test.local,/tmp/CSIVersion_StorageLocalResourceProviderTest_OperatorOperationsWithResourceProviderResources_v1_iptpzV/2GB-cacaf8a8-e9dd-4688-a28e-09d620d7c945,test)]:2048;
 cpus(allocated: storage/default-role):2; mem(allocated: 
storage/default-role):1024; disk(allocated: storage/default-role):1024 (total: 
cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations: 
[(DYNAMIC,storage),(DYNAMIC,storage/default-role,test-principal)])[MOUNT(org.apache.mesos.csi.test.local,/tmp/CSIVersion_StorageLocalResourceProviderTest_OperatorOperationsWithResourceProviderResources_v1_iptpzV/2GB-cacaf8a8-e9dd-4688-a28e-09d620d7c945,test)]:2048,
 offered or allocated: {}) on agent 4bb2a1df-7aad-4ad9-83a7-35b437e64572-S0 
from framework 4bb2a
 1df-7aad-4ad9-83a7-35b437e64572-
I0312 15:30:43.739643  3110 hierarchical.cpp:1625] Framework 
4bb2a1df-7aad-4ad9-83a7-35b437e64572- filtered agent 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-S0 for 5secs
I0312 15:30:43.743119  3100 master.cpp:11649] Sending operation '' (uuid: 
de10685e-a41f-4fd5-ac37-7919e7571473) to agent 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-S0 at slave(1237)@172.17.0.2:35181 
(5083c546e440)
I0312 15:30:43.743981  3097 slave.cpp:4466] Ignoring new checkpointed resources 
and operations identical to the current version
I0312 15:30:43.745810  3112 master.cpp:5533] Processing REVIVE call for 
framework 4bb2a1df-7aad-4ad9-83a7-35b437e64572- (default) at 
scheduler-74557b8a-2606-4667-924a-4afa5bd4dbfb@172.17.0.2:35181
I0312 15:30:43.746536  3101 hierarchical.cpp:1721] Unsuppressed offers and 
cleared filters for roles { storage/default-role } of framework 
4bb2a1df-7aad-4ad9-83a7-35b437e64572-
I0312 15:30:43.747983  3107 pro

Re: Review Request 71966: Cgroups isolator: added support for nested cgroups during recovery.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71966/#review219922
---


Ship it!




Ship It!

- Qian Zhang


On Feb. 12, 2020, 11:47 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71966/
> ---
> 
> (Updated Feb. 12, 2020, 11:47 p.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Bugs: MESOS-10079
> https://issues.apache.org/jira/browse/MESOS-10079
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch enables recovery for nested cgroups and implements
> the detection of orphaned nested cgroups.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp 
> b12b73d8e0161d448075378765e77867521de04e 
> 
> 
> Diff: https://reviews.apache.org/r/71966/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 71965: Cgroups isolator: added support for nested cgroups during launch.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71965/#review219921
---


Fix it, then Ship it!





src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp
Lines 741-745 (original), 762-765 (patched)


I think we should not change this message, the original message is more 
accurate.


- Qian Zhang


On Feb. 14, 2020, 1:19 a.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71965/
> ---
> 
> (Updated Feb. 14, 2020, 1:19 a.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Bugs: MESOS-10076
> https://issues.apache.org/jira/browse/MESOS-10076
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds support for nested cgroups for nested containers.
> Nested cgroups are created only for a nested container with explicitly
> disabled `share_cgroups` flag. The cgroups isolator stores info about
> nested cgroups in the `infos` class variable, which is used to
> determine whether a nested container has its nested cgroup.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.hpp 
> 4bd3d6dad37dee031660c15e957cc36f63e21fcb 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp 
> b12b73d8e0161d448075378765e77867521de04e 
> 
> 
> Diff: https://reviews.apache.org/r/71965/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 72189: Updated nested mesos containerizer tests to support `shared_cgroups`.

2020-03-12 Thread Greg Mann

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72189/#review219920
---



In commit message: s/shared_cgroups/share_cgroups/

- Greg Mann


On March 4, 2020, 10:56 a.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72189/
> ---
> 
> (Updated March 4, 2020, 10:56 a.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Parametrized some `NestedMesosContainerizerTest` tests on whether we
> are launching nested containers with `shared_cgroups=false` or not.
> Previously, all nested containers shared cgroups with their parent by
> default. Now, since we've added support for nested containers with
> their own cgroups, we need to verify that a nested container with
> `shared_cgroups=false` can be successfully launched and it does not
> lead to errors during the recovery of Mesos containerizer.
> 
> 
> Diffs
> -
> 
>   src/tests/containerizer/nested_mesos_containerizer_tests.cpp 
> c6f96e6b1bc0d9989ca87d23e112604820ac1d51 
> 
> 
> Diff: https://reviews.apache.org/r/72189/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 72190: Added a test `LaunchNestedShareCgroups`.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72190/#review219919
---


Ship it!




Ship It!

- Qian Zhang


On March 4, 2020, 6:56 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72190/
> ---
> 
> (Updated March 4, 2020, 6:56 p.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Added a test `LaunchNestedShareCgroups`.
> 
> 
> Diffs
> -
> 
>   src/tests/containerizer/nested_mesos_containerizer_tests.cpp 
> c6f96e6b1bc0d9989ca87d23e112604820ac1d51 
> 
> 
> Diff: https://reviews.apache.org/r/72190/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 72189: Updated nested mesos containerizer tests to support `shared_cgroups`.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72189/#review219918
---


Ship it!




Ship It!

- Qian Zhang


On March 4, 2020, 6:56 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72189/
> ---
> 
> (Updated March 4, 2020, 6:56 p.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Parametrized some `NestedMesosContainerizerTest` tests on whether we
> are launching nested containers with `shared_cgroups=false` or not.
> Previously, all nested containers shared cgroups with their parent by
> default. Now, since we've added support for nested containers with
> their own cgroups, we need to verify that a nested container with
> `shared_cgroups=false` can be successfully launched and it does not
> lead to errors during the recovery of Mesos containerizer.
> 
> 
> Diffs
> -
> 
>   src/tests/containerizer/nested_mesos_containerizer_tests.cpp 
> c6f96e6b1bc0d9989ca87d23e112604820ac1d51 
> 
> 
> Diff: https://reviews.apache.org/r/72189/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 72122: Fixed `cgroups::create` for nested cgroups.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72122/#review219916
---


Fix it, then Ship it!





src/linux/cgroups.cpp
Line 665 (original), 677 (patched)


Do we need `path::join("/", ` like the previous code?


- Qian Zhang


On Feb. 12, 2020, 11:48 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72122/
> ---
> 
> (Updated Feb. 12, 2020, 11:48 p.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch modifies `cgroups::create` function to call
> `cloneCpusetCpusMems` for all absent nested cgroups along
> the path to a cgroup that is accepted as an argument of this function.
> For instance, if `cgroups::create` is called to create three
> non-existent cgroups recursively for the path `/a/b/c`, then
> `cloneCpusetCpusMems` is called to clone both `cpuset.cpus` and
> `cpuset.mems` for `/a` from its parent, then `/a/b` from `/a`,
> and so on down the path.
> 
> 
> Diffs
> -
> 
>   src/linux/cgroups.cpp 73646c9eb39948192acedb67e3d2fb13acb14b30 
> 
> 
> Diff: https://reviews.apache.org/r/72122/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 71966: Cgroups isolator: added support for nested cgroups during recovery.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71966/#review219914
---




src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp
Lines 278-280 (patched)


This seems not a proper example since cgroups isolator does not care about 
freezer hierarchy at all, maybe we should change it to something like:

e.g., the nested cgroups created by the container itself.


- Qian Zhang


On Feb. 12, 2020, 11:47 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71966/
> ---
> 
> (Updated Feb. 12, 2020, 11:47 p.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Bugs: MESOS-10079
> https://issues.apache.org/jira/browse/MESOS-10079
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch enables recovery for nested cgroups and implements
> the detection of orphaned nested cgroups.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp 
> b12b73d8e0161d448075378765e77867521de04e 
> 
> 
> Diff: https://reviews.apache.org/r/71966/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>



Re: Review Request 71965: Cgroups isolator: added support for nested cgroups during launch.

2020-03-12 Thread Qian Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71965/#review219915
---




src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp
Lines 415-416 (original), 423-425 (patched)


If there are already a top-level container and a 1st level nested container 
(`share_cgroup==false`) running, now a 2nd level nested container 
(`share_cgroup==true`) with resource requests/limits specified is launched, I 
think here we need to increase the resource limits of the 1st level nested 
cgroup to ensure it includes the 2nd level nested container's resources. 
Similarly when that 2nd level nested container terminates, we should decrease 
the 1st level nested cgroup's limits.

Or in MVP, we can just disallow launching 2nd level nested container with 
resource requests/limits specified? I.e., if executor wants to launch a 2nd 
level nested container, that container's `share_cgroups` can only be true and 
it can not have resource requests/limits specified.


- Qian Zhang


On Feb. 14, 2020, 1:19 a.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71965/
> ---
> 
> (Updated Feb. 14, 2020, 1:19 a.m.)
> 
> 
> Review request for mesos, Greg Mann and Qian Zhang.
> 
> 
> Bugs: MESOS-10076
> https://issues.apache.org/jira/browse/MESOS-10076
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds support for nested cgroups for nested containers.
> Nested cgroups are created only for a nested container with explicitly
> disabled `share_cgroups` flag. The cgroups isolator stores info about
> nested cgroups in the `infos` class variable, which is used to
> determine whether a nested container has its nested cgroup.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.hpp 
> 4bd3d6dad37dee031660c15e957cc36f63e21fcb 
>   src/slave/containerizer/mesos/isolators/cgroups/cgroups.cpp 
> b12b73d8e0161d448075378765e77867521de04e 
> 
> 
> Diff: https://reviews.apache.org/r/71965/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>