Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-17 Thread Gilbert Song 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218274
---


Ship it!




Ship It!

- Gilbert Song


On Oct. 17, 2019, 12:15 a.m., Qian Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71608/
> ---
> 
> (Updated Oct. 17, 2019, 12:15 a.m.)
> 
> 
> Review request for mesos and Gilbert Song.
> 
> 
> Bugs: MESOS-9964
> https://issues.apache.org/jira/browse/MESOS-9964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Previously in MESOS-3736, we made Docker store support pulling same
> image simultaneously which is a performance improvement, however it
> may cause an issue: If the pulling hangs somehow, all the subsequent
> pulling request for the same image will hang as well, and as a result
> the container destroy will also hang since destroy has to wait for
> provisioning to finish, see MESOS-4985 for details.
> 
> So in this patch we removed that performance improvement and made UCR
> can destroy the container which is being provisioned, i.e., UCR will
> discard the container provisioning and then keep doing the container
> destroy. And we also improved Docker fetcher plugin so that when
> container provisioning is discarded the `curl` process used to fetch
> manifest or blob will be killed immediately.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 6537f6550b353b1c65a30381f6d68f61508d4960 
>   src/slave/containerizer/mesos/containerizer.cpp 
> c61b954635ebcaed3e498452bddeee6e1b34f388 
>   src/slave/containerizer/mesos/provisioner/docker/store.cpp 
> 286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
>   src/slave/containerizer/mesos/provisioner/provisioner.hpp 
> 38664177ae3805041b412ed52dd0134a6f3aa679 
>   src/slave/containerizer/mesos/provisioner/provisioner.cpp 
> 3d0b291fa878ab22625435afe2a219d6f776e52c 
>   src/tests/containerizer/mesos_containerizer_tests.cpp 
> 449928c10b897061642af8ad267f8b70695940e6 
>   src/tests/containerizer/provisioner_docker_tests.cpp 
> 5d5a355afd9c4fda1c653d6cecb75703b0fe 
>   src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 
> 
> 
> Diff: https://reviews.apache.org/r/71608/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-17 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218257
---



Patch looks great!

Reviews applied: [71608]

Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose --disable-libtool-wrappers 
--disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; 
./support/docker-build.sh

- Mesos Reviewbot


On Oct. 17, 2019, 7:15 a.m., Qian Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71608/
> ---
> 
> (Updated Oct. 17, 2019, 7:15 a.m.)
> 
> 
> Review request for mesos and Gilbert Song.
> 
> 
> Bugs: MESOS-9964
> https://issues.apache.org/jira/browse/MESOS-9964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Previously in MESOS-3736, we made Docker store support pulling same
> image simultaneously which is a performance improvement, however it
> may cause an issue: If the pulling hangs somehow, all the subsequent
> pulling request for the same image will hang as well, and as a result
> the container destroy will also hang since destroy has to wait for
> provisioning to finish, see MESOS-4985 for details.
> 
> So in this patch we removed that performance improvement and made UCR
> can destroy the container which is being provisioned, i.e., UCR will
> discard the container provisioning and then keep doing the container
> destroy. And we also improved Docker fetcher plugin so that when
> container provisioning is discarded the `curl` process used to fetch
> manifest or blob will be killed immediately.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 6537f6550b353b1c65a30381f6d68f61508d4960 
>   src/slave/containerizer/mesos/containerizer.cpp 
> c61b954635ebcaed3e498452bddeee6e1b34f388 
>   src/slave/containerizer/mesos/provisioner/docker/store.cpp 
> 286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
>   src/slave/containerizer/mesos/provisioner/provisioner.hpp 
> 38664177ae3805041b412ed52dd0134a6f3aa679 
>   src/slave/containerizer/mesos/provisioner/provisioner.cpp 
> 3d0b291fa878ab22625435afe2a219d6f776e52c 
>   src/tests/containerizer/mesos_containerizer_tests.cpp 
> 449928c10b897061642af8ad267f8b70695940e6 
>   src/tests/containerizer/provisioner_docker_tests.cpp 
> 5d5a355afd9c4fda1c653d6cecb75703b0fe 
>   src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 
> 
> 
> Diff: https://reviews.apache.org/r/71608/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-17 Thread Qian Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/
---

(Updated Oct. 17, 2019, 3:15 p.m.)


Review request for mesos and Gilbert Song.


Bugs: MESOS-9964
https://issues.apache.org/jira/browse/MESOS-9964


Repository: mesos


Description
---

Previously in MESOS-3736, we made Docker store support pulling same
image simultaneously which is a performance improvement, however it
may cause an issue: If the pulling hangs somehow, all the subsequent
pulling request for the same image will hang as well, and as a result
the container destroy will also hang since destroy has to wait for
provisioning to finish, see MESOS-4985 for details.

So in this patch we removed that performance improvement and made UCR
can destroy the container which is being provisioned, i.e., UCR will
discard the container provisioning and then keep doing the container
destroy. And we also improved Docker fetcher plugin so that when
container provisioning is discarded the `curl` process used to fetch
manifest or blob will be killed immediately.


Diffs (updated)
-

  src/slave/containerizer/mesos/containerizer.hpp 
6537f6550b353b1c65a30381f6d68f61508d4960 
  src/slave/containerizer/mesos/containerizer.cpp 
c61b954635ebcaed3e498452bddeee6e1b34f388 
  src/slave/containerizer/mesos/provisioner/docker/store.cpp 
286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
  src/slave/containerizer/mesos/provisioner/provisioner.hpp 
38664177ae3805041b412ed52dd0134a6f3aa679 
  src/slave/containerizer/mesos/provisioner/provisioner.cpp 
3d0b291fa878ab22625435afe2a219d6f776e52c 
  src/tests/containerizer/mesos_containerizer_tests.cpp 
449928c10b897061642af8ad267f8b70695940e6 
  src/tests/containerizer/provisioner_docker_tests.cpp 
5d5a355afd9c4fda1c653d6cecb75703b0fe 
  src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 


Diff: https://reviews.apache.org/r/71608/diff/5/

Changes: https://reviews.apache.org/r/71608/diff/4-5/


Testing
---


Thanks,

Qian Zhang

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-16 Thread Gilbert Song 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218254
---


Ship it!




Ship It!

- Gilbert Song


On Oct. 16, 2019, 6:43 a.m., Qian Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71608/
> ---
> 
> (Updated Oct. 16, 2019, 6:43 a.m.)
> 
> 
> Review request for mesos and Gilbert Song.
> 
> 
> Bugs: MESOS-9964
> https://issues.apache.org/jira/browse/MESOS-9964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Previously in MESOS-3736, we made Docker store support pulling same
> image simultaneously which is a performance improvement, however it
> may cause an issue: If the pulling hangs somehow, all the subsequent
> pulling request for the same image will hang as well, and as a result
> the container destroy will also hang since destroy has to wait for
> provisioning to finish, see MESOS-4985 for details.
> 
> So in this patch we removed that performance improvement and made UCR
> can destroy the container which is being provisioned, i.e., UCR will
> discard the container provisioning and then keep doing the container
> destroy. And we also improved Docker fetcher plugin so that when
> container provisioning is discarded the `curl` process used to fetch
> manifest or blob will be killed immediately.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 6537f6550b353b1c65a30381f6d68f61508d4960 
>   src/slave/containerizer/mesos/containerizer.cpp 
> c61b954635ebcaed3e498452bddeee6e1b34f388 
>   src/slave/containerizer/mesos/provisioner/docker/store.cpp 
> 286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
>   src/slave/containerizer/mesos/provisioner/provisioner.hpp 
> 38664177ae3805041b412ed52dd0134a6f3aa679 
>   src/slave/containerizer/mesos/provisioner/provisioner.cpp 
> 3d0b291fa878ab22625435afe2a219d6f776e52c 
>   src/tests/containerizer/mesos_containerizer_tests.cpp 
> 449928c10b897061642af8ad267f8b70695940e6 
>   src/tests/containerizer/provisioner_docker_tests.cpp 
> 5d5a355afd9c4fda1c653d6cecb75703b0fe 
>   src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 
> 
> 
> Diff: https://reviews.apache.org/r/71608/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-16 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218230
---



Bad patch!

Reviews applied: [71608]

Failed command: ['bash', '-c', "set -o pipefail; export OS='ubuntu:14.04' 
BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose 
--disable-libtool-wrappers --disable-parallel-test-execution' 
ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh 2>&1 | tee 
build_71608"]

Error:
..
 -DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DHAVE_LIBSASL2=1 
-DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 
-DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" 
-DMESOS_HAS_PYTHON=1 -I. -I../../../3rdparty/libprocess 
-DBUILD_DIR=\"/mesos/mesos-1.10.0/_build/3rdparty/libprocess\" 
-I../../../3rdparty/libprocess/include -I../../../3rdparty/libprocess/src 
-I../boost-1.65.0 -I../concurrentqueue-7b69a8f -I../elfio-3.2 
-I../glog-0.4.0/src -I../grpc-1.10.0/include -I../http-parser-2.6.2 
-I../libev-4.22 -D__STDC_FORMAT_MACROS -I../picojson-1.3.0 
-I../protobuf-3.5.0/src -I../ra
 pidjson-1.1.0/include -I../../../3rdparty/libprocess/../stout/include 
-DLIBPROCESS_ALLOW_JEMALLOC -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0 -Wall -Wsign-compare -Wformat-security -fstack-protector 
-fPIC -g1 -O0 -Wno-unused-local-typedefs -std=c++11 -c 
../../../3rdparty/libprocess/src/authenticator.cpp  -fPIC -DPIC -o 
src/.libs/libprocess_la-authenticator.o
libtool: compile:  g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" 
-DPACKAGE_VERSION=\"1.10.0\" "-DPACKAGE_STRING=\"mesos 1.10.0\"" 
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" 
-DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DHAVE_LIBSASL2=1 
-DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 
-DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" 
-DMESOS_HAS_PYTHON=1 -I. -I../../../3rdparty/libprocess 
-DBUILD_DIR=\"/mesos/mesos-1.10.0/_build/3rdparty/libprocess\" 
-I../../../3rdparty/libprocess/include -I../../../3rdparty/libprocess/src 
-I../boost-1.65.
 0 -I../concurrentqueue-7b69a8f -I../elfio-3.2 -I../glog-0.4.0/src 
-I../grpc-1.10.0/include -I../http-parser-2.6.2 -I../libev-4.22 
-D__STDC_FORMAT_MACROS -I../picojson-1.3.0 -I../protobuf-3.5.0/src 
-I../rapidjson-1.1.0/include -I../../../3rdparty/libprocess/../stout/include 
-DLIBPROCESS_ALLOW_JEMALLOC -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0 -Wall -Wsign-compare -Wformat-security -fstack-protector 
-fPIC -g1 -O0 -Wno-unused-local-typedefs -std=c++11 -c 
../../../3rdparty/libprocess/src/gtest_constants.cpp  -fPIC -DPIC -o 
src/.libs/libprocess_la-gtest_constants.o
libtool: compile:  g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" 
-DPACKAGE_VERSION=\"1.10.0\" "-DPACKAGE_STRING=\"mesos 1.10.0\"" 
-DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" 
-DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 
-DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 
-DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 
-DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 
-DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 -DMESOS_HAS_JAVA=1 -DHAVE_LIBSASL2=1 
-DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 
-DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" 
-DMESOS_HAS_PYTHON=1 -I. -I../../../3rdparty/libprocess 
-DBUILD_DIR=\"/mesos/mesos-1.10.0/_build/3rdparty/libprocess\" 
-I../../../3rdparty/libprocess/include -I../../../3rdparty/libprocess/src 
-I../boost-1.65.
 0 -I../concurrentqueue-7b69a8f -I../elfio-3.2 -I../glog-0.4.0/src 
-I../grpc-1.10.0/include -I../http-parser-2.6.2 -I../libev-4.22 
-D__STDC_FORMAT_MACROS -I../picojson-1.3.0 -I../protobuf-3.5.0/src 
-I../rapidjson-1.1.0/include -I../../../3rdparty/libprocess/../stout/include 
-DLIBPROCESS_ALLOW_JEMALLOC -I/usr/include/subversion-1 -I/usr/include/apr-1 
-I/usr/include/apr-1.0

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-16 Thread Qian Zhang 


> On Oct. 15, 2019, 4:28 a.m., Gilbert Song wrote:
> > src/slave/containerizer/mesos/provisioner/docker/store.cpp
> > Lines 296-312 (patched)
> > 
> >
> > could you just do:
> > 
> > ```
> > promise->associate(future);
> > 
> > return promise->future()
> >   .onDiscard([promise]() {
> > promise->discard();
> >   });
> > ```

We cannot do that because once a promise is associated with a future, its own 
future cannot be discared anymore, see 
https://github.com/apache/mesos/blob/1.9.0/3rdparty/libprocess/include/process/future.hpp#L812:L814
 for details.


- Qian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218206
---


On Oct. 16, 2019, 9:43 p.m., Qian Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71608/
> ---
> 
> (Updated Oct. 16, 2019, 9:43 p.m.)
> 
> 
> Review request for mesos and Gilbert Song.
> 
> 
> Bugs: MESOS-9964
> https://issues.apache.org/jira/browse/MESOS-9964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Previously in MESOS-3736, we made Docker store support pulling same
> image simultaneously which is a performance improvement, however it
> may cause an issue: If the pulling hangs somehow, all the subsequent
> pulling request for the same image will hang as well, and as a result
> the container destroy will also hang since destroy has to wait for
> provisioning to finish, see MESOS-4985 for details.
> 
> So in this patch we removed that performance improvement and made UCR
> can destroy the container which is being provisioned, i.e., UCR will
> discard the container provisioning and then keep doing the container
> destroy. And we also improved Docker fetcher plugin so that when
> container provisioning is discarded the `curl` process used to fetch
> manifest or blob will be killed immediately.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 6537f6550b353b1c65a30381f6d68f61508d4960 
>   src/slave/containerizer/mesos/containerizer.cpp 
> c61b954635ebcaed3e498452bddeee6e1b34f388 
>   src/slave/containerizer/mesos/provisioner/docker/store.cpp 
> 286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
>   src/slave/containerizer/mesos/provisioner/provisioner.hpp 
> 38664177ae3805041b412ed52dd0134a6f3aa679 
>   src/slave/containerizer/mesos/provisioner/provisioner.cpp 
> 3d0b291fa878ab22625435afe2a219d6f776e52c 
>   src/tests/containerizer/mesos_containerizer_tests.cpp 
> 449928c10b897061642af8ad267f8b70695940e6 
>   src/tests/containerizer/provisioner_docker_tests.cpp 
> 5d5a355afd9c4fda1c653d6cecb75703b0fe 
>   src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 
> 
> 
> Diff: https://reviews.apache.org/r/71608/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-16 Thread Qian Zhang 


> On Oct. 12, 2019, 4:34 a.m., Gilbert Song wrote:
> > src/slave/containerizer/mesos/provisioner/docker/store.cpp
> > Line 354 (original), 348 (patched)
> > 
> >
> > we dont need this promise in this case.

We need it to install the `onDiscard` handler and discard its future in the 
handler.


> On Oct. 12, 2019, 4:34 a.m., Gilbert Song wrote:
> > src/slave/containerizer/mesos/provisioner/docker/store.cpp
> > Lines 379 (patched)
> > 
> >
> > call future.discard() if we remove this promise

See my above comment.


- Qian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218194
---


On Oct. 16, 2019, 9:43 p.m., Qian Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71608/
> ---
> 
> (Updated Oct. 16, 2019, 9:43 p.m.)
> 
> 
> Review request for mesos and Gilbert Song.
> 
> 
> Bugs: MESOS-9964
> https://issues.apache.org/jira/browse/MESOS-9964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Previously in MESOS-3736, we made Docker store support pulling same
> image simultaneously which is a performance improvement, however it
> may cause an issue: If the pulling hangs somehow, all the subsequent
> pulling request for the same image will hang as well, and as a result
> the container destroy will also hang since destroy has to wait for
> provisioning to finish, see MESOS-4985 for details.
> 
> So in this patch we removed that performance improvement and made UCR
> can destroy the container which is being provisioned, i.e., UCR will
> discard the container provisioning and then keep doing the container
> destroy. And we also improved Docker fetcher plugin so that when
> container provisioning is discarded the `curl` process used to fetch
> manifest or blob will be killed immediately.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 6537f6550b353b1c65a30381f6d68f61508d4960 
>   src/slave/containerizer/mesos/containerizer.cpp 
> c61b954635ebcaed3e498452bddeee6e1b34f388 
>   src/slave/containerizer/mesos/provisioner/docker/store.cpp 
> 286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
>   src/slave/containerizer/mesos/provisioner/provisioner.hpp 
> 38664177ae3805041b412ed52dd0134a6f3aa679 
>   src/slave/containerizer/mesos/provisioner/provisioner.cpp 
> 3d0b291fa878ab22625435afe2a219d6f776e52c 
>   src/tests/containerizer/mesos_containerizer_tests.cpp 
> 449928c10b897061642af8ad267f8b70695940e6 
>   src/tests/containerizer/provisioner_docker_tests.cpp 
> 5d5a355afd9c4fda1c653d6cecb75703b0fe 
>   src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 
> 
> 
> Diff: https://reviews.apache.org/r/71608/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-16 Thread Qian Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/
---

(Updated Oct. 16, 2019, 9:43 p.m.)


Review request for mesos and Gilbert Song.


Changes
---

Updated test `DestroyWhileProvisioning`.


Bugs: MESOS-9964
https://issues.apache.org/jira/browse/MESOS-9964


Repository: mesos


Description
---

Previously in MESOS-3736, we made Docker store support pulling same
image simultaneously which is a performance improvement, however it
may cause an issue: If the pulling hangs somehow, all the subsequent
pulling request for the same image will hang as well, and as a result
the container destroy will also hang since destroy has to wait for
provisioning to finish, see MESOS-4985 for details.

So in this patch we removed that performance improvement and made UCR
can destroy the container which is being provisioned, i.e., UCR will
discard the container provisioning and then keep doing the container
destroy. And we also improved Docker fetcher plugin so that when
container provisioning is discarded the `curl` process used to fetch
manifest or blob will be killed immediately.


Diffs (updated)
-

  src/slave/containerizer/mesos/containerizer.hpp 
6537f6550b353b1c65a30381f6d68f61508d4960 
  src/slave/containerizer/mesos/containerizer.cpp 
c61b954635ebcaed3e498452bddeee6e1b34f388 
  src/slave/containerizer/mesos/provisioner/docker/store.cpp 
286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
  src/slave/containerizer/mesos/provisioner/provisioner.hpp 
38664177ae3805041b412ed52dd0134a6f3aa679 
  src/slave/containerizer/mesos/provisioner/provisioner.cpp 
3d0b291fa878ab22625435afe2a219d6f776e52c 
  src/tests/containerizer/mesos_containerizer_tests.cpp 
449928c10b897061642af8ad267f8b70695940e6 
  src/tests/containerizer/provisioner_docker_tests.cpp 
5d5a355afd9c4fda1c653d6cecb75703b0fe 
  src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 


Diff: https://reviews.apache.org/r/71608/diff/4/

Changes: https://reviews.apache.org/r/71608/diff/3-4/


Testing
---


Thanks,

Qian Zhang

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-15 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/#review218220
---



Patch looks great!

Reviews applied: [71608]

Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose --disable-libtool-wrappers 
--disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; 
./support/docker-build.sh

- Mesos Reviewbot


On Oct. 15, 2019, 2:18 p.m., Qian Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71608/
> ---
> 
> (Updated Oct. 15, 2019, 2:18 p.m.)
> 
> 
> Review request for mesos and Gilbert Song.
> 
> 
> Bugs: MESOS-9964
> https://issues.apache.org/jira/browse/MESOS-9964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Previously in MESOS-3736, we made Docker store support pulling same
> image simultaneously which is a performance improvement, however it
> may cause an issue: If the pulling hangs somehow, all the subsequent
> pulling request for the same image will hang as well, and as a result
> the container destroy will also hang since destroy has to wait for
> provisioning to finish, see MESOS-4985 for details.
> 
> So in this patch we removed that performance improvement and made UCR
> can destroy the container which is being provisioned, i.e., UCR will
> discard the container provisioning and then keep doing the container
> destroy. And we also improved Docker fetcher plugin so that when
> container provisioning is discarded the `curl` process used to fetch
> manifest or blob will be killed immediately.
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 6537f6550b353b1c65a30381f6d68f61508d4960 
>   src/slave/containerizer/mesos/containerizer.cpp 
> c61b954635ebcaed3e498452bddeee6e1b34f388 
>   src/slave/containerizer/mesos/provisioner/docker/store.cpp 
> 286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
>   src/slave/containerizer/mesos/provisioner/provisioner.hpp 
> 38664177ae3805041b412ed52dd0134a6f3aa679 
>   src/slave/containerizer/mesos/provisioner/provisioner.cpp 
> 3d0b291fa878ab22625435afe2a219d6f776e52c 
>   src/tests/containerizer/mesos_containerizer_tests.cpp 
> 449928c10b897061642af8ad267f8b70695940e6 
>   src/tests/containerizer/provisioner_docker_tests.cpp 
> 5d5a355afd9c4fda1c653d6cecb75703b0fe 
>   src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 
> 
> 
> Diff: https://reviews.apache.org/r/71608/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>

Re: Review Request 71608: Supported destroying UCR container in `PROVISIONING` state.

2019-10-15 Thread Qian Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71608/
---

(Updated Oct. 15, 2019, 10:18 p.m.)


Review request for mesos and Gilbert Song.


Changes
---

Fixed/removed the failed tests.


Summary (updated)
-

Supported destroying UCR container in `PROVISIONING` state.


Bugs: MESOS-9964
https://issues.apache.org/jira/browse/MESOS-9964


Repository: mesos


Description (updated)
---

Previously in MESOS-3736, we made Docker store support pulling same
image simultaneously which is a performance improvement, however it
may cause an issue: If the pulling hangs somehow, all the subsequent
pulling request for the same image will hang as well, and as a result
the container destroy will also hang since destroy has to wait for
provisioning to finish, see MESOS-4985 for details.

So in this patch we removed that performance improvement and made UCR
can destroy the container which is being provisioned, i.e., UCR will
discard the container provisioning and then keep doing the container
destroy. And we also improved Docker fetcher plugin so that when
container provisioning is discarded the `curl` process used to fetch
manifest or blob will be killed immediately.


Diffs (updated)
-

  src/slave/containerizer/mesos/containerizer.hpp 
6537f6550b353b1c65a30381f6d68f61508d4960 
  src/slave/containerizer/mesos/containerizer.cpp 
c61b954635ebcaed3e498452bddeee6e1b34f388 
  src/slave/containerizer/mesos/provisioner/docker/store.cpp 
286ee5f00933b4c86a96dc4e10e42f9e7eac0ce2 
  src/slave/containerizer/mesos/provisioner/provisioner.hpp 
38664177ae3805041b412ed52dd0134a6f3aa679 
  src/slave/containerizer/mesos/provisioner/provisioner.cpp 
3d0b291fa878ab22625435afe2a219d6f776e52c 
  src/tests/containerizer/mesos_containerizer_tests.cpp 
449928c10b897061642af8ad267f8b70695940e6 
  src/tests/containerizer/provisioner_docker_tests.cpp 
5d5a355afd9c4fda1c653d6cecb75703b0fe 
  src/uri/fetchers/docker.cpp 8f5fc964f056b349ce57ced139e07f538cb1cfd2 


Diff: https://reviews.apache.org/r/71608/diff/3/

Changes: https://reviews.apache.org/r/71608/diff/2-3/


Testing
---


Thanks,

Qian Zhang