Jie Yu created MESOS-2914:
-----------------------------
Summary: Port mapping isolator should cleanup unknown orphan
containers after all known orphan containers are recovered during recovery.
Key: MESOS-2914
URL: https://issues.apache.org/jira/browse/MESOS-2914
Project: Mesos
Issue Type: Bug
Reporter: Jie Yu
Otherwise, the icmp/arp filter on host eth0 might be removed as a result of
_cleanup if 'infos' is empty, causing subsequent '_cleanup' to fail on both
known/unknown orphan containers.
{noformat}
I0612 17:46:51.518501 16308 containerizer.cpp:314] Recovering containerizer
I0612 17:46:51.520612 16308 port_mapping.cpp:1567] Discovered network namespace
handle symlink ddcb8397-3552-44f9-bc99-b5b69aa72944 -> 31607
I0612 17:46:51.521183 16308 port_mapping.cpp:1567] Discovered network namespace
handle symlink d8c48a4a-fdfb-47dd-b8d8-07188c21600d -> 41020
I0612 17:46:51.521883 16308 port_mapping.cpp:1567] Discovered network namespace
handle symlink 8953fc7f-9fca-4931-b0cb-2f4959ddee74 -> 3302
I0612 17:46:51.522542 16308 port_mapping.cpp:1567] Discovered network namespace
handle symlink 50f9986f-ebbc-440d-86a7-9fa1a7c55a75 -> 19805
I0612 17:46:51.523643 16308 port_mapping.cpp:2597] Removing IP packet filters
with ports [33792,34815] for container with pid 52304
I0612 17:46:51.525063 16308 port_mapping.cpp:2616] Freed ephemeral ports
[33792,34816) for container with pid 52304
I0612 17:46:51.547696 16308 port_mapping.cpp:2762] Successfully performed
cleanup for pid 52304
I0612 17:46:51.550027 16308 port_mapping.cpp:1698] Network isolator recovery
complete
I0612 17:46:51.550946 16329 containerizer.cpp:449] Removing orphan container
111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.552686 16329 containerizer.cpp:449] Removing orphan container
ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.552734 16309 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.554932 16329 containerizer.cpp:449] Removing orphan container
8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:46:51.555032 16309 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.555629 16308 cgroups.cpp:1420] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 after
1.730304ms
I0612 17:46:51.557507 16329 containerizer.cpp:449] Removing orphan container
50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:46:51.557611 16309 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:46:51.557896 16313 cgroups.cpp:1420] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 after
1.685248ms
I0612 17:46:51.559412 16310 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.561564 16329 containerizer.cpp:449] Removing orphan container
d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.562489 16315 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:46:51.562988 16313 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.563303 16310 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 after
2.076928ms
I0612 17:46:51.566052 16308 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.566102 16313 slave.cpp:3911] Finished recovery
W0612 17:46:51.566432 16323 disk.cpp:299] Ignoring cleanup for unknown
container 111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.566651 16317 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 after
2.12096ms
I0612 17:46:51.566987 16313 slave.cpp:3944] Garbage collecting old slave
20150319-213133-2080910346-5050-57551-S3314
I0612 17:46:51.567777 16318 cgroups.cpp:1420] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d after
1.323008ms
W0612 17:46:51.568042 16323 port_mapping.cpp:2544] Ignoring cleanup for unknown
container 111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.569522 16311 gc.cpp:56] Scheduling
'/var/lib/mesos/slaves/20150319-213133-2080910346-5050-57551-S3314' for gc
6.99999341503407days in the future
W0612 17:46:51.569725 16329 disk.cpp:299] Ignoring cleanup for unknown
container ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.570911 16325 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.573581 16316 port_mapping.cpp:2597] Removing IP packet filters
with ports [35840,36863] for container with pid 31607
I0612 17:46:51.575127 16316 port_mapping.cpp:2616] Freed ephemeral ports
[35840,36864) for container with pid 31607
I0612 17:46:51.588284 16330 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d after
14.503936ms
E0612 17:46:51.622140 16310 containerizer.cpp:480] Failed to clean up an
isolator when destroying orphan container ddcb8397-3552-44f9-bc99-b5b69aa72944:
The ICMP packet filter on host eth0 does not exist, The ARP packet filter on
host eth0 does not exist
W0612 17:46:51.773123 16313 disk.cpp:299] Ignoring cleanup for unknown
container d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.774153 16325 port_mapping.cpp:2597] Removing IP packet filters
with ports [32768,33791] for container with pid 41020
I0612 17:46:51.775167 16325 port_mapping.cpp:2616] Freed ephemeral ports
[32768,33792) for container with pid 41020
E0612 17:46:51.817221 16323 containerizer.cpp:480] Failed to clean up an
isolator when destroying orphan container d8c48a4a-fdfb-47dd-b8d8-07188c21600d:
The ICMP packet filter on host eth0 does not exist, The ARP packet filter on
host eth0 does not exist
I0612 17:46:51.872231 16314 cgroups.cpp:1420] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 after
308.33792ms
I0612 17:46:51.874572 16314 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:46:51.876566 16314 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 after
1.593344ms
2015-06-12
17:46:54,833:16307(0x7f172eb07940):ZOO_INFO@auth_completion_func@1286:
Authentication scheme digest succeeded
I0612 17:46:54.835737 16321 group.cpp:385] Trying to create path
'/home/mesos/prod/master' in ZooKeeper
I0612 17:46:54.839110 16321 detector.cpp:138] Detected a new leader: (id='1')
I0612 17:46:54.840276 16330 group.cpp:659] Trying to get
'/home/mesos/prod/master/info_0000000001' in ZooKeeper
I0612 17:46:54.842350 16330 detector.cpp:452] A new leading master
([email protected]:5050) is detected
I0612 17:46:54.843297 16330 slave.cpp:653] New master detected at
[email protected]:5050
I0612 17:46:54.843298 16312 status_update_manager.cpp:171] Pausing sending
status updates
I0612 17:46:54.844091 16330 slave.cpp:678] No credentials provided. Attempting
to register without authentication
I0612 17:46:54.845087 16330 slave.cpp:689] Detecting new master
I0612 17:47:01.561920 16309 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:01.564687 16309 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after
1.924096ms
I0612 17:47:01.565467 16309 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
W0612 17:47:09.978946 16326 disk.cpp:299] Ignoring cleanup for unknown
container 50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:47:09.979818 16327 port_mapping.cpp:2597] Removing IP packet filters
with ports [34816,35839] for container with pid 19805
I0612 17:47:09.981474 16327 port_mapping.cpp:2616] Freed ephemeral ports
[34816,35840) for container with pid 19805
E0612 17:47:10.278715 16325 containerizer.cpp:480] Failed to clean up an
isolator when destroying orphan container 50f9986f-ebbc-440d-86a7-9fa1a7c55a75:
The ICMP packet filter on host eth0 does not exist, The ARP packet filter on
host eth0 does not exist
I0612 17:47:11.568151 16326 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:11.570915 16326 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after
1.987072ms
I0612 17:47:11.571728 16326 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:14.549536 16316 slave.cpp:821] Registered with master
[email protected]:5050; given slave ID
20150602-190100-2215521290-5050-39399-S23257
I0612 17:47:14.550220 16318 status_update_manager.cpp:178] Resuming sending
status updates
I0612 17:47:21.574513 16319 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:21.576817 16319 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after
1.587968ms
I0612 17:47:21.577466 16319 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:31.580281 16310 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:31.582365 16310 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after
1.410048ms
I0612 17:47:31.582895 16310 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:41.585619 16322 cgroups.cpp:2394] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:41.587703 16322 cgroups.cpp:1449] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after
1.418752ms
I0612 17:47:41.588436 16322 cgroups.cpp:2377] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:51.515689 16330 slave.cpp:3733] Current disk usage 11.43%. Max
allowed age: 5.499938588861215days
E0612 17:47:51.557831 16330 containerizer.cpp:468] Failed to destroy orphan
container 8953fc7f-9fca-4931-b0cb-2f4959ddee74: Timed out after 1mins
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
