[ https://issues.apache.org/jira/browse/MESOS-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194995#comment-16194995 ]
James Peach commented on MESOS-5309: ------------------------------------ On my Fedora 26 system, this reliably fails with: {noformat} + mount --make-rslave /run/netns + test -f /proc/sys/net/ipv6/conf/all/disable_ipv6 + echo 1 + ip link set lo address 00:3e:e1:c8:84:d1 mtu 1500 up + ethtool -K enp12s0 rx off + ip link set enp12s0 address 00:3e:e1:c8:84:d1 up + ip addr add 17.228.224.108/24 dev enp12s0 + ip route add default via 17.228.224.1 + echo 30016 30031 + echo 1 + echo 1 + echo 1 + '[' -f /proc/sys/net/ipv4/tcp_keepalive_time ']' + echo 7200 + '[' -f /proc/sys/net/core/rmem_max ']' + '[' -f /proc/sys/net/ipv4/tcp_keepalive_intvl ']' + echo 75 + '[' -f /proc/sys/net/core/somaxconn ']' + echo 128 + '[' -f /proc/sys/net/core/wmem_max ']' + '[' -f /proc/sys/net/core/netdev_max_backlog ']' + '[' -f /proc/sys/net/ipv4/tcp_keepalive_probes ']' + echo 9 + '[' -f /proc/sys/net/ipv4/tcp_max_syn_backlog ']' + echo 2048 + '[' -f /proc/sys/net/ipv4/neigh/default/gc_thresh2 ']' + '[' -f /proc/sys/net/ipv4/neigh/default/gc_thresh3 ']' + '[' -f /proc/sys/net/ipv4/tcp_wmem ']' + '[' -f /proc/sys/net/ipv4/neigh/default/gc_thresh1 ']' + '[' -f /proc/sys/net/ipv4/tcp_synack_retries ']' + echo 5 + '[' -f /proc/sys/net/ipv4/tcp_rmem ']' + '[' -f /proc/sys/net/ipv4/tcp_retries2 ']' + echo 15 + tc qdisc add dev lo ingress + tc qdisc add dev enp12s0 ingress + tc filter add dev lo parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 17.228.224.108 action mirred egress redirect dev enp12s0 + tc filter add dev lo parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 action mirred egress redirect dev enp12s0 + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 30016 fff0 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 30016 fff0 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31000 fff8 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31000 fff8 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31008 ffe0 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31008 ffe0 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31040 ffc0 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31040 ffc0 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31104 ff80 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31104 ff80 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31232 ff00 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31232 ff00 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31488 fff8 + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31488 fff8 action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31496 fffc + tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31496 fffc action mirred egress redirect dev lo + tc filter add dev lo parent ffff:0 protocol ip prio 514 u32 flowid ffff:0 match ip protocol 1 0xff match ip dst 17.228.224.108 + tc filter add dev lo parent ffff:0 protocol ip prio 514 u32 flowid ffff:0 match ip protocol 1 0xff match ip dst 127.0.0.1 + tc filter show dev enp12s0 parent ffff:0 filter protocol ip pref 770 u32 filter protocol ip pref 770 u32 fh 800: ht divisor 1 filter protocol ip pref 770 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007540/0000fff0 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 3 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007918/0000fff8 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 4 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::802 order 2050 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007920/0000ffe0 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 5 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::803 order 2051 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007940/0000ffc0 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 6 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::804 order 2052 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007980/0000ff80 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 7 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::805 order 2053 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007a00/0000ff00 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 8 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::806 order 2054 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007b00/0000fff8 at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 9 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::807 order 2055 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 match 00007b08/0000fffc at 20 action order 1: mirred (Egress Redirect to device lo) stolen index 10 ref 1 bind 1 + tc filter show dev lo parent ffff:0 filter protocol ip pref 514 u32 filter protocol ip pref 514 u32 fh 802: ht divisor 1 filter protocol ip pref 514 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid ffff: match 00010000/00ff0000 at 8 match 11e4e06c/ffffffff at 16 filter protocol ip pref 514 u32 fh 802::801 order 2049 key ht 802 bkt 0 flowid ffff: match 00010000/00ff0000 at 8 match 7f000001/ffffffff at 16 filter protocol ip pref 769 u32 filter protocol ip pref 769 u32 fh 801: ht divisor 1 filter protocol ip pref 769 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid ffff: match 00007540/0000fff0 at 20 filter protocol ip pref 769 u32 fh 801::801 order 2049 key ht 801 bkt 0 flowid ffff: match 00007918/0000fff8 at 20 filter protocol ip pref 769 u32 fh 801::802 order 2050 key ht 801 bkt 0 flowid ffff: match 00007920/0000ffe0 at 20 filter protocol ip pref 769 u32 fh 801::803 order 2051 key ht 801 bkt 0 flowid ffff: match 00007940/0000ffc0 at 20 filter protocol ip pref 769 u32 fh 801::804 order 2052 key ht 801 bkt 0 flowid ffff: match 00007980/0000ff80 at 20 filter protocol ip pref 769 u32 fh 801::805 order 2053 key ht 801 bkt 0 flowid ffff: match 00007a00/0000ff00 at 20 filter protocol ip pref 769 u32 fh 801::806 order 2054 key ht 801 bkt 0 flowid ffff: match 00007b00/0000fff8 at 20 filter protocol ip pref 769 u32 fh 801::807 order 2055 key ht 801 bkt 0 flowid ffff: match 00007b08/0000fffc at 20 filter protocol ip pref 770 u32 filter protocol ip pref 770 u32 fh 800: ht divisor 1 filter protocol ip pref 770 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid ffff: match 11e4e06c/ffffffff at 16 action order 1: mirred (Egress Redirect to device enp12s0) stolen index 1 ref 1 bind 1 filter protocol ip pref 770 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid ffff: match 7f000001/ffffffff at 16 action order 1: mirred (Egress Redirect to device enp12s0) stolen index 2 ref 1 bind 1 ../../src/tests/containerizer/port_mapping_tests.cpp:588: Failure (isolator.get()->isolate(containerId2, pid.get())).failure(): Failed to append a ICMP mirror action from host enp12s0 to mesos24638: Invalid input data or parameter {noformat} > PortMappingIsolatorTest.ROOT_NC_ContainerToContainerTCP failed. > --------------------------------------------------------------- > > Key: MESOS-5309 > URL: https://issues.apache.org/jira/browse/MESOS-5309 > Project: Mesos > Issue Type: Bug > Components: containerization > Environment: Fedora 23 with network isolator enabled > Reporter: Gilbert Song > Labels: isolation > > Here is the log: > {code} > [01:22:18] : [ RUN ] > PortMappingIsolatorTest.ROOT_NC_ContainerToContainerTCP > [01:22:18]W: I0429 01:22:18.416817 24850 port_mapping_tests.cpp:229] Using > eth0 as the public interface > [01:22:18]W: I0429 01:22:18.417135 24850 port_mapping_tests.cpp:237] Using lo > as the loopback interface > [01:22:18]W: I0429 01:22:18.429095 24850 resources.cpp:572] Parsing resources > as JSON failed: > cpus:2;mem:1024;disk:1024;ephemeral_ports:[30001-30999];ports:[31000-32000] > [01:22:18]W: Trying semicolon-delimited string format instead > [01:22:18]W: I0429 01:22:18.430194 24850 port_mapping.cpp:1557] Using eth0 as > the public interface > [01:22:18]W: I0429 01:22:18.430490 24850 port_mapping.cpp:1582] Using lo as > the loopback interface > [01:22:18]W: I0429 01:22:18.431619 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/neigh/default/gc_thresh3 = '1024' > [01:22:18]W: I0429 01:22:18.431668 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/neigh/default/gc_thresh1 = '128' > [01:22:18]W: I0429 01:22:18.431723 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_wmem = '4096 16384 4194304' > [01:22:18]W: I0429 01:22:18.431761 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_synack_retries = '5' > [01:22:18]W: I0429 01:22:18.431797 24850 port_mapping.cpp:1869] > /proc/sys/net/core/rmem_max = '212992' > [01:22:18]W: I0429 01:22:18.431830 24850 port_mapping.cpp:1869] > /proc/sys/net/core/somaxconn = '128' > [01:22:18]W: I0429 01:22:18.431864 24850 port_mapping.cpp:1869] > /proc/sys/net/core/wmem_max = '212992' > [01:22:18]W: I0429 01:22:18.431900 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_rmem = '4096 87380 6291456' > [01:22:18]W: I0429 01:22:18.431933 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_keepalive_time = '7200' > [01:22:18]W: I0429 01:22:18.431967 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/neigh/default/gc_thresh2 = '512' > [01:22:18]W: I0429 01:22:18.432001 24850 port_mapping.cpp:1869] > /proc/sys/net/core/netdev_max_backlog = '1000' > [01:22:18]W: I0429 01:22:18.432036 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_keepalive_intvl = '75' > [01:22:18]W: I0429 01:22:18.432070 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_keepalive_probes = '9' > [01:22:18]W: I0429 01:22:18.432101 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_max_syn_backlog = '512' > [01:22:18]W: I0429 01:22:18.432134 24850 port_mapping.cpp:1869] > /proc/sys/net/ipv4/tcp_retries2 = '15' > [01:22:18]W: F0429 01:22:18.432205 24850 port_mapping_tests.cpp:448] > CHECK_SOME(isolator): Failed to get realpath for bind mount root > '/var/run/netns': Not found > [01:22:18]W: *** Check failure stack trace: *** > [01:22:18]W: @ 0x7fc8dccfc986 google::LogMessage::Fail() > [01:22:18]W: @ 0x7fc8dccfc8df google::LogMessage::SendToLog() > [01:22:18]W: @ 0x7fc8dccfc2d5 google::LogMessage::Flush() > [01:22:18]W: @ 0x7fc8dccff146 > google::LogMessageFatal::~LogMessageFatal() > [01:22:18]W: @ 0xa6031d _CheckFatal::~_CheckFatal() > [01:22:18]W: @ 0x1890d61 > mesos::internal::tests::PortMappingIsolatorTest_ROOT_NC_ContainerToContainerTCP_Test::TestBody() > [01:22:18]W: @ 0x19437a8 > testing::internal::HandleSehExceptionsInMethodIfSupported<>() > [01:22:18]W: @ 0x193e81a > testing::internal::HandleExceptionsInMethodIfSupported<>() > [01:22:18]W: @ 0x191f264 testing::Test::Run() > [01:22:18]W: @ 0x191fa1c testing::TestInfo::Run() > [01:22:18]W: @ 0x192006d testing::TestCase::Run() > [01:22:18]W: @ 0x1926bab > testing::internal::UnitTestImpl::RunAllTests() > [01:22:18]W: @ 0x194446f > testing::internal::HandleSehExceptionsInMethodIfSupported<>() > [01:22:18]W: @ 0x193f35a > testing::internal::HandleExceptionsInMethodIfSupported<>() > [01:22:18]W: @ 0x1925887 testing::UnitTest::Run() > [01:22:18]W: @ 0xf9131d RUN_ALL_TESTS() > [01:22:18]W: @ 0xf90f15 main > [01:22:18]W: @ 0x7fc8d68d8580 __libc_start_main > [01:22:18]W: @ 0xa5e919 _start > [01:22:19]W: /mnt/teamcity/temp/agentTmp/custom_script1282998915150293546: > line 3: 24850 Aborted (core dumped) GLOG_v=1 > ./bin/mesos-tests.sh --verbose --gtest_filter="$GTEST_FILTER" > [01:22:19]W: Process exited with code 134 > [01:22:19]i: ##teamcity[buildStatisticValue > key='buildStageDuration:buildStepRUNNER_299' value='815124.0'] > [01:22:19]E: Step Run tests (Command Line) failed > [01:22:19]i: ##teamcity[buildStatisticValue > key='buildStageDuration:buildFinishing' value='8.0'] > [01:22:19] : Publishing internal artifacts > [01:22:19] : Publishing 1 file using [WebPublisher] > [01:22:19] : Publishing 1 file using [ArtifactsCachePublisher] > [01:22:19]i: ##teamcity[buildStatisticValue > key='buildStageDuration:artifactsPublishing' value='49.0'] > [01:22:20] : Build finished > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)