[ https://issues.apache.org/jira/browse/MESOS-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761769#comment-16761769 ]
Vinod Kone commented on MESOS-8796: ----------------------------------- Saw this again on internal CI (on Mac). {code} [ RUN ] GroupTest.GroupPathWithRestrictivePerms I0205 21:14:33.530055 296834496 zookeeper_test_server.cpp:156] Started ZooKeeperTestServer on port 50946 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/ RELEASE_X86_64 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/bui ld 2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 s essionPasswd=<null> context=0x7fb3e0c9bc90 flags=0 2019-02-05 21:14:33,530:8369(0x700003fcf000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:33,532:8369(0x700003fcf000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90000, negotiated timeou t=10000 2019-02-05 21:14:36,875:8369(0x700003fcf000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/ RELEASE_X86_64 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/bui ld 2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 s essionPasswd=<null> context=0x7fb3e0a4db10 flags=0 2019-02-05 21:14:36,879:8369(0x700004767000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:36,880:8369(0x700004767000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90001, negotiated timeou t=10000 I0205 21:14:36.880167 55189504 group.cpp:341] Group process (zookeeper-group(48)@10.0.49.4:65013) connected to ZooKeeper I0205 21:14:36.880213 55189504 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0205 21:14:36.880225 55189504 group.cpp:395] Authenticating with ZooKeeper using digest 2019-02-05 21:14:40,222:8369(0x700004767000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded I0205 21:14:40.222224 55189504 group.cpp:419] Trying to create path '/read-only' in ZooKeeper 2019-02-05 21:14:40,223:8369(0x7000036ae000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/ RELEASE_X86_64 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/bui ld 2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 s essionPasswd=<null> context=0x7fb3e03c00c0 flags=0 I0205 21:14:40.224323 55189504 group.cpp:758] Found non-sequence node 'writable' at '/read-only' in ZooKeeper 2019-02-05 21:14:40,224:8369(0x70000486d000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:40,225:8369(0x70000486d000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90002, negotiated timeou t=10000 I0205 21:14:40.225809 57335808 group.cpp:341] Group process (zookeeper-group(49)@10.0.49.4:65013) connected to ZooKeeper I0205 21:14:40.225847 57335808 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0205 21:14:40.225859 57335808 group.cpp:395] Authenticating with ZooKeeper using digest 2019-02-05 21:14:49,320:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5764ms 2019-02-05 21:14:49,320:8369(0x700003fcf000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5766ms 2019-02-05 21:14:49,320:8369(0x700004767000):ZOO_ERROR@handle_socket_error_msg@1702: Socket [127.0.0.1:50946] zk retcode=-7, errno=60(Operation timed out): connection to 127.0.0.1:50946 timed out (exceeded timeout by 2430ms) 2019-02-05 21:14:49,321:8369(0x700003fcf000):ZOO_ERROR@handle_socket_error_msg@1702: Socket [127.0.0.1:50946] zk retcode=-7, errno=60(Operation timed out): connection to 127.0.0.1:50946 timed out (exceeded timeout by 2432ms) 2019-02-05 21:14:49,321:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5764ms 2019-02-05 21:14:49,321:8369(0x700003fcf000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5766ms I0205 21:14:49.321324 54652928 group.cpp:452] Lost connection to ZooKeeper, attempting to reconnect ... 2019-02-05 21:14:49,330:8369(0x70000486d000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5773ms 2019-02-05 21:14:49,330:8369(0x70000486d000):ZOO_ERROR@handle_socket_error_msg@1702: Socket [127.0.0.1:50946] zk retcode=-7, errno=60(Operation timed out): connection to 127.0.0.1:50946 timed out (exceeded timeout by 2438ms) 2019-02-05 21:14:49,330:8369(0x70000486d000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5773ms I0205 21:14:49.330642 57335808 group.cpp:452] Lost connection to ZooKeeper, attempting to reconnect ... 2019-02-05 21:14:52,664:8369(0x700003fcf000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 9109ms 2019-02-05 21:14:52,664:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 9107ms 2019-02-05 21:14:52,664:8369(0x70000486d000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 9106ms 2019-02-05 21:14:52,664:8369(0x700004767000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:52,664:8369(0x700003fcf000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:52,664:8369(0x70000486d000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:52,665:8369(0x700003fcf000):ZOO_ERROR@handle_socket_error_msg@1800: Socket [127.0.0.1:50946] zk retcode=-112, errno=70(Stale NFS file handle): sessionId=0x168c13aa8b90000 has expired. 2019-02-05 21:14:52,665:8369(0x700004767000):ZOO_ERROR@handle_socket_error_msg@1800: Socket [127.0.0.1:50946] zk retcode=-112, errno=70(Stale NFS file handle): sessionId=0x168c13aa8b90001 has expired. I0205 21:14:52.665789 56799232 group.cpp:511] ZooKeeper session expired 2019-02-05 21:14:52,665:8369(0x70000362b000):ZOO_INFO@zookeeper_close@2579: Freeing zookeeper resources for sessionId=0x168c13aa8b90001 2019-02-05 21:14:52,665:8369(0x70000486d000):ZOO_ERROR@handle_socket_error_msg@1800: Socket [127.0.0.1:50946] zk retcode=-112, errno=70(Stale NFS file handle): sessionId=0x168c13aa8b90002 has expired. 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8 I0205 21:14:52.666157 55189504 group.cpp:511] ZooKeeper session expired 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins 2019-02-05 21:14:52,666:8369(0x7000034a2000):ZOO_INFO@zookeeper_close@2579: Freeing zookeeper resources for sessionId=0x168c13aa8b90002 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/build 2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 sessionPasswd=<null> context=0x7fb3e0cad090 flags=0 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/: RELEASE_X86_64 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/build 2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 sessionPasswd=<null> context=0x7fb3e03c00c0 flags=0 2019-02-05 21:14:52,666:8369(0x700004767000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:52,666:8369(0x70000486d000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946] 2019-02-05 21:14:52,667:8369(0x700004767000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90003, negotiated timeout=10000 I0205 21:14:52.668120 54116352 group.cpp:341] Group process (zookeeper-group(48)@10.0.49.4:65013) connected to ZooKeeper I0205 21:14:52.668161 54116352 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0205 21:14:52.668203 54116352 group.cpp:395] Authenticating with ZooKeeper using digest 2019-02-05 21:14:52,668:8369(0x70000486d000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90004, negotiated timeout=10000 I0205 21:14:52.668565 54652928 group.cpp:341] Group process (zookeeper-group(49)@10.0.49.4:65013) connected to ZooKeeper I0205 21:14:52.668601 54652928 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0205 21:14:52.668615 54652928 group.cpp:395] Authenticating with ZooKeeper using digest ../../src/tests/group_tests.cpp:333: Failure Failed to wait 15secs for failedGroup2.join("fail") 2019-02-05 21:14:56,010:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 11ms 2019-02-05 21:14:56,010:8369(0x70000486d000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded 2019-02-05 21:14:56,010:8369(0x700004767000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded I0205 21:14:56.011027 54652928 group.cpp:419] Trying to create path '/read-only/new' in ZooKeeper I0205 21:14:56.011297 54116352 group.cpp:419] Trying to create path '/read-only' in ZooKeeper I0205 21:14:56.012171 54116352 group.cpp:758] Found non-sequence node 'writable' at '/read-only' in ZooKeeper E0205 21:14:56.012761 54652928 group.cpp:961] Group aborting: Failed to create '/read-only/new' in ZooKeeper: not authenticated 2019-02-05 21:14:56,012:8369(0x70000341f000):ZOO_INFO@zookeeper_close@2562: Closing zookeeper sessionId=0x168c13aa8b90004 to [127.0.0.1:50946] 2019-02-05 21:14:56,013:8369(0x111b155c0):ZOO_INFO@zookeeper_close@2562: Closing zookeeper sessionId=0x168c13aa8b90003 to [127.0.0.1:50946] 2019-02-05 21:14:56,013:8369(0x111b155c0):ZOO_INFO@zookeeper_close@2579: Freeing zookeeper resources for sessionId=0x168c13aa8b90000 I0205 21:14:56.013558 296834496 zookeeper_test_server.cpp:116] Shutting down ZooKeeperTestServer on port 50946 [ FAILED ] GroupTest.GroupPathWithRestrictivePerms (22486 ms) {code} > Some GroupTest.* are flaky on Mac. > ---------------------------------- > > Key: MESOS-8796 > URL: https://issues.apache.org/jira/browse/MESOS-8796 > Project: Mesos > Issue Type: Bug > Components: test > Affects Versions: 1.6.0, 1.8.0 > Environment: Mac OS with SSL enabled > Reporter: Alexander Rukletsov > Priority: Minor > Labels: flaky, flaky-test, zookeeper > Attachments: GroupJoinWithDisconnect-badrun.txt, > GroupPathWithRestrictivePerms-badrun.txt, > GroupPathWithRestrictivePerms-goodrun.txt, > GroupTest.GroupPathWithRest-another-bad-run.txt, RetryableErrors-badrun.txt > > > I see some failures related to zookeeper on our Mac machine. Current list of > failing tests: > {noformat} > GroupTest.GroupPathWithRestrictivePerms > GroupTest.RetryableErrors > {noformat} > Full logs attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)