foreverneverer opened a new issue, #1092: URL: https://github.com/apache/incubator-pegasus/issues/1092
## Bug Report https://github.com/XiaoMi/rdsn/pull/979 upgrade zk c client to 3.7(from 3.4.10) for kerberos, but it sometime can't access zk server 3.4.5 In my test, when I upgrade the meta to latest pegasus server, the meta error log: ``` E2022-07-27 11:41:16.67 (1658893276067532317 139122) meta.default0.00001f3100010001: meta_service.cpp:463:start(): initialize server state from remote storage failed, err = ERR_INCONSISTENT_STATE, retry ... F2022-07-27 11:41:16.69 (1658893276069053850 139122) meta.default0.00001f3100010001: server_state.cpp:709:sync_apps_from_remote_storage(): assertion expression: ERR_OK == err F2022-07-27 11:41:16.69 (1658893276069061403 139122) meta.default0.00001f3100010001: server_state.cpp:709:sync_apps_from_remote_storage(): can't handle this error (ERR_TIMEOUT) ``` `ERR_INCONSISTENT_STATE` is from zk error `ZRUNTIMEINCONSISTENCY`, see the zk error log: ``` 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1250: Client environment:zookeeper.version=zookeeper C client 3.7.0 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1254: Client environment:host.name=host 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1261: Client environment:os.name=Linux 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1262: Client environment:os.arch=3.18.6-2.el7.centos.x86_64 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1263: Client environment:os.version=#1 SMP Mon Oct 24 13:01:33 CST 2016 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1271: Client environment:user.name=(null) 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1279: Client environment:user.home=/home/work 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@log_env@1291: Client environment:user.dir=/home/work/app/pegasus/tjwqtst-msg/meta 2022-07-28 16:25:15,412:63056(0x7f2f8d699700):ZOO_INFO@zookeeper_init_internal@1344: Initiating client connection, host=host:21000 sessionTimeout=10000 watcher=0x7f2fa97f1fc0 sessionId=0 sessionPasswd=<null> context=0x2082790 flags=0 2022-07-28 16:25:15,449:63056(0x7f2f7c652700):ZOO_INFO@check_events@2988: initiated connection to server IP:21000 2022-07-28 16:25:15,467:63056(0x7f2f7c652700):ZOO_INFO@finalize_session_establishment@2868: session establishment complete on server IP:21000, sessionId=0xff82392b354205db, negotiated timeout=10000 2022-07-28 16:25:47,814:63056(0x7f2f7c652700):ZOO_WARN@zookeeper_interest@2532: Exceeded deadline by 1021ms 2022-07-28 16:26:13,404:63056(0x7f2f7c652700):ZOO_WARN@zookeeper_interest@2532: Exceeded deadline by 1021ms 2022-07-28 16:26:29,014:63056(0x7f2f7c652700):ZOO_ERROR@resolve_hosts@947: getaddrinfo: Name or service not known ``` The above bug does not always occur, I only find the bug in one cluster of our company, The following resolution will be ok: - revert zk client version to 3.4.10 - change zk config from host to ip ## Related jira: https://issues.apache.org/jira/browse/ZOOKEEPER-4603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
