huangds3527 opened a new issue, #2217: URL: https://github.com/apache/incubator-hugegraph/issues/2217
### Bug Type (问题类型) server status (启动/运行异常) ### Before submit - [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents) ### Environment (环境信息) - Server Version: 1.0.0 (Apache Release Version) - Backend: RocksDB 3 nodes, HDD or SSD - OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x - Data Size: 6500万 vertices, 18亿 edges ### Expected & Actual behavior (期望与实际表现) 启动失败 Starting HugeGraphServer... Connecting to HugeGraphServer (http://0.0.0.0:8080/graphs)......Starting HugeGraphServer failed See /opt/hugegraph/hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log output. 查看日志后报错Failed to decompress snapshot tar java.lang.NullPointerException: null 2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [INFO] o.a.h.b.s.r.StoreSnapshotFile - The snapshot tar files to be loaded are [snapshot_rocksdb-data.zip] 2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [ERROR] o.a.h.b.s.r.StoreSnapshotFile - Failed to decompress snapshot tar java.lang.NullPointerException: null at org.apache.hugegraph.backend.store.raft.StoreSnapshotFile.decompressSnapshot(StoreSnapshotFile.java:212) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.backend.store.raft.StoreSnapshotFile.load(StoreSnapshotFile.java:123) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.backend.store.raft.StoreStateMachine.onSnapshotLoad(StoreStateMachine.java:209) ~[hugegraph-core-1.0.0.jar:1.0.0] at com.alipay.sofa.jraft.core.FSMCallerImpl.doSnapshotLoad(FSMCallerImpl.java:690) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:406) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:72) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:149) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:141) ~[jraft-core-1.3.11.jar:?] at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) [disruptor-3.3.7.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] 2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [WARN] c.a.s.j.c.NodeImpl - Node <hugegraph/192.168.30.31:8091> got error: Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>: StateMachine onSnapshotLoad failed]]. 2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [WARN] c.a.s.j.c.FSMCallerImpl - FSMCaller already in error status, ignore new error. com.alipay.sofa.jraft.error.RaftException: StateMachine onSnapshotLoad failed at com.alipay.sofa.jraft.core.FSMCallerImpl.doSnapshotLoad(FSMCallerImpl.java:692) [jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:406) [jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:72) [jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:149) [jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:141) [jraft-core-1.3.11.jar:?] at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) [disruptor-3.3.7.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] 2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [ERROR] o.a.h.b.s.r.StoreStateMachine - Raft error: StateMachine onSnapshotLoad failed com.alipay.sofa.jraft.error.RaftException: StateMachine onSnapshotLoad failed at com.alipay.sofa.jraft.core.FSMCallerImpl.doSnapshotLoad(FSMCallerImpl.java:692) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:406) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:72) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:149) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:141) ~[jraft-core-1.3.11.jar:?] at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) [disruptor-3.3.7.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] 2023-05-24 18:07:39 [main] [ERROR] c.a.s.j.s.s.SnapshotExecutorImpl - Fail to load snapshot from ./raft-log/snapshot, FirstSnapshotLoadDone status is Status[UNKNOWN<-1>: StateMachine onSnapshotLoad failed]. 2023-05-24 18:07:39 [main] [ERROR] c.a.s.j.c.NodeImpl - Node <hugegraph/192.168.30.31:8091> initSnapshotStorage failed. 2023-05-24 18:07:39 [main] [INFO] o.a.h.StandardHugeGraph - Close graph standardhugegraph[hugegraph] 2023-05-24 18:07:39 [main] [INFO] o.a.h.b.s.r.RaftContext - Stop raft server: 192.168.30.31:8091 2023-05-24 18:07:39 [main] [ERROR] c.a.s.r.s.ServerFactory - RPC-020100018: 关闭 server 时出错,key=[8,091]. com.alipay.remoting.LifeCycleException: this component has closed at com.alipay.remoting.AbstractLifeCycle.shutdown(AbstractLifeCycle.java:41) ~[bolt-1.6.4.jar:?] at com.alipay.remoting.AbstractRemotingServer.shutdown(AbstractRemotingServer.java:110) ~[bolt-1.6.4.jar:?] at com.alipay.remoting.AbstractRemotingServer.stop(AbstractRemotingServer.java:84) ~[bolt-1.6.4.jar:?] at com.alipay.sofa.rpc.server.bolt.BoltServer.stop(BoltServer.java:166) ~[sofa-rpc-all-5.7.6.jar:5.7.6] at com.alipay.sofa.rpc.server.bolt.BoltServer.destroy(BoltServer.java:228) ~[sofa-rpc-all-5.7.6.jar:5.7.6] at com.alipay.sofa.rpc.server.ServerFactory.destroyServer(ServerFactory.java:159) [sofa-rpc-all-5.7.6.jar:5.7.6] at com.alipay.sofa.rpc.config.ServerConfig.destroy(ServerConfig.java:245) [sofa-rpc-all-5.7.6.jar:5.7.6] at org.apache.hugegraph.rpc.RpcServer.destroy(RpcServer.java:138) [hugegraph-rpc-1.0.0.jar:1.0.0] at org.apache.hugegraph.core.GraphManager.destroyRpcServer(GraphManager.java:320) [hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.core.GraphManager.close(GraphManager.java:266) [hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:132) [hugegraph-api-1.0.0.jar:0.69.0.0] at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49) [jersey-server-3.0.3.jar:?] at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56) [jersey-server-3.0.3.jar:?] at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711) [jersey-server-3.0.3.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330) [jersey-container-grizzly2-http-3.0.3.jar:?] at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376) [grizzly-http-server-3.0.1.jar:3.0.1] at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268) [grizzly-http-server-3.0.1.jar:3.0.1] at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245) [grizzly-http-server-3.0.1.jar:3.0.1] at org.apache.hugegraph.server.RestServer.start(RestServer.java:71) [hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.server.RestServer.start(RestServer.java:178) [hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:32) [hugegraph-dist-1.0.0.jar:1.0.0] at org.apache.hugegraph.dist.HugeGraphServer.<init>(HugeGraphServer.java:60) [hugegraph-dist-1.0.0.jar:1.0.0] at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:120) [hugegraph-dist-1.0.0.jar:1.0.0] 2023-05-24 18:07:39 [main] [INFO] o.a.h.r.RpcServer - RpcServer stop on port 8091 2023-05-24 18:07:39 [main] [ERROR] o.a.h.d.HugeGraphServer - HugeRestServer start error: java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at com.alipay.sofa.jraft.RaftServiceFactory.createAndInitRaftNode(RaftServiceFactory.java:48) ~[jraft-core-1.3.11.jar:?] at com.alipay.sofa.jraft.RaftGroupService.start(RaftGroupService.java:129) ~[jraft-core-1.3.11.jar:?] at org.apache.hugegraph.backend.store.raft.RaftNode.initRaftNode(RaftNode.java:279) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.backend.store.raft.RaftNode.<init>(RaftNode.java:61) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.backend.store.raft.RaftContext.initRaftNode(RaftContext.java:156) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.backend.store.raft.RaftBackendStoreProvider.waitReady(RaftBackendStoreProvider.java:152) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.StandardHugeGraph.waitReady(StandardHugeGraph.java:318) ~[hugegraph-core-1.0.0.jar:1.0.0] at org.apache.hugegraph.core.GraphManager.waitGraphsReady(GraphManager.java:387) ~[hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.core.GraphManager.init(GraphManager.java:111) ~[hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:130) ~[hugegraph-api-1.0.0.jar:0.69.0.0] at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49) ~[jersey-server-3.0.3.jar:?] at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56) ~[jersey-server-3.0.3.jar:?] at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711) ~[jersey-server-3.0.3.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330) ~[jersey-container-grizzly2-http-3.0.3.jar:?] at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376) ~[grizzly-http-server-3.0.1.jar:3.0.1] at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268) ~[grizzly-http-server-3.0.1.jar:3.0.1] at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245) ~[grizzly-http-server-3.0.1.jar:3.0.1] at org.apache.hugegraph.server.RestServer.start(RestServer.java:71) ~[hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.server.RestServer.start(RestServer.java:178) ~[hugegraph-api-1.0.0.jar:0.69.0.0] at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:32) ~[hugegraph-dist-1.0.0.jar:1.0.0] at org.apache.hugegraph.dist.HugeGraphServer.<init>(HugeGraphServer.java:60) ~[hugegraph-dist-1.0.0.jar:1.0.0] at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:120) ~[hugegraph-dist-1.0.0.jar:1.0.0] 2023-05-24 18:07:39 [main] [INFO] o.a.h.HugeFactory - HugeFactory shutdown 删除hugeserver目录下 rm -rf raft-log rocksdb-data/ 然后重新初始化 启动hugeserver能启动成功 ### Vertex/Edge example (问题点 / 边数据举例) _No response_ ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构) _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
