javeme commented on PR #1897:
URL:
https://github.com/apache/incubator-hugegraph/pull/1897#issuecomment-1147106735
rocksdb ci:
```java
2022-06-06 05:54:39 [JRaft-FSMCaller-Disruptor-0] [WARN] c.a.s.j.c.NodeImpl
- Node <hugegraph/127.0.0.1:8093> got error: Error
[type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
StateMachine meet critical error when applying one or more tasks since index=3,
Status[ESTATEMACHINE<10002>: StateMachine occurred critical error:
com.baidu.hugegraph.backend.BackendException: Failed to execute backend
command]]].
2022-06-06 05:54:39 [JRaft-FSMCaller-Disruptor-0] [WARN]
c.a.s.j.c.FSMCallerImpl - FSMCaller already in error status, ignore new error.
com.alipay.sofa.jraft.error.RaftException: null
at
com.alipay.sofa.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:157)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:148)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:73)
~[jraft-core-1.3.9.jar:?]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.onApply(StoreStateMachine.java:99)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:541)
[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:510)
[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:442)
[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:73)
[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:148)
[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:142)
[jraft-core-1.3.9.jar:?]
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137)
[disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
```
```java
Error: -06 05:54:39 [store-backend-executor1] [ERROR]
c.a.s.j.u.LogThreadPoolExecutor - Uncaught exception in pool:
store-backend-executor,
com.alipay.sofa.jraft.util.LogThreadPoolExecutor@2736577f[Running, pool size =
2, active threads = 1, queued tasks = 0, completed tasks = 1].
com.baidu.hugegraph.backend.BackendException: Failed to execute backend
command
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.lambda$onApplyFollower$1(StoreStateMachine.java:142)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_332]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: java.lang.RuntimeException: Too many listeners for: cache
at com.baidu.hugegraph.event.EventHub.call(EventHub.java:175)
~[hugegraph-common-2.1.2.jar:2.1.2.0]
at
com.baidu.hugegraph.backend.store.raft.RaftContext.notifyCache(RaftContext.java:332)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.RaftContext.updateCacheIfNeeded(RaftContext.java:310)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.applyCommand(StoreStateMachine.java:175)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.lambda$onApplyFollower$1(StoreStateMachine.java:138)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
... 4 more
```
```java
Error: -06 05:54:39 [store-backend-executor1] [ERROR]
c.b.h.b.s.r.StoreStateMachine - Failed to execute backend command: COMMIT_TX
java.lang.RuntimeException: Too many listeners for: cache
at com.baidu.hugegraph.event.EventHub.call(EventHub.java:175)
~[hugegraph-common-2.1.2.jar:2.1.2.0]
at
com.baidu.hugegraph.backend.store.raft.RaftContext.notifyCache(RaftContext.java:332)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.RaftContext.updateCacheIfNeeded(RaftContext.java:310)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.applyCommand(StoreStateMachine.java:175)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.lambda$onApplyFollower$1(StoreStateMachine.java:138)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_332]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Error: -06 05:54:39 [JRaft-FSMCaller-Disruptor-0] [ERROR]
c.b.h.b.s.r.StoreStateMachine - StateMachine occurred critical error
java.util.concurrent.ExecutionException:
com.baidu.hugegraph.backend.BackendException: Failed to execute backend command
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[?:1.8.0_332]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
~[?:1.8.0_332]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.onApply(StoreStateMachine.java:90)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:541)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:510)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:442)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:73)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:148)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:142)
~[jraft-core-1.3.9.jar:?]
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137)
[disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: com.baidu.hugegraph.backend.BackendException: Failed to execute
backend command
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.lambda$onApplyFollower$1(StoreStateMachine.java:142)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_332]
... 1 more
Caused by: java.lang.RuntimeException: Too many listeners for: cache
at com.baidu.hugegraph.event.EventHub.call(EventHub.java:175)
~[hugegraph-common-2.1.2.jar:2.1.2.0]
at
com.baidu.hugegraph.backend.store.raft.RaftContext.notifyCache(RaftContext.java:332)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.RaftContext.updateCacheIfNeeded(RaftContext.java:310)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.applyCommand(StoreStateMachine.java:175)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.lambda$onApplyFollower$1(StoreStateMachine.java:138)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_332]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_332]
... 1 more
Error: -06 05:54:39 [JRaft-FSMCaller-Disruptor-0] [ERROR]
c.b.h.b.s.r.StoreStateMachine - Raft error: null
com.alipay.sofa.jraft.error.RaftException: null
at
com.alipay.sofa.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:157)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:148)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:73)
~[jraft-core-1.3.9.jar:?]
at
com.baidu.hugegraph.backend.store.raft.StoreStateMachine.onApply(StoreStateMachine.java:99)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:541)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:510)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:442)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:73)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:148)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:142)
~[jraft-core-1.3.9.jar:?]
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137)
[disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
2022-06-06 05:54:39 [JRaft-FSMCaller-Disruptor-0] [INFO]
c.b.h.b.s.r.StoreStateMachine - The node 127.0.0.1:8093 abdicated from follower
2022-06-06 05:54:39 [server-info-db-worker-1] [WARN]
c.b.h.b.s.r.RaftBackendStore - Failed to execute query '`Query * from VERTEX
where id in [0x8673657276657233]`': Status[EPERM<1008>: Invalid state for
readIndex: STATE_ERROR.]
com.baidu.hugegraph.backend.BackendException: Failed to do raft read-index:
Status[EPERM<1008>: Invalid state for readIndex: STATE_ERROR.]
at
com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:235)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexResponseClosure.notifyFail(ReadOnlyServiceImpl.java:212)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexResponseClosure.run(ReadOnlyServiceImpl.java:168)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.NodeImpl.handleReadIndexRequest(NodeImpl.java:1509)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl.executeReadIndexEvents(ReadOnlyServiceImpl.java:234)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl.access$200(ReadOnlyServiceImpl.java:70)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexEventHandler.onEvent(ReadOnlyServiceImpl.java:133)
~[jraft-core-1.3.9.jar:?]
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexEventHandler.onEvent(ReadOnlyServiceImpl.java:116)
~[jraft-core-1.3.9.jar:?]
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137)
[disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
```
```java
Error: -06 05:54:39 [main] [ERROR] c.b.h.d.HugeGraphServer - HugeRestServer
start error:
com.baidu.hugegraph.HugeException: Failed to update/query server info:
java.util.concurrent.ExecutionException:
com.baidu.hugegraph.backend.BackendException: Failed to execute query: `Query *
from VERTEX where id in [0x8673657276657233]`
at
com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:298)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.task.ServerInfoManager.serverInfo(ServerInfoManager.java:311)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.task.ServerInfoManager.initServerInfo(ServerInfoManager.java:114)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.StandardHugeGraph.serverStarted(StandardHugeGraph.java:267)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.auth.HugeGraphAuthProxy.serverStarted(HugeGraphAuthProxy.java:675)
~[hugegraph-api-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.core.GraphManager.serverStarted(GraphManager.java:421)
~[hugegraph-api-0.13.0.jar:0.69.0.0]
at com.baidu.hugegraph.core.GraphManager.<init>(GraphManager.java:115)
~[hugegraph-api-0.13.0.jar:0.69.0.0]
at
com.baidu.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:130)
~[hugegraph-api-0.13.0.jar:0.69.0.0]
at
org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49)
~[jersey-server-3.0.3.jar:?]
at
org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56)
~[jersey-server-3.0.3.jar:?]
at
org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711)
~[jersey-server-3.0.3.jar:?]
at
org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330)
~[jersey-container-grizzly2-http-3.0.3.jar:?]
at
org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376)
~[grizzly-http-server-3.0.1.jar:3.0.1]
at
org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268)
~[grizzly-http-server-3.0.1.jar:3.0.1]
at
org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245)
~[grizzly-http-server-3.0.1.jar:3.0.1]
at com.baidu.hugegraph.server.RestServer.start(RestServer.java:73)
~[hugegraph-api-0.13.0.jar:0.69.0.0]
at com.baidu.hugegraph.server.RestServer.start(RestServer.java:180)
~[hugegraph-api-0.13.0.jar:0.69.0.0]
at
com.baidu.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:34)
~[hugegraph-dist-0.13.0.jar:?]
at
com.baidu.hugegraph.dist.HugeGraphServer.<init>(HugeGraphServer.java:62)
~[hugegraph-dist-0.13.0.jar:?]
at
com.baidu.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:121)
~[hugegraph-dist-0.13.0.jar:?]
Caused by: java.util.concurrent.ExecutionException:
com.baidu.hugegraph.backend.BackendException: Failed to execute query: `Query *
from VERTEX where id in [0x8673657276657233]`
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[?:1.8.0_332]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
~[?:1.8.0_332]
at
com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:295)
~[hugegraph-core-0.13.0.jar:0.13.0.0]
at
com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:295)
... 19 more
Caused by: com.baidu.hugegraph.backend.BackendException: Failed to execute
query: `Query * from VERTEX where id in [0x8673657276657233]`
at
com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:247)
at
com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:219)
at
com.baidu.hugegraph.backend.store.raft.RaftBackendStore.query(RaftBackendStore.java:154)
at
com.baidu.hugegraph.backend.tx.AbstractTransaction.query(AbstractTransaction.java:163)
at
com.baidu.hugegraph.backend.tx.GraphTransaction.query(GraphTransaction.java:532)
at
com.baidu.hugegraph.backend.tx.GraphTransaction.queryVerticesFromBackend(GraphTransaction.java:802)
at
com.baidu.hugegraph.backend.tx.GraphTransaction.queryVerticesByIds(GraphTransaction.java:750)
at
com.baidu.hugegraph.backend.tx.GraphTransaction.queryVertices(GraphTransaction.java:704)
at
com.baidu.hugegraph.task.ServerInfoManager.lambda$serverInfo$3(ServerInfoManager.java:312)
at
com.baidu.hugegraph.task.TaskManager$ContextCallable.call(TaskManager.java:354)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.baidu.hugegraph.backend.BackendException: Failed to do raft
read-index: Status[EPERM<1008>: Invalid state for readIndex: STATE_ERROR.]
at
com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:235)
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexResponseClosure.notifyFail(ReadOnlyServiceImpl.java:212)
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexResponseClosure.run(ReadOnlyServiceImpl.java:168)
at
com.alipay.sofa.jraft.core.NodeImpl.handleReadIndexRequest(NodeImpl.java:1509)
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl.executeReadIndexEvents(ReadOnlyServiceImpl.java:234)
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl.access$200(ReadOnlyServiceImpl.java:70)
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexEventHandler.onEvent(ReadOnlyServiceImpl.java:133)
at
com.alipay.sofa.jraft.core.ReadOnlyServiceImpl$ReadIndexEventHandler.onEvent(ReadOnlyServiceImpl.java:116)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137)
... 1 more
Error: Process completed with exit code 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
