liugddx opened a new issue, #4375: URL: https://github.com/apache/incubator-seatunnel/issues/4375
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened testStreamJobRestoreInAllNodeDown test case error refer to https://github.com/apache/incubator-seatunnel/actions/runs/4465230468/jobs/7842139684?pr=4140 ### SeaTunnel Version dev ### SeaTunnel Config ```conf - ``` ### Running Command ```shell - ``` ### Error Exception ```log 2023-03-20T06:18:55.1601934Z 2023-03-20 06:18:55,159 WARN com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector - [localhost]:5801 [runner_ClusterFaultToleranceIT_testStreamJobRestoreInAllNodeDown_1679293080166] [5.1] Slow operation detected: org.apache.seatunnel.engine.server.task.operation.GetTaskGroupAddressOperation 2023-03-20T06:18:55.1641178Z sun.misc.Unsafe.park(Native Method) 2023-03-20T06:18:55.1641764Z java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) 2023-03-20T06:18:55.1642507Z com.hazelcast.spi.impl.AbstractInvocationFuture.manageParking(AbstractInvocationFuture.java:693) 2023-03-20T06:18:55.1643307Z com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:615) 2023-03-20T06:18:55.1644120Z com.hazelcast.map.impl.query.QueryEngineImpl.addResultsOfPredicate(QueryEngineImpl.java:214) 2023-03-20T06:18:55.1644959Z com.hazelcast.map.impl.query.QueryEngineImpl.doRunOnQueryThreads(QueryEngineImpl.java:165) 2023-03-20T06:18:55.1645771Z com.hazelcast.map.impl.query.QueryEngineImpl.runOnGivenPartitions(QueryEngineImpl.java:141) 2023-03-20T06:18:55.1646534Z com.hazelcast.map.impl.query.QueryEngineImpl.execute(QueryEngineImpl.java:109) 2023-03-20T06:18:55.1647307Z com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1415) 2023-03-20T06:18:55.1648143Z com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1383) 2023-03-20T06:18:55.1648922Z com.hazelcast.map.impl.proxy.MapProxyImpl.executePredicate(MapProxyImpl.java:796) 2023-03-20T06:18:55.1649713Z com.hazelcast.map.impl.proxy.MapProxyImpl.keySet(MapProxyImpl.java:716) 2023-03-20T06:18:55.1650375Z com.hazelcast.map.impl.proxy.MapProxyImpl.keySet(MapProxyImpl.java:710) 2023-03-20T06:18:55.1656226Z org.apache.seatunnel.engine.server.master.JobMaster.queryTaskGroupAddress(JobMaster.java:404) 2023-03-20T06:18:55.1657840Z org.apache.seatunnel.engine.server.task.operation.GetTaskGroupAddressOperation.lambda$run$0(GetTaskGroupAddressOperation.java:55) 2023-03-20T06:18:55.1658905Z org.apache.seatunnel.engine.server.task.operation.GetTaskGroupAddressOperation$$Lambda$2167/226429161.execute(Unknown Source) 2023-03-20T06:18:55.1659784Z org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) 2023-03-20T06:18:55.1660881Z org.apache.seatunnel.engine.server.task.operation.GetTaskGroupAddressOperation.run(GetTaskGroupAddressOperation.java:51) 2023-03-20T06:18:55.1661763Z com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) 2023-03-20T06:18:55.1689960Z com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) 2023-03-20T06:18:55.1691321Z com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) 2023-03-20T06:18:55.1694872Z com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471) 2023-03-20T06:18:55.1695847Z com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197) 2023-03-20T06:18:55.1696822Z com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137) 2023-03-20T06:18:55.1697852Z com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) 2023-03-20T06:18:55.1700921Z com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) 2023-03-20T06:18:55.2304807Z 2023-03-20 06:18:55,230 WARN com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector - [localhost]:5802 [runner_ClusterFaultToleranceIT_testStreamJobRestoreInAllNodeDown_1679293080166] [5.1] Slow operation detected: org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation 2023-03-20T06:18:55.2306262Z sun.misc.Unsafe.park(Native Method) 2023-03-20T06:18:55.2307260Z java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) 2023-03-20T06:18:55.2308147Z com.hazelcast.spi.impl.AbstractInvocationFuture.manageParking(AbstractInvocationFuture.java:693) 2023-03-20T06:18:55.2309217Z com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:615) 2023-03-20T06:18:55.2310214Z org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.restoreState(SourceFlowLifeCycle.java:259) 2023-03-20T06:18:55.2311259Z org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$14(SeaTunnelTask.java:383) 2023-03-20T06:18:55.2372646Z org.apache.seatunnel.engine.server.task.SeaTunnelTask$$Lambda$2172/1542567937.accept(Unknown Source) 2023-03-20T06:18:55.2373260Z java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) 2023-03-20T06:18:55.2373777Z java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) 2023-03-20T06:18:55.2374313Z java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) 2023-03-20T06:18:55.2374858Z java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) 2023-03-20T06:18:55.2375423Z java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) 2023-03-20T06:18:55.2375997Z java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) 2023-03-20T06:18:55.2376555Z java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) 2023-03-20T06:18:55.2377103Z java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) 2023-03-20T06:18:55.2377650Z java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 2023-03-20T06:18:55.2378196Z java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) 2023-03-20T06:18:55.2378837Z org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:380) 2023-03-20T06:18:55.2379750Z org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:96) 2023-03-20T06:18:55.2380948Z org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation$$Lambda$2155/1874431408.execute(Unknown Source) 2023-03-20T06:18:55.2381766Z org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) 2023-03-20T06:18:55.2382626Z org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85) 2023-03-20T06:18:55.2383530Z com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) 2023-03-20T06:18:55.2384234Z com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) 2023-03-20T06:18:55.2385030Z com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) 2023-03-20T06:18:55.2385814Z com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471) 2023-03-20T06:18:55.2386585Z com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197) 2023-03-20T06:18:55.2387355Z com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137) 2023-03-20T06:18:55.2388143Z com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) 2023-03-20T06:18:55.2388922Z com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) 2023-03-20T06:18:55.2614760Z 2023-03-20 06:18:55,241 WARN com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector - [localhost]:5802 [runner_ClusterFaultToleranceIT_testStreamJobRestoreInAllNodeDown_1679293080166] [5.1] Slow operation detected: org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation (2 invocations) 2023-03-20T06:18:55.2616029Z sun.misc.Unsafe.park(Native Method) 2023-03-20T06:18:55.2616470Z java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) 2023-03-20T06:18:55.2617110Z com.hazelcast.spi.impl.AbstractInvocationFuture.manageParking(AbstractInvocationFuture.java:693) 2023-03-20T06:18:55.2617809Z (...) 2023-03-20T06:18:56.1368233Z 2023-03-20 06:18:56,136 DEBUG org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - start send heartbeat to resource manager, this address: [localhost]:5801 2023-03-20T06:18:56.1370502Z 2023-03-20 06:18:56,136 DEBUG org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received worker heartbeat from: [localhost]:5801 2023-03-20T06:18:56.1612541Z 2023-03-20 06:18:56,153 WARN com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector - [localhost]:5801 [runner_ClusterFaultToleranceIT_testStreamJobRestoreInAllNodeDown_1679293080166] [5.1] Slow operation detected: org.apache.seatunnel.engine.server.task.operation.GetTaskGroupAddressOperation (2 invocations) 2023-03-20T06:18:56.1614057Z sun.misc.Unsafe.park(Native Method) 2023-03-20T06:18:56.1615406Z java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) 2023-03-20T06:18:56.1616331Z com.hazelcast.spi.impl.AbstractInvocationFuture.manageParking(AbstractInvocationFuture.java:693) 2023-03-20T06:18:56.1617079Z (...) ``` ### Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
