cqutwangyu opened a new issue, #6333: URL: https://github.com/apache/seatunnel/issues/6333
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened I've seen similar problems but they haven't been solved. https://github.com/apache/seatunnel/issues/5442 I started a SeaTunnelServer process via seatunnel-cluster.sh. I ran the seatunnel streaming task for RocketMQ message forwarding in cluster mode in dolphinscheduler (DS). In DS, I set this task to be scheduled every minute, and the execution strategy is serial abandonment, so as to realize fast recovery after the seatunnel flow task fails. RocketMQ was out of service for some time during the run of the scheduled task, while DS continued to pull seatunnel once every minute for a longer time. After that, RocketMQ resumes normal service, but running any seatunnel task at this time will result in an OOM exception. ### SeaTunnel Version 2.3.3-release ### SeaTunnel Config ```conf env { parallelism = 4 job.mode = "STREAMING" } source { Rocketmq { name.srv.addr = "127.0.0.1:9876" topics = "seatunnel_source" } } transform { } sink { Rocketmq { name.srv.addr = "127.0.0.1:9876" topic = "seatunnel_sink" } } ``` ### Running Command ```shell sh bin/seatunnel.sh --config config/v2.batch.config.template -m cluster ``` ### Error Exception ```log Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/opt/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config' Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /opt/apache-seatunnel-2.3.3/config/seatunnel.yaml Feb 04, 2024 1:51:42 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig INFO: seatunnel.home is /opt/apache-seatunnel-2.3.3 Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/opt/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config' Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /opt/apache-seatunnel-2.3.3/config/hazelcast.yaml Feb 04, 2024 1:51:43 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/opt/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config' Feb 04, 2024 1:51:43 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /opt/apache-seatunnel-2.3.3/config/hazelcast-client.yaml 2024-02-04 13:51:43,544 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true 2024-02-04 13:51:43,612 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING 2024-02-04 13:51:43,617 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED 2024-02-04 13:51:43,644 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel 2024-02-04 13:51:43,648 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801 2024-02-04 13:51:43,683 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED 2024-02-04 13:51:43,683 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:382638e7-e321-4675-acea-247579799b4b, server version: 5.1, local address: /127.0.0.1:49198 2024-02-04 13:51:43,685 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. 2024-02-04 13:51:43,705 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] Members [1] { Member [localhost]:5801 - 382638e7-e321-4675-acea-247579799b4b } 2024-02-04 13:51:43,730 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds. 2024-02-04 13:51:43,879 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[] 2024-02-04 13:51:43,890 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: config/v2.batch.config.template 2024-02-04 13:51:43,956 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64] 2024-02-04 13:51:43,999 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: { "env" : { "execution.parallelism" : 2, "job.mode" : "BATCH", "checkpoint.interval" : 10000 }, "source" : [ { "schema" : { "fields" : { "name" : "string", "age" : "int" } }, "row.num" : 16, "parallelism" : 2, "result_table_name" : "fake", "plugin_name" : "FakeSource" } ], "sink" : [ { "plugin_name" : "Console" } ] } 2024-02-04 13:51:44,024 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,025 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,029 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /opt/apache-seatunnel-2.3.3/connectors/seatunnel 2024-02-04 13:51:44,035 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar 2024-02-04 13:51:44,036 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar 2024-02-04 13:51:44,039 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations. 2024-02-04 13:51:44,039 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,039 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,039 WARN org.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option. 2024-02-04 13:51:44,040 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources. 2024-02-04 13:51:44,041 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,094 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /opt/apache-seatunnel-2.3.3/connectors/seatunnel 2024-02-04 13:51:44,099 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar 2024-02-04 13:51:44,101 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath 2024-02-04 13:51:44,121 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms. 2024-02-04 13:51:44,122 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks. 2024-02-04 13:51:44,122 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,128 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-02-04 13:51:44,169 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 806401517069271041, with plugin jar [file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar] 2024-02-04 13:51:49,218 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN 2024-02-04 13:51:49,221 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:382638e7-e321-4675-acea-247579799b4b, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:49198->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-02-04 13:51:49.215, lastWriteTime=2024-02-04 13:51:48.739, closedTime=2024-02-04 13:51:49.219, connected server version=5.1} 2024-02-04 13:51:49,222 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2024-02-04 13:51:49,227 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN 2024-02-04 13:51:49,228 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client...... 2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - =============================================================================== 2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/seatunnel/issues 2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 2024-02-04 13:51:49,230 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Compressed class space at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347) at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340) at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65) at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759) at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709) at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89) at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: java.lang.OutOfMemoryError: Compressed class space 2024-02-04 13:51:49,230 ERROR org.apache.seatunnel.core.starter.SeaTunnel - =============================================================================== Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Compressed class space at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347) at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340) at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65) at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759) at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709) at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89) at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: java.lang.OutOfMemoryError: Compressed class space ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version jdk1.8 ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
