spihiker opened a new issue, #6014: URL: https://github.com/apache/seatunnel/issues/6014
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened service crash。 ### SeaTunnel Version 2.3.3 ### SeaTunnel Config ```conf env { # You can set SeaTunnel environment configuration here execution.parallelism = 2 job.mode = "STREAMING" # 10秒检查一次，可以适当加大这个值 checkpoint.interval = 10000 #execution.checkpoint.interval = 10000 #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint" } # 配置数据源 source { MySQL-CDC { # 数据库账号 username = "cdc" password = "cdc" # 源表，格式：数据库名.表名 table-names = ["mydb.t1_202310","mydb.t1_202311"] base-url = "jdbc:mysql://172.20.*.*:3307/mydb" } } # 配置目标库 sink { Clickhouse { host = "172.20.*.*:8123" database = "mydb" table = "t1" username = "dev" password = "dev" # cdc options primary_key = "c1" support_upsert = true allow_experimental_lightweight_delete = true } } ``` ### Running Command ```shell ./bin/seatunnel.sh --config ./config/mycdc-2-ch.conf -e local ``` ### Error Exception ```log java.lang.RuntimeException: org.apache.seatunnel.connectors.seatunnel.clickhouse.exception.ClickhouseConnectorException: ErrorCode:[COMMON-10], ErrorDescription:[Flush data operation that in sink connector failed] - Clickhouse execute batch statement error at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:232) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:61) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:71) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:51) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:52) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613) [seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_391] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_391] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_391] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_391] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_391] Caused by: org.apache.seatunnel.connectors.seatunnel.clickhouse.exception.ClickhouseConnectorException: ErrorCode:[COMMON-10], ErrorDescription:[Flush data operation that in sink connector failed] - Clickhouse execute batch statement error at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:119) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:134) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.prepareCommit(ClickhouseSinkWriter.java:93) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:166) ~[seatunnel-starter.jar:2.3.3] ... 15 more Suppressed: java.sql.SQLException: Cannot set null to non-nullable column #3 [c3 String] at com.clickhouse.jdbc.SqlExceptionUtils.clientError(SqlExceptionUtils.java:73) ~[connector-clickhouse-2.3.3.jar:2.3.3] at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.addBatch(InputBasedPreparedStatement.java:328) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.FieldNamedPreparedStatement.addBatch(FieldNamedPreparedStatement.java:371) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.InsertOrUpdateBatchStatementExecutor.addToBatch(InsertOrUpdateBatchStatementExecutor.java:90) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.ReduceBufferedBatchStatementExecutor.executeBatch(ReduceBufferedBatchStatementExecutor.java:76) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.ReduceBufferedBatchStatementExecutor.closeStatements(ReduceBufferedBatchStatementExecutor.java:99) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.JdbcBatchStatementExecutor.close(JdbcBatchStatementExecutor.java:37) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:137) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.prepareCommit(ClickhouseSinkWriter.java:93) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:166) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:61) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:71) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:51) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:52) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613) [seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_391] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_391] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_391] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_391] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_391] Caused by: java.sql.SQLException: Cannot set null to non-nullable column #3 [c3 String] at com.clickhouse.jdbc.SqlExceptionUtils.clientError(SqlExceptionUtils.java:73) ~[connector-clickhouse-2.3.3.jar:2.3.3] at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.addBatch(InputBasedPreparedStatement.java:328) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.FieldNamedPreparedStatement.addBatch(FieldNamedPreparedStatement.java:371) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.InsertOrUpdateBatchStatementExecutor.addToBatch(InsertOrUpdateBatchStatementExecutor.java:90) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.ReduceBufferedBatchStatementExecutor.executeBatch(ReduceBufferedBatchStatementExecutor.java:76) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:117) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:134) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.prepareCommit(ClickhouseSinkWriter.java:93) ~[connector-clickhouse-2.3.3.jar:2.3.3] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:166) ~[seatunnel-starter.jar:2.3.3] ... 15 more 2023-12-15 18:11:51,799 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-943069] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=787984932684169217, pipelineId=1, taskGroupId=30000} 2023-12-15 18:11:51,799 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-943069] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@6951fe5c org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.InterruptedException: sleep interrupted at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:166) ~[connector-cdc-mysql-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:92) ~[connector-cdc-mysql-2.3.3.jar:2.3.3] at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:98) ~[connector-cdc-mysql-2.3.3.jar:2.3.3] ``` ### Zeta or Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
