duanmuyh opened a new issue, #6240: URL: https://github.com/apache/seatunnel/issues/6240
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened Collecting local text files into Doris, and attempting to put the files into HDFS has a low processing speed. ### SeaTunnel Version seatunnel 2.3.3 Doris 2.0.2 Java 1.8.0_333 ### SeaTunnel Config ```conf env { # You can set flink configuration here execution.parallelism = 10 job.mode = "BATCH" } source { localFile { file_format_type = "text" path = "/home/hadoop/yangst/wudao_20240111.txt" } } sink { jdbc { url = "jdbc:mysql://xxx.xx.xx:port/wudao?rewriteBatchedStatements=true" driver = "com.mysql.cj.jdbc.Driver" user = "root" password = "xxxx" query = "insert into wudao(line) values(?)" } } ``` ### Running Command ```shell nohup sh bin/seatunnel.sh -c config/wudao.config -e local > nohup.out 2>&1 & ``` ### Error Exception ```log Job Progress Information *********************************************** Job Id : 799900012245942273 Read Count So Far : 35192 Write Count So Far : 26996 Average Read Count : 50/s Average Write Count : 50/s Last Statistic Time : 2024-01-17 15:35:05 Current Statistic Time : 2024-01-17 15:36:05 *********************************************** 2024-01-17 15:37:02,846 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 1 completedTaskCount : 67 taskCount : 68 *********************************************** 2024-01-17 15:37:02,848 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 restartingJobCount : 0 suspendedJobCount : 0 reconcilingJobCount : 0 *********************************************** 2024-01-17 15:37:05,614 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner - *********************************************** Job Progress Information *********************************************** Job Id : 799900012245942273 Read Count So Far : 37192 Write Count So Far : 28996 Average Read Count : 33/s Average Write Count : 33/s Last Statistic Time : 2024-01-17 15:36:05 Current Statistic Time : 2024-01-17 15:37:05 *********************************************** 2024-01-17 15:38:02,845 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 1 completedTaskCount : 67 taskCount : 68 *********************************************** 2024-01-17 15:38:02,849 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 restartingJobCount : 0 suspendedJobCount : 0 reconcilingJobCount : 0 *********************************************** 2024-01-17 15:38:05,613 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner - *********************************************** Job Progress Information *********************************************** Job Id : 799900012245942273 Read Count So Far : 38192 Write Count So Far : 29996 Average Read Count : 16/s Average Write Count : 16/s Last Statistic Time : 2024-01-17 15:37:05 Current Statistic Time : 2024-01-17 15:38:05 *********************************************** 2024-01-17 15:39:02,846 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 1 completedTaskCount : 67 taskCount : 68 *********************************************** 2024-01-17 15:39:02,849 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 restartingJobCount : 0 suspendedJobCount : 0 reconcilingJobCount : 0 *********************************************** 2024-01-17 15:39:05,613 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner - *********************************************** Job Progress Information *********************************************** Job Id : 799900012245942273 Read Count So Far : 41192 Write Count So Far : 32996 Average Read Count : 50/s Average Write Count : 50/s Last Statistic Time : 2024-01-17 15:38:05 Current Statistic Time : 2024-01-17 15:39:05 *********************************************** 2024-01-17 15:39:45,040 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 4 2024-01-17 15:40:02,845 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 4 completedTaskCount : 72 taskCount : 73 *********************************************** 2024-01-17 15:40:02,847 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 restartingJobCount : 0 suspendedJobCount : 0 reconcilingJobCount : 0 *********************************************** 2024-01-17 15:40:05,609 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner - *********************************************** Job Progress Information *********************************************** Job Id : 799900012245942273 Read Count So Far : 44192 Write Count So Far : 35996 Average Read Count : 50/s Average Write Count : 50/s Last Statistic Time : 2024-01-17 15:39:05 Current Statistic Time : 2024-01-17 15:40:05 *********************************************** ``` ### Zeta or Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
