maqw666 opened a new issue, #5962: URL: https://github.com/apache/seatunnel/issues/5962
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened Using seatunel, read hdfs data to Elasticsearch. I'm not sure if there are any issues with the configuration. After executing the command, there were no errors at the beginning, but after a few minutes, they started reporting errors. I feel like it shouldn't be, or there may be issues with my own configuration. I am using the ES version 7. x, which also meets the requirements of Seattle. Could you please help me take a look? Thank you! ### SeaTunnel Version 2.3.3 ### SeaTunnel Config ```conf env { execution.parallelism = 2 job.mode = "BATCH" checkpoint.interval = 10000 } source { HdfsFile { #read_columns = ["pid","actor_exh_times","latest_journal_sd","latest_journal_ed","booth_typebooth_area"] path = "/user/hive/warehouse/***/dt=2023-12-04" file_format_type = "orc" fs.defaultFS = "hdfs://****:8020" } } transform { } sink { Elasticsearch { hosts = ["http://****:9200";] index = "*****" primary_keys = ["pid"] username = "elastic" password = "*****" } } ``` ### Running Command ```shell ./bin/seatunnel.sh --config ./config/hdfs2es -e local ``` ### Error Exception ```log 2023-12-05 13:23:45,392 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0 2023-12-05 13:23:45,392 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (784287899842510849), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-HdfsFile-default-identifier]-SourceTask (1/1)] turn from state DEPLOYING to RUNNING. 2023-12-05 13:23:45,392 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784287899842510849), Pipeline: [(1/1)] turn from state DEPLOYING to RUNNING. 2023-12-05 13:23:45,397 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0 2023-12-05 13:23:45,498 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=784287899842510849, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0} 2023-12-05 13:23:45,502 INFO org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplitEnumerator - SubTask 0 is assigned to [hdfs://master03:8020/user/hive/warehouse/dw/app_exhibition_actor/dt=2023-12-04/000000_0.orc] 2023-12-05 13:23:45,613 WARN org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils - Principal [null] or keytabPath [null] is empty, it will skip kerberos authentication 2023-12-05 13:23:45,672 INFO org.apache.seatunnel.shade.connector.file.org.apache.orc.impl.OrcCodecPool - Got brand-new codec ZLIB 2023-12-05 13:23:45,684 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received enough reader, starting enumerator... 2023-12-05 13:23:45,711 INFO org.apache.seatunnel.shade.connector.file.org.apache.orc.impl.ReaderImpl - Reading ORC rows from hdfs://master03:8020/user/hive/warehouse/dw/app_exhibition_actor/dt=2023-12-04/000000_0.orc with {include: null, offset: 0, length: 9223372036854775807, schema: struct<pid:string,ent_name:string,exhibitor_name_brief:string,latest_journal_sd:string,is_actor:string,actor_exh_list:array<string>,actor_tags:string,booth_area:decimal(20,4),booth_type:array<int>,actor_exh_times:int,ent_province:string,ent_city:string,ent_district:string,ent_industry1:string,ent_industry2:string,latest_journal_ed:string,latest_journal_id:string,latest_exh_name:string,actor_contact_cnt:string>, includeAcidColumns: true} 2023-12-05 13:23:55,595 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 22 2023-12-05 13:24:05,183 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-33151] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 10 completedTaskCount : 180 taskCount : 181 *********************************************** 2023-12-05 13:24:05,185 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-33151] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 restartingJobCount : 0 suspendedJobCount : 0 reconcilingJobCount : 0 *********************************************** 2023-12-05 13:24:05,595 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 23 2023-12-05 13:24:08,336 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner - *********************************************** Job Progress Information *********************************************** Job Id : 784287899842510849 Read Count So Far : 1248042 Write Count So Far : 1239847 Average Read Count : 5533/s Average Write Count : 5499/s Last Statistic Time : 2023-12-05 13:23:08 Current Statistic Time : 2023-12-05 13:24:08 *********************************************** 2023-12-05 13:24:15,596 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 24 2023-12-05 13:24:25,596 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 25 2023-12-05 13:24:35,596 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 26 2023-12-05 13:24:45,597 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 27 A few minutes report error: 2023-12-05 11:08:57,482 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - start cancel job Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] count = 1 2023-12-05 11:08:57,483 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] turn to end state FAILED. 2023-12-05 11:08:57,483 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - start cancel job Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] count = 0 2023-12-05 11:08:57,483 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] is in end state FAILED, can not be cancel 2023-12-05 11:08:57,483 ERROR org.apache.seatunnel.engine.server.dag.physical.SubPlan - Pipeline is trying to leave terminal state FAILED 2023-12-05 11:08:57,483 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - cancel job Job SeaTunnel_Job (784253674368008193) because makeJobEndWhenPipelineEnded is true 2023-12-05 11:08:57,483 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] cancel error java.lang.IllegalStateException: Pipeline is trying to leave terminal state FAILED at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:348) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.dag.physical.SubPlan.cancelPipeline(SubPlan.java:414) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.dag.physical.SubPlan.handleCheckpointError(SubPlan.java:659) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.master.JobMaster.lambda$handleCheckpointError$2(JobMaster.java:341) ~[seatunnel-starter.jar:2.3.3] at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_181] at org.apache.seatunnel.engine.server.master.JobMaster.handleCheckpointError(JobMaster.java:338) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.handleCheckpointError(CheckpointManager.java:180) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:251) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$7(CheckpointCoordinator.java:474) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) [?:1.8.0_181] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) [?:1.8.0_181] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] 2023-12-05 11:08:57,484 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - start cancel job Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] count = 0 2023-12-05 11:08:57,484 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] is in end state FAILED, can not be cancel 2023-12-05 11:08:57,484 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (784253674368008193) turn from state RUNNING to CANCELLING. 2023-12-05 11:08:57,484 ERROR org.apache.seatunnel.engine.server.dag.physical.SubPlan - Pipeline is trying to leave terminal state FAILED 2023-12-05 11:08:57,484 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] cancel error java.lang.IllegalStateException: Pipeline is trying to leave terminal state FAILED at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:348) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.dag.physical.SubPlan.cancelPipeline(SubPlan.java:414) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.dag.physical.SubPlan.handleCheckpointError(SubPlan.java:659) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.master.JobMaster.lambda$handleCheckpointError$2(JobMaster.java:341) ~[seatunnel-starter.jar:2.3.3] at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_181] at org.apache.seatunnel.engine.server.master.JobMaster.handleCheckpointError(JobMaster.java:338) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.handleCheckpointError(CheckpointManager.java:180) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:251) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$7(CheckpointCoordinator.java:474) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) [?:1.8.0_181] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) [?:1.8.0_181] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] 2023-12-05 11:08:57,484 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - start cancel job Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] count = 0 2023-12-05 11:08:57,484 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] is in end state FAILED, can not be cancel 2023-12-05 11:08:57,484 ERROR org.apache.seatunnel.engine.server.dag.physical.SubPlan - Pipeline is trying to leave terminal state FAILED 2023-12-05 11:08:57,485 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (784253674368008193), Pipeline: [(1/1)] cancel error java.lang.IllegalStateException: Pipeline is trying to leave terminal state FAILED at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:348) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.dag.physical.SubPlan.cancelPipeline(SubPlan.java:414) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.dag.physical.SubPlan.handleCheckpointError(SubPlan.java:659) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.master.JobMaster.lambda$handleCheckpointError$2(JobMaster.java:341) ~[seatunnel-starter.jar:2.3.3] at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_181] at org.apache.seatunnel.engine.server.master.JobMaster.handleCheckpointError(JobMaster.java:338) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.handleCheckpointError(CheckpointManager.java:180) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:251) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$7(CheckpointCoordinator.java:474) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) [?:1.8.0_181] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) [?:1.8.0_181] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] ``` ### Zeta or Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
