[I] [Bug] [Source & Sink] After the local file is read and sinked, there is duplicate data [seatunnel]

[via GitHub]

CCweixiao opened a new issue, #6896:
URL: https://github.com/apache/seatunnel/issues/6896

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### What happened
   
   export version="2.3.5"
   wget 
"https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz";
   tar -zxvf apache-seatunnel-2.3.5-bin.tar.gz
   cd apache-seatunnel-2.3.5
   sh bin/install-plugin.sh 2.3.5
   cd config
   cp v2.batch.config.template local.file.config.template
   
   **config content is:**
   env {
     parallelism = 10
     job.mode = "BATCH"
   }
   
   source {
     LocalFile {
       path = "/tmp/input"
       file_format_type = "text"
       field_delimiter = ","
       schema = {
         fields {
           name = "string"
           age = "int",
           address = "string"
         }
       }
     }
   }
   
   sink {
     LocalFile {
       path = "/tmp/output"
       file_format_type = "text"
       field_delimiter = ","
     }
   }
   
   **input source files are:**
   <img width="646" alt="image" 
src="https://github.com/apache/seatunnel/assets/27730211/ef0f3e5f-fd36-441a-81ed-464877ecc8b8";>
   
   **exec commond**
   ./bin/seatunnel.sh --config ./config/local.file.config.template -e local
   
   **output sink files are:**
   <img width="1018" alt="image" 
src="https://github.com/apache/seatunnel/assets/27730211/1c195fdd-cb43-4241-9bfb-6971f6588f33";>
   
   
   ### SeaTunnel Version
   
   2.3.5
   
   ### SeaTunnel Config
   
   ```conf
   env {
     parallelism = 10
     job.mode = "BATCH"
   }
   
   source {
     LocalFile {
       path = "/tmp/input"
       file_format_type = "text"
       field_delimiter = ","
       schema = {
         fields {
           name = "string"
           age = "int",
           address = "string"
         }
       }
     }
   }
   
   sink {
     LocalFile {
       path = "/tmp/output"
       file_format_type = "text"
       field_delimiter = ","
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh --config ./config/local.file.config.template -e local
   ```
   
   
   ### Error Exception
   
   ```log
   Duplicate data
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org