[I] [Bug] [SqlServer][Doris] Writing monitoring data from sqlserverCDC to Doris will add an additional column [seatunnel]

Wed, 29 Nov 2023 23:39:50 -0800 


moonshineyou opened a new issue, #5942:
URL: https://github.com/apache/seatunnel/issues/5942

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### What happened
   
   Writing monitoring data from sqlserverCDC to Doris will add an additional 
column, which will always have a value of 0. I am using the console to output 
normally, but writing to Doris will result in an error, which is an extra 
column.
   从sqlserverCDC监控数据写入Doris会增加一列，这一列值一直为0。我使用控制台输出是正常的，但写入Doris就会报错，多了一列。
   
   ### SeaTunnel Version
   
   2.3.3
   
   ### SeaTunnel Config
   
   ```conf
   env {
     execution.parallelism = 1
     job.mode = "STREAMING"
     job.name = "t_sqlserver"
   }
   
   source {
     SqlServer-CDC {
       result_table_name = "ts1"
   
       username = "sa"
       password = "Passswd!"
       database-names = ["cs"]
       table-names = ["cs.dbo.tbl1"]
       startup.mode = "latest"
       base-url="jdbc:sqlserver://localhost:7000;databaseName=cs"
       stop.mode="never"
       connect.timeout=120
       debezium {
           snapshot.mode = "never"
           decimal.handling.mode = "double"
           skipped.operations = "d"
           database.server.timezone = "Asia/Shanghai"
       }
     }
   }
   
   transform {
     Sql {
       source_table_name = "ts1"
       result_table_name = "tsr1"
       query = "select concat_ws(';',id,temp,dt) as d from ts1"
     }
   }
   
   sink {
     #Console {
     # source_table_name = "tsr1"
     #}
     Doris {
       source_table_name = "tsr1"
       fenodes = "localhost:8030"
       username = root
       password = "123456"
       table.identifier = "db.ods_tbl1"
       sink.label-prefix = "t-cdc"
       sink.enable-2pc = "true"
       sink.enable-delete = "true"
       doris.config {
         format = "csv"
         column_separator = ";"
         columns = "id,temp,dt"
       }
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   bin/seatunnel.sh --config work/v2.sqlserver.doris.t.config -e local
   ```
   
   
   ### Error Exception
   
   ```log
   Caused by: 
org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: 
java.lang.RuntimeException: 
org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: 
ErrorCode:[Doris-01], ErrorDescription:[stream load error] - stream load error: 
[INTERNAL_ERROR]too many filtered rows
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   Zeta 2.3.3
   
   ### Java or Scala Version
   
   java 1.8
   
   ### Screenshots
   
   Using Console Sink Output：
   
![image](https://github.com/apache/seatunnel/assets/44424339/1a5b9f58-3f21-474d-b287-d22eb9f96285)
   Using Doris Sink Output：
   
![image](https://github.com/apache/seatunnel/assets/44424339/3ff2d851-b762-4224-b969-4e1fd9d8bfa5)
   
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to