eee27 opened a new issue, #8164: URL: https://github.com/apache/seatunnel/issues/8164
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened I use seatunnel to sync about 30 tables data with streaming task,max table contains about 70B data. Source: Rocketmq Sink: Mongo and Starrocks Rocketmq->Mongo tasks all right,but Rocketmq->Starrocks task in two table occurs a problem: Every day 0:10 i will send 100-1000 data to 'Video' topic in Mq,and as my think,it should be sync to Mongo and starrocks. But i only found data sync to mongo right,Starrocks task shows those data count,with no error,but i cant found those data. So later,i try sync Mongo data to Starrocks,to let my application use those data for work. But i try Mongo-CDC->Starrocks task,it run as Rocketmq->Starrocks,i can see it read count righ,with no error,but Starrocks still no data. So,i temp use Mongo->Starrocks Batch task,every day exec 1 times,its all ok. My data type in Starrocks only contains VARCHAR,INT and DATETIME, in my schema,i use STRING for VARCHAR,STRING for DATETIME,and INT for INT. > 每天凌晨我会推送几百条数据到Rocketmq的一个topic,用于分别给Mongo和Starrocks消费.每个表一个topic. > 但是在我总计三十多张表中出现了两张小表,单独出现了如下现象: > Rocketmq的数据如实同步到了Mongo中,但是Starrocks的任务,日志显示出来了对应的条目数,数据却没有进入库内,也没有显示任何异常. > 后续我尝试将Mongo的数据通过CDC流式同步到Starrocks,出现了同样的问题,所以我临时使用Mongo的批量模式单次同步到Starrocks,数据正常进库. > 在Starrocks表中只包含VARCHAR,DATETIME和INT类型的字段,对于VARCHAR和DATETIME,使用STRING类型,对于INT类型使用了INT. ### SeaTunnel Version 2.3.8 ### SeaTunnel Config ```conf env { job.mode = "streaming" } source{ Rocketmq { name.srv.addr = "rocketmq-sss:9876" topics = "TB_XXX_VIDEO_XXX" consumer.group = "aaa" schema = { fields { LAST_MODIFIED_TIME = STRING A_ID = STRING B_ID = STRING SIGNATURE = STRING VIDEO_CODEC = STRING VIDEO_BIT_RATE = STRING AUDIO_CODEC = STRING AUDIO_BIT_RATE = STRING FILE_SIZE = STRING VIDEO_TYPE = STRING SYSTEM_STATUS = STRING PREVIEW_URL = STRING KEY_FRAME_IMAGE_URL = STRING CREATE_TIME = STRING VIDEO_PROFILE_NAME = STRING AUDIO_PROFILE_NAME = STRING SOURCE_TYPE = STRING STATUS = STRING CHECK_COUNT = STRING DESCRIPTION = STRING CHECKED_OR_NOT = STRING MEDIA_TYPE = STRING MEDIA_ID = STRING TT_VIDEO_ID = STRING UPLOAD_URL = STRING IS_DEL_FLG = STRING CREATE_USER_ID = STRING CREATE_DATE = STRING UPDATE_USER_ID = STRING UPDATE_DATE = STRING } } } } sink { StarRocks { nodeUrls = ["1.1.1.1:8030","1.1.1.2:8030","1.1.1.3:8030"] base-url = "jdbc:mysql:loadbalance://1.1.1.1:9030,1.1.1.2:9030,1.1.1.3:9030/starrocks_aaa?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=Asia/Shanghai&nullCatalogMeansCurrent=true&allowMultiQueries=true" username = aaa password = "aaaaaaa" database = "starrocks_aaa" table = "TB_XXX_VIDEO_XXX" } } ``` ### Running Command ```shell ${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/root/124460871130784/125776733731297_7/207/224/seatunnel_207_224.conf --deploy-mode local ``` ### Error Exception ```log No Error,it shows task is running and show data count right,but data not write into sink. ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version JDK1.8 ### Screenshots 2024-11-25 18:49:05,867 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-913369193359867905] - *********************************************** Job Progress Information *********************************************** Job Id : 913369193359867905 Read Count So Far : 1 Write Count So Far : 1 Average Read Count : 0/s Average Write Count : 0/s Last Statistic Time : 2024-11-25 18:48:05 Current Statistic Time : 2024-11-25 18:49:05 *********************************************** [INFO] 2024-11-27 11:19:06.194 +0800 - -> 2024-11-27 11:19:05,858 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-913369193359867905] - *********************************************** Job Progress Information *********************************************** Job Id : 913369193359867905 Read Count So Far : 664 Write Count So Far : 664 Average Read Count : 0/s Average Write Count : 0/s Last Statistic Time : 2024-11-27 11:18:05 Current Statistic Time : 2024-11-27 11:19:05 *********************************************** It shows count right,but no data in db. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
