wangxiaoliang001 commented on issue #8785: URL: https://github.com/apache/seatunnel/issues/8785#issuecomment-2674352388
> Thanks for the detailed information. There're many users using MySQL connector, Can you paste your MySQL job conf? TiDB cdc is delelvoping. > > By the way, developing and debugging is not very difficult, you can refer this doc: [seatunnel-examples/seatunnel-engine-examples/src/main/java/org/apache/seatunnel/example/engine/SeaTunnelEngineLocalExample.java](https://seatunnel.apache.org/docs/2.3.9/contribution/setup), I think we can make connection ( my Linkedin: davidzollo or Wechat: davidzollo) 您好，我所看到的情况是这样的： Local 方式启动，需要要在 pom 当中集成相关连接器及其依赖，然后项目启动过程中无法找到相关连接器，因为通过 Main方式启动，相关连接器是无法注册和初始化的，所以只能对调试的连接器通过SPI实现对应的注册和初始化操作，才能运行起来。 org.apache.seatunnel.api.sink.SeaTunnelSink org.apache.seatunnel.api.source.SeaTunnelSource server 方式启动，需要先编译出 connectors, 并指定 SEATUNNEL_HOME 路径才能启动，并且同时需要在 pom 中删除相关连接器，否则会由不同的 classloader 加载，导致服务启动失败。虽然能调试，但对源代码的改动不能热加载，不管由哪种方式调试源代码，都是比较困难的。 Mysql 相关的问题遇到的很少，只是我们的表大一些，全量过程在检查索引，并做 JdbcQuery，并且后续的 binlog 也很大，因为 mysql server 当中有非常多的库表，所以并不是问题，不清楚能否通过什么方式进行优化。 这是相关配置： { "env": { "parallelism": 1, "job.mode": "STREAMING", "job.name": "mysql=>kafka1 : realtimecrowdlog_sv.log_crowdlog", "checkpoint.interval": "60000", "checkpoint.timeout": "60000", "flush.timeout.ms": "10000" }, "source": [ { "plugin_name": "MySQL-CDC", "driver": "com.mysql.cj.jdbc.Driver", "base-url": "jdbc:mysql://xxx:3306?useSSL=false", "username": "xxx", "password": "xxx", "database-names": [ "realtimecrowdlog_sv" ], "table-names": [ "realtimecrowdlog_sv.log_crowdlog" ], "startup.mode": "earliest", "server-time-zone": "Asia/Shanghai", "connect.max-retries": 3, "format" : "compatible_debezium_json", "debezium_record_include_schema": "false", "debezium" : { "database.server.name" : "oracle" } } ], "transform" : [ ], "sink" : [ { "plugin_name" : "Kafka", "bootstrap.servers" : "xxxx", "topic": "xxxx", "reroute": [ { "pattern": "oracle.realtimecrowdlog_sv.log_crowdlog", "topic": "short_video_composite", "partition": 1 } ], "kafka.config": { "acks": "1", "request.timeout.ms": 240000, "batch.size": 10240, "buffer.memory": 67108864, "send.buffer.bytes": 262144, "compression.gzip.level": 6, "compression.type": "gzip" }, "semantics" : "NON", "format" : "compatible_debezium_json" } ] } -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
