zhilinli123 commented on code in PR #5992:
URL: https://github.com/apache/seatunnel/pull/5992#discussion_r1586291075
##########
docs/en/connector-v2/source/kafka.md:
##########
@@ -180,3 +181,65 @@ source {
}
```
+### Multiple Kafka Source
+
+> Currently, multiple kafka source reads are supported using the Zeta engine,
but note that you can only configure one instance of `bootstrap.servers`, and
only one in the `table_list` and `topic` parameters,Currently, when you use
multiple topics in `table_list`, you are still free to set parameters for each
`topic`
+
+```hocon
+
+env {
+ execution.parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ Kafka {
+ bootstrap.servers = "kafka_e2e:9092"
+ table_list = [
+ {
+ topic = "^test-ogg-sou.*"
+ pattern = "true"
+ consumer.group = "ogg_multi_group"
+ start_mode = earliest
+ schema = {
+ fields {
+ id = "int"
+ name = "string"
+ description = "string"
+ weight = "string"
+ }
+ },
+ format = ogg_json
+ }, {
+ topic = "test-cdc_mds"
+ consumer.group = "canal_multi_group"
+ start_mode = earliest
+ schema = {
+ fields {
+ id = "int"
+ name = "string"
+ description = "string"
+ weight = "string"
+ }
+ },
+ format = canal_json
+ }
+ ]
+ # Each topic in the `table_list` shares the config
+ kafka.config = {
+ client.id = client_1
+ max.poll.records = 50000
+ }
+ }
+}
+
+sink {
+ Assert {
+ rules {
+ // The current table name is the same as the topic name in order to
maintain fewer configuration parameters
+ table-names = ["^test-ogg-sou.*","test-cdc_mds"]
+ }
+ }
+}
Review Comment:
Ok, I will finish the e2e addition of this soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
