hudi-agent commented on code in PR #19597: URL: https://github.com/apache/hudi/pull/19597#discussion_r3773295105
########## website/docs/hoodie_streaming_ingestion.md: ########## @@ -628,6 +628,86 @@ Using `org.apache.hudi.utilities.sources.SqlFileBasedSource` allows setting the table. SQL file path should be configured using this hoodie config: `hoodie.streamer.source.sql.file = 'hdfs://xxx/source.sql'` +#### Debezium + +Hudi Streamer can keep a Hudi table in sync with an upstream database by ingesting change data capture (CDC) events +produced by [Debezium](https://debezium.io/). Debezium publishes each change as an Avro message on a Kafka topic and +registers the schema with a Confluent schema registry. The Debezium sources read that topic, flatten the nested Debezium +change envelope into ordinary table columns, and apply the resulting inserts, updates and deletes to the target table. + +There is one source and one matching payload class per database: + +| Database | Source class | Payload class | +|------------|---------------------------------------------------------------------|---------------------------------------------------------------------| +| PostgreSQL | `org.apache.hudi.utilities.sources.debezium.PostgresDebeziumSource` | `org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload` | +| MySQL | `org.apache.hudi.utilities.sources.debezium.MysqlDebeziumSource` | `org.apache.hudi.common.model.debezium.MySqlDebeziumAvroPayload` | + +Note that the two halves spell MySQL differently: the source is `Mysql...` while the payload is `MySql...`. + +Both sources read Avro and require a schema registry, so set `--schemaprovider-class` to +`org.apache.hudi.utilities.schema.SchemaRegistryProvider` and point `hoodie.streamer.schemaprovider.registry.url` at the +subject for the topic. The Kafka value deserializer already defaults to +`io.confluent.kafka.serializers.KafkaAvroDeserializer`, so `hoodie.streamer.source.kafka.value.deserializer.class` only +needs setting in order to override it. + +A property file for a PostgreSQL table: + +```properties +hoodie.streamer.source.kafka.topic=postgres.public.customers +hoodie.streamer.schemaprovider.registry.url=http://localhost:8081/subjects/postgres.public.customers-value/versions/latest +bootstrap.servers=localhost:9092 Review Comment: 🤖 The property file (and the deserializer note just above it) appears to be missing the native `schema.registry.url` config. Since the source keeps the default `io.confluent.kafka.serializers.KafkaAvroDeserializer`, that deserializer needs `schema.registry.url` set on the Kafka consumer to resolve writer schemas — and it's a separate, required property from `hoodie.streamer.schemaprovider.registry.url`. `KafkaOffsetGen.excludeHoodieConfigs` strips all `hoodie.*` props before building the consumer, so the schemaprovider URL is not forwarded, and `configureSchemaDeserializer` only runs for Hudi's `KafkaAvroSchemaDeserializer`, not the Confluent default. As written, a user copying this example would hit `ConfigException: Missing required configuration "schema.registry.url"` at startup (cf. `TestAvroKafkaSource` and the existing `kafka-source.properties` test resource, which both set it explicitly). It would help to add e.g. `schema.registry.url=http://localhost:8081` to the prop erty file and mention it in the prose. Same fix applies to all five versioned copies. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
