deepakpanda93 opened a new pull request, #19597:
URL: https://github.com/apache/hudi/pull/19597
### Describe the issue this Pull Request addresses
Closes #14971.
The **Sources** section of the Hudi Streamer page documents DFS, Kafka,
Pulsar, Kinesis, cloud storage events, JDBC and
SQL — but not the Debezium sources, even though Hudi ships a source and a
matching payload for both PostgreSQL and MySQL.
I confirmed the gap rather than assuming it. Across `website/docs` and all
five 1.x versioned copies, `debezium` appears
in exactly two places: `record_merger.md`, where the two payloads are named
as examples of a custom payload, and a
feature list in `use_cases.md`. Nothing anywhere names the source classes or
shows how to configure them.
### Summary and Changelog
Adds a `#### Debezium` subsection under **Sources**, inserted as the last
child before `### Error Table`, covering:
- the source and payload class per database, as a table
- the schema registry requirement, and which Kafka config is *not* needed
- a property file plus a `spark-submit` example, following the shape of the
existing Kafka example on the same page
- how ordering is decided, and how deletes are recognised
- `hoodie.debezium.override.initial.checkpoint.key`, with a caution about
its actual lifetime
### Every fact was re-derived from the release tags
There is an earlier closed attempt at this, #4481. Rather than carry
anything over from it, I read each fact out of the
tagged source — which is how several of the details below ended up different
from what I'd initially assumed:
| Documented as | Verified against |
|---|---|
| `MysqlDebeziumSource` but `MySqlDebeziumAvroPayload` | file listings in
all five 1.x tags — the two halves genuinely capitalise MySQL differently, so
this is called out explicitly |
| Postgres orders on `_event_lsn` | `PostgresDebeziumAvroPayload.extractLSN`
reads that column straight off the record, so merge correctness does **not**
depend on a configured ordering field |
| MySQL orders on `_event_seq` | `MysqlDebeziumSource:95,107` — derived via
`withColumn` from `_event_bin_file` and `_event_pos` |
| Delete marker is `_change_operation_type` = `d` |
`DebeziumConstants:52,70`. Worth flagging: this is **not** `_event_op`, which
is the kind of plausible-looking name that is easy to write and wrong |
| The Avro deserializer is optional | `KafkaSourceConfig:64-66` gives
`hoodie.streamer.source.kafka.value.deserializer.class` a default of
`io.confluent.kafka.serializers.KafkaAvroDeserializer`, so it is documented as
an override, not a requirement |
| The checkpoint override applies to **every** batch |
`DebeziumSource:119,132` — `fetchNextBatch` returns `new
StreamerCheckpointV2(overrideCheckpointStr.isEmpty() ? offsets :
overrideCheckpointStr)` unconditionally, so while the config is set the
committed checkpoint is always the override and the job restarts from the same
offset forever |
That last one is the footgun in this source, so it gets a `:::caution`
rather than a passing mention.
### Merge mode is deliberately not documented
This is the part I want reviewers to look at hardest, because it is where a
single sentence would have been wrong.
Setting a Debezium payload does **not** land in `CUSTOM` merge mode, which
is what
`inferRecordMergeModeFromPayloadClass` on its own suggests. Both payloads
are listed in `HoodieTableConfig`'s
`PAYLOADS_UNDER_DEPRECATION` *and* in `EVENT_TIME_ORDERING_PAYLOADS`, so the
deprecation branch wins: it reconciles to
`EVENT_TIME_ORDERING`, sets `LEGACY_PAYLOAD_CLASS_NAME`, and via
`handlePayloadAdhocConfigs` /
`handlePartialUpdateModeConfigs` also fills in the ordering fields, delete
key, delete marker and (for Postgres)
`PartialUpdateMode.FILL_UNAVAILABLE`.
But that machinery **does not exist across all of 1.x**:
| Release | `PAYLOADS_UNDER_DEPRECATION` | `handlePayloadAdhocConfigs` |
`inferRecordMergeModeFromPayloadClass` |
|---|---|---|---|
| release-1.0.0 | — | — | — |
| release-1.0.1 | — | — | present |
| release-1.0.2 | — | — | present |
| release-1.1.1 | present | present | present |
| release-1.2.0 | present | present | present |
So any statement about reconciled merge mode or auto-populated ordering
fields would be false on three of the six pages
this PR touches. The section therefore restricts itself to behaviour that
holds on every 1.x release. I'm raising it here
instead of silently omitting it, in case a reviewer would rather see it
documented with an explicit version qualifier.
Relatedly, `--source-ordering-field _event_lsn` in the example is safe on
every release: it is still an accepted alias of
`--source-ordering-fields` at 1.2.0 (`HoodieStreamer:278`), and at 1.0.x it
displaces a default of `"ts"`
(`HoodieStreamer:271`) that no Debezium table has. `--payload-class` is
marked deprecated in the CLI help, but it remains
the mechanism for Debezium — Hudi special-cases both of these payload
classes by name in table config reconciliation.
### Version scope
Applied to `next` and all five 1.x copies. Applicability was checked per
release rather than inferred from the class
merely existing: `PostgresDebeziumSource`, `MysqlDebeziumSource` and
`DebeziumSource` are present in all five tags, every
`DebeziumConstants` column name used here is present in every tag, and the
MySQL `_event_seq` derivation is identical
throughout. 0.14.x and 0.15.x are excluded.
The added text is **byte-identical in all six files** (md5 `7742c1236a64`
over the added lines, normalised) apart from the
bundle version in the `spark-submit`, which is pinned to the release each
copy documents rather than left stale.
### Site verification
`npm run build` passes with the warning block **byte-identical** to a
baseline rebuilt from the same base commit
(`d11a5b0adee4`, 13,265 lines both ways); the stash/restore round-trip used
to produce it was re-checked byte-identical.
No warning mentions Debezium.
Rendering confirmed under `npm run serve` on `/docs`, `/docs/next`,
`/docs/1.1.1`, `/docs/1.0.2`, `/docs/1.0.1` and
`/docs/1.0.0`: the heading, its `id="debezium"` anchor, the TOC entry, both
class names, `_event_lsn`, `_event_seq`,
`_change_operation_type` and the caution admonition all render on each.
`/docs/0.15.1` correctly shows none of it.
Two honest limits on that, both worth stating rather than leaving implied:
1. Byte-identical warning parity proves only that this PR introduces no
**new** warnings. It cannot surface pre-existing
ones, since `onBrokenAnchors` is unset and defaults to `warn`.
2. This is documentation verified by reading the tagged source, not by
execution. I have not stood up a live
Debezium → Kafka → schema registry → Hudi pipeline, so the `spark-submit`
recipe is verified against the code paths it
invokes rather than by running it. Happy to do that before merge if a
reviewer wants the stronger evidence.
### Impact
Documentation only. No code, config, or behaviour change.
### Risk Level
none
### Documentation Update
This PR is the documentation update — the Hudi Streamer page,
`/docs/hoodie_streaming_ingestion#debezium` and
`/docs/next/hoodie_streaming_ingestion#debezium`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
