linliu-code opened a new pull request, #18655:
URL: https://github.com/apache/hudi/pull/18655
Adds a new utility, SchemaChangeLogger, called from StreamSync at the
existing schema-detection point. When a SchemaProvider returns a schema
different from the previous one, the logger emits a single INFO line naming the
SchemaProvider class plus a per-field diff (added, removed, type-changed). For
changes the Avro reader/writer compatibility checker flags as INCOMPATIBLE, an
additional WARN line includes the full pre/post schemas so an operator has
everything needed to investigate.
Motivation: when a schema-related write failure surfaces (AIOOBE,
MatchError, or silent NULL columns — see ENG-40683, prod_cdc_sage_history,
catapult_period_stats, Bswift incidents), the first oncall question is "did the
customer change the schema, and was the change legal?". Today the only signal
is a verbose dump of two large schemas in "Seeing new schema" — operators have
to diff by eye. With this change the diff is precomputed and the compatibility
classification is logged inline.
Comparison is shallow (top-level fields). For nested record evolution the
diff records a top-level type change; the full schemas are still logged on
INCOMPATIBLE so an operator can drill into nested differences. SchemaProvider
class name is included so it's clear whether the schema came from
FilebasedSchemaProvider, SchemaRegistryProvider, or another source.
The existing "Seeing new schema" log line is preserved unchanged for
backward compatibility — anyone parsing it continues to work.
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the
changes
(or) link to an issue by including `Closes #<issue-number>` for
context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in
behavior.
Followed by a detailed log of all the changes. Highlight if any code
was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any
performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain
the risk.
If medium or high, explain what verification was done to mitigate the
risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new
feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the
default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website.
Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website)
to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
