nsivabalan opened a new pull request #2927:
URL: https://github.com/apache/hudi/pull/2927
## What is the purpose of the pull request
Once table's schema got evolved, if a new batch of ingestion has old schema
records, it will fail. This patch fixes this by recreating the records with
latest table schema.
## Brief change log
- Add a conversion step in HoodieSparkSqlWriter to regenerate
GenericRecords with latest schema if table schema got evolved, but incoming
batch is of old schema.
## Verify this pull request
This change added tests and can be verified as follows:
- Fixed tests in HoodieSparkSqlWriterSuite to verify the change.
Todo:
- Add UT for new code in TableSchemaResolver.
- Check if a new config is required to guard the new change in
HoodieSparkSqlWriter so that only interested users will enable it.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
