prashantwason opened a new pull request #1457: [HUDI-741] Added checks to validate Hoodie's schema evolution. URL: https://github.com/apache/incubator-hudi/pull/1457 ## What is the purpose of the pull request HUDI specific validation of schema evolution should ensure that a newer schema can be used for the dataset by checking that the data written using the old schema can be read using the new schema. Please see [HUDI-741 ](https://issues.apache.org/jira/browse/HUDI-741) for my detailed analysis of why this is required and how this is implemented. ## Brief change log Code changes: 1. Added a new config in HoodieWriteConfig to enable schema validation check (disabled by default) 2. Moved code that reads schema from base/log files into hudi-common from hudi-hive-sync 3. Added a class org.apache.hudi.common.util.SchemaCompatibility which implements checks for schema evolution. This is based on org.apache.avro.SchemaCompatibility but performs HUDI specific checks. Testing changes: 4. Extended HoodieTestDataGenerator to generate records using a custom Schema 5. Extended TestHoodieClientBase to add insertBatch API which allows inserting a new batch of unique records into a HUDI table 6. Added a unit test to verify schema evolution for both COW and MOR tables. ## Verify this pull request This change added tests and can be verified as follows: - Added unit test TestTableSchemaEvolution ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
