prashantwason commented on a change in pull request #1457: [HUDI-741] Added
checks to validate Hoodie's schema evolution.
URL: https://github.com/apache/incubator-hudi/pull/1457#discussion_r400527359
##########
File path:
hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -457,6 +461,37 @@ private void
saveWorkloadProfileMetadataToInflight(WorkloadProfile profile, Hood
private JavaRDD<WriteStatus> upsertRecordsInternal(JavaRDD<HoodieRecord<T>>
preppedRecords, String commitTime,
HoodieTable<T> hoodieTable, final boolean isUpsert) {
+ if (getConfig().getSchemaCheck()) {
+ // Ensure that the current writerSchema is compatible with the latest
schema of this
+ // dataset.
+ // When inserting/updating data, we read records using the schema saved
in the
+ // data/log files and convert them to the GenericRecords with
writerSchema.
+ // Hence, we need to ensure that this conversion can take place without
errors.
+ try {
+ SchemaUtil schemaUtil = new SchemaUtil(hoodieTable.getMetaClient());
+ MessageType savedParquetSchema = schemaUtil.getDataSchema();
+ Schema savedSchema =
schemaUtil.convertParquetSchemaToAvro(savedParquetSchema);
+ Schema writerSchema =
HoodieWriteHandle.createHoodieWriteSchema(config.getSchema());
+ if (! schemaUtil.isSchemaCompatible(savedSchema, writerSchema)) {
+ String msg = "WriterSchema is not compatible with the schema present
in the Table";
+ LOG.error(msg);
+ LOG.warn("WriterSchema: " + writerSchema);
+ LOG.warn("Table latest schema: " + savedSchema);
+ throw new HoodieUpsertException(msg);
+ }
+ } catch (Exception e) {
+ // If this is the first insert into the table then schema will not be
present
+ if
(hoodieTable.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().countInstants()
> 0) {
Review comment:
Reading the schema may fail due to IO / timeout errors too which will raise
their exceptions.
Having this code here:
1. removes code duplication (as we need to throw HoodieUpsertException here)
2. Handles the rare case gracefully - when reading schema fails due to IO
errors but there was no schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
