prashantwason commented on code in PR #8467:
URL: https://github.com/apache/hudi/pull/8467#discussion_r1172143179
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -170,9 +171,12 @@ protected <T extends SpecificRecordBase>
HoodieBackedTableMetadataWriter(Configu
"Cleaning is controlled internally for Metadata table.");
ValidationUtils.checkArgument(!this.metadataWriteConfig.inlineCompactionEnabled(),
"Compaction is controlled internally for metadata table.");
- // Metadata Table cannot have metadata listing turned on. (infinite
loop, much?)
+ // Auto commit is required
ValidationUtils.checkArgument(this.metadataWriteConfig.shouldAutoCommit(),
"Auto commit is required for Metadata Table");
+
ValidationUtils.checkArgument(this.metadataWriteConfig.getWriteStatusClassName().equals(FailOnFirstErrorWriteStatus.class.getName()),
+ "MDT should use " + FailOnFirstErrorWriteStatus.class.getName());
+ // Metadata Table cannot have metadata listing turned on. (infinite
loop, much?)
Review Comment:
Both.
We are explicitly checking the commit only. We are not checking compaction.
In our production tables, there was an issue where compaction had errors and
the WriteStatus had errors but since we did not check it the compaction lost
data in MDT. Using the FailOnFirstErrorWriteStatus is better than the ugly code
to manually check each WriteStatus.
Also, once the commit has completed, we should not iterate over the RDDs
used in that commit since the RDDs may have been un-persisted and Spark may
perform the operation again to re-compute them. The WriteCLient have a
releaseResources() method which un-persists all RDDs part of that operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
