prashantwason commented on a change in pull request #3083:
URL: https://github.com/apache/hudi/pull/3083#discussion_r655608706
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metadata/TestHoodieBackedMetadata.java
##########
@@ -120,46 +120,63 @@ public void testDefaultNoMetadataTable() throws Exception
{
assertThrows(TableNotFoundException.class, () ->
HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePath).build());
// Metadata table is not created if disabled by config
+ String firstCommitTime = HoodieActiveTimeline.createNewInstantTime();
try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext,
getWriteConfig(true, false))) {
- client.startCommitWithTime("001");
- client.insert(jsc.emptyRDD(), "001");
+ client.startCommitWithTime(firstCommitTime);
+ client.insert(jsc.parallelize(dataGen.generateInserts(firstCommitTime,
5)), firstCommitTime);
Review comment:
In the bootstrap path this will do nothing. But in the normal path (post
bootstrap), this path also syncs the metadata table.
The metadata table sync is indepdendant of the dataset commits. So its
possible that the dataset commit may have completed and metadata table sync
failed (due to some error or crash) in postWrite. So calling the sync again in
preWrite ensures we sync again.
@vinothchandar Do you think the sync in preWrite can be removed as an
optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
