[
https://issues.apache.org/jira/browse/HUDI-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17396939#comment-17396939
]
ASF GitHub Bot commented on HUDI-1292:
--------------------------------------
vinothchandar commented on a change in pull request #3427:
URL: https://github.com/apache/hudi/pull/3427#discussion_r686385905
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java
##########
@@ -141,6 +148,10 @@ public boolean useFileListingMetadata() {
return getBoolean(METADATA_ENABLE_PROP);
Review comment:
ideally we should rename this method
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java
##########
@@ -192,6 +192,54 @@ public void testMetadataTableBootstrap() throws Exception {
}
}
+ /**
+ * Test enable/disable sync via the config.
+ */
+ @Test
+ public void testSyncConfig() throws Exception {
+ init(HoodieTableType.COPY_ON_WRITE);
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+
+ // Create the metadata table
+ String firstCommitTime = HoodieActiveTimeline.createNewInstantTime();
+ try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext,
getWriteConfig(true, true), true)) {
+ client.startCommitWithTime(firstCommitTime);
+ client.insert(jsc.parallelize(dataGen.generateInserts(firstCommitTime,
2)), firstCommitTime);
+ client.syncTableMetadata();
+ assertTrue(fs.exists(new Path(metadataTableBasePath)));
+ validateMetadata(client);
+ }
+
+ // If sync is disabled, the table will not sync
+ HoodieWriteConfig config = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+
.enable(true).enableMetrics(false).enableSync(false).build()).build();
+ final String metadataTableMetaPath = metadataTableBasePath +
Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ String secondCommitTime = HoodieActiveTimeline.createNewInstantTime();
+ try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext,
config, true)) {
+ client.startCommitWithTime(secondCommitTime);
+ client.insert(jsc.parallelize(dataGen.generateInserts(secondCommitTime,
2)), secondCommitTime);
Review comment:
wondering if theere is a lighter weight way of testing this, without
incurring the full write path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> [Umbrella] RFC-15 : File Listing and Query Planning Optimizations
> ------------------------------------------------------------------
>
> Key: HUDI-1292
> URL: https://issues.apache.org/jira/browse/HUDI-1292
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration, Writer Core
> Affects Versions: 0.9.0
> Reporter: Vinoth Chandar
> Assignee: Prashant Wason
> Priority: Major
> Labels: hudi-umbrellas, pull-request-available
> Fix For: 0.10.0
>
>
> This is the umbrella ticket that tracks the overall implementation of RFC-15
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
