nsivabalan commented on code in PR #9667:
URL: https://github.com/apache/hudi/pull/9667#discussion_r1320486278
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -726,10 +734,9 @@ public void
testBulkInsertsAndUpsertsWithBootstrap(HoodieRecordType recordType)
cfg.configs.add("hoodie.datasource.write.hive_style_partitioning=true");
cfg.configs.add("hoodie.bootstrap.parallelism=5");
cfg.targetBasePath = newDatasetBasePath;
- new HoodieDeltaStreamer(cfg, jsc).sync();
+ HoodieDeltaStreamer ds = new HoodieDeltaStreamer(cfg, jsc);
+ ds.sync();
Review Comment:
do you think we can add try finally and move df.shutdownGracefully into
finally ?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerMetrics.java:
##########
@@ -157,6 +159,11 @@ public void shutdown() {
if (metrics != null) {
metrics.shutdown();
}
+ // if metadata table is enabled, make sure to shut down the metrics for
that table as well
+ if (writeConfig.getMetadataConfig().enabled()) {
+ HoodieWriteConfig metadataWriteConfig =
HoodieMetadataWriteUtils.createMetadataWriteConfig(writeConfig,
HoodieFailedWritesCleaningPolicy.LAZY);
Review Comment:
Probably better option is:
the metrics within WriteClient should be closed when write client is closed.
Since we are at this, can you also take care of below?
We also have HoodieMetadataMetrics within HoodieBackedTableMetadataWriter
which also need to be closed when the MetadataWriter is closed.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -780,10 +788,12 @@ public void testModifiedTableConfigs() throws Exception {
}
private void syncAndAssertRecordCount(HoodieDeltaStreamer.Config cfg,
Integer expected, String tableBasePath, String metadata, Integer totalCommits)
throws Exception {
- new HoodieDeltaStreamer(cfg, jsc).sync();
+ HoodieDeltaStreamer ds = new HoodieDeltaStreamer(cfg, jsc);
+ ds.sync();
TestHelpers.assertRecordCount(expected, tableBasePath, sqlContext);
TestHelpers.assertDistanceCount(expected, tableBasePath, sqlContext);
TestHelpers.assertCommitMetadata(metadata, tableBasePath, fs,
totalCommits);
+ ds.shutdownGracefully();
Review Comment:
same suggestion as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
