vinothchandar commented on a change in pull request #923: HUDI-247 Unify the
initialization of HoodieTableMetaClient in test for hoodie-client module
URL: https://github.com/apache/incubator-hudi/pull/923#discussion_r329066558
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -477,4 +478,43 @@ public void setActiveTimeline(HoodieActiveTimeline
activeTimeline) {
public void setTableConfig(HoodieTableConfig tableConfig) {
this.tableConfig = tableConfig;
}
+
+ //------------------------------------------------------
+ // Resource initialization and cleanup
+ //------------------------------------------------------
+
+ private void init(Configuration conf, String basePath,
+ boolean loadActiveTimelineOnLoad, ConsistencyGuardConfig
consistencyGuardConfig) {
+ log.info("Loading HoodieTableMetaClient from " + basePath);
+ this.basePath = basePath;
+ this.consistencyGuardConfig = consistencyGuardConfig;
+ this.hadoopConf = new SerializableConfiguration(conf);
+ Path basePathDir = new Path(this.basePath);
+ this.metaPath = new Path(basePath, METAFOLDER_NAME).toString();
+ Path metaPathDir = new Path(this.metaPath);
+ this.fs = getFs();
+ DatasetNotFoundException.checkValidDataset(fs, basePathDir, metaPathDir);
+ this.tableConfig = new HoodieTableConfig(fs, metaPath);
+ this.tableType = tableConfig.getTableType();
+ log.info("Finished Loading Table of type " + tableType + " from " +
basePath);
+ if (loadActiveTimelineOnLoad) {
+ log.info("Loading Active commit timeline for " + basePath);
+ getActiveTimeline();
+ }
+ }
+
+ private void cleanup() throws IOException {
+ this.basePath = null;
+ this.consistencyGuardConfig = null;
+ this.hadoopConf = null;
+ this.metaPath = null;
+ if (this.fs != null) {
Review comment:
ah.. lets not close the fs object... This instance will be cached at the
FileSystem level and closing this here, would make other threads (e.g with
num.cores > 1, other spark executor's thread on same jvm to fail). #620 we have
worked through this before..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
