hudi-bot opened a new issue, #14543:
URL: https://github.com/apache/hudi/issues/14543
When I write data to hudi, I find it's lots of loading table properties log,
and each take about 75ms, like below.
{code:java}
20/03/05 09:20:32.379 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml],
FileSystem: [org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem@563f38c4]
20/03/05 09:20:32.431 INFO HoodieTableConfig: Loading table properties from
oss://shenhong-test/mr_100_100_100/.hoodie/hoodie.properties
20/03/05 09:20:32.453 INFO HoodieTableMetaClient: Finished Loading Table of
type MERGE_ON_READ(version=1) from oss://shenhong-test/mr_100_100_100
{code}
It's seems we have to create HoodieTableMetaClient when we use it. but it's
not need to do that, eg, when we call getHoodieTable, it transfer metaClient,
but the constructed function in HoodieTable still createMetaClient .
{code:java}
public static <T extends HoodieRecordPayload> HoodieTable<T>
getHoodieTable(HoodieTableMetaClient metaClient,
HoodieWriteConfig config, JavaSparkContext jsc) {
switch (metaClient.getTableType()) {
case COPY_ON_WRITE:
return new HoodieCopyOnWriteTable<>(config, jsc);
case MERGE_ON_READ:
return new HoodieMergeOnReadTable<>(config, jsc);
default:
throw new HoodieException("Unsupported table type :" +
metaClient.getTableType());
}
}
{code}
{code:java}
protected HoodieTable(HoodieWriteConfig config, JavaSparkContext jsc) {
this.config = config;
this.hadoopConfiguration = new
SerializableConfiguration(jsc.hadoopConfiguration());
this.viewManager = FileSystemViewManager.createViewManager(new
SerializableConfiguration(jsc.hadoopConfiguration()),
config.getViewStorageConfig());
this.metaClient = ClientUtils.createMetaClient(jsc, config, true);
this.index = HoodieIndex.createIndex(config, jsc);
}
{code}
Can we optimize it.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-657
- Type: Bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
