umehrot2 commented on a change in pull request #2417:
URL: https://github.com/apache/hudi/pull/2417#discussion_r554245704
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java
##########
@@ -159,12 +160,12 @@ private static HoodieTableFileSystemView
createInMemoryFileSystemView(Serializab
return new HoodieTableFileSystemView(metaClient, timeline,
viewConf.isIncrementalTimelineSyncEnabled());
}
- public static HoodieTableFileSystemView
createInMemoryFileSystemView(HoodieTableMetaClient metaClient,
- boolean
useFileListingFromMetadata,
- boolean
verifyListings) {
+ public static HoodieTableFileSystemView
createInMemoryFileSystemViewForReaders(HoodieTableMetaClient metaClient,
+ boolean useFileListingFromMetadata, boolean verifyListings) {
LOG.info("Creating InMemory based view for basePath " +
metaClient.getBasePath());
if (useFileListingFromMetadata) {
- return new HoodieMetadataFileSystemView(metaClient,
+ HoodieMREngineContext engineContext = new
HoodieMREngineContext(metaClient.getHadoopConf());
Review comment:
Hudi has added support for other engines like Flink. Also
HoodieSparkEngineContext cannot be used here this API is called inside
individual map reduce tasks from engines like spark and hive. In case of spark,
the individual tasks do not have access to spark context. The spark context is
not serializable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
