codope commented on a change in pull request #4848:
URL: https://github.com/apache/hudi/pull/4848#discussion_r820837156
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
##########
@@ -799,30 +824,20 @@ public static HoodieTableFileSystemView
getFileSystemView(HoodieTableMetaClient
/**
* Create column stats from write status.
*
- * @param engineContext - Engine context
- * @param datasetMetaClient - Dataset meta client
- * @param allWriteStats - Write status to convert
- * @param isMetaIndexColumnStatsForAllColumns - Are all columns enabled for
indexing
+ * @param engineContext - Engine context
+ * @param allWriteStats - Write status to convert
+ * @param recordsGenerationParams - Parameters for columns stats record
generation
*/
- public static List<HoodieRecord>
createColumnStatsFromWriteStats(HoodieEngineContext engineContext,
-
HoodieTableMetaClient datasetMetaClient,
-
List<HoodieWriteStat> allWriteStats,
- boolean
isMetaIndexColumnStatsForAllColumns) throws Exception {
+ public static HoodieData<HoodieRecord>
createColumnStatsFromWriteStats(HoodieEngineContext engineContext,
+
List<HoodieWriteStat> allWriteStats,
+
MetadataRecordsGenerationParams recordsGenerationParams) {
if (allWriteStats.isEmpty()) {
- return Collections.emptyList();
- }
-
- List<HoodieWriteStat> prunedWriteStats =
allWriteStats.stream().filter(writeStat -> {
- return !(writeStat instanceof HoodieDeltaWriteStat);
- }).collect(Collectors.toList());
- if (prunedWriteStats.isEmpty()) {
- return Collections.emptyList();
+ return engineContext.emptyHoodieData();
}
-
- return engineContext.flatMap(prunedWriteStats,
- writeStat -> translateWriteStatToColumnStats(writeStat,
datasetMetaClient,
- getLatestColumns(datasetMetaClient,
isMetaIndexColumnStatsForAllColumns)),
- prunedWriteStats.size());
+ HoodieData<HoodieWriteStat> allWriteStatsRDD = engineContext.parallelize(
+ allWriteStats, Math.max(allWriteStats.size(),
recordsGenerationParams.getColumnStatsIndexParallelism()));
+ return allWriteStatsRDD.flatMap(writeStat ->
translateWriteStatToColumnStats(writeStat,
recordsGenerationParams.getDataMetaClient(),
+ getColumnsToIndex(recordsGenerationParams.getDataMetaClient(),
recordsGenerationParams.isAllColumnStatsIndexEnabled())).iterator());
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
