alexeykudinkin commented on a change in pull request #4106:
URL: https://github.com/apache/hudi/pull/4106#discussion_r765345671
##########
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/columnstats/ColumnStatsIndexHelper.java
##########
@@ -402,52 +297,52 @@ public static void updateZIndexFor(
.filter(f -> !completedCommits.contains(f))
.collect(Collectors.toList());
- Dataset<Row> finalZIndexDf;
-
- // Before writing out new version of the Z-index table we need to merge
it
+ Dataset<Row> finalColStatsIndexDf;
+
+ // Before writing out new version of the col-stats-index table we need
to merge it
// with the most recent one that were successfully persisted previously
if (validIndexTables.isEmpty()) {
- finalZIndexDf = newZIndexDf;
+ finalColStatsIndexDf = newColStatsIndexDf;
} else {
// NOTE: That Parquet schema might deviate from the original table
schema (for ex,
// by upcasting "short" to "integer" types, etc), and hence we
need to re-adjust it
// prior to merging, since merging might fail otherwise due to
schemas incompatibility
- finalZIndexDf =
+ finalColStatsIndexDf =
tryMergeMostRecentIndexTableInto(
sparkSession,
- newZIndexDf,
- // Load current most recent Z-index table
+ newColStatsIndexDf,
+ // Load current most recent col-stats-index table
sparkSession.read().load(
- new Path(zindexFolderPath,
validIndexTables.get(validIndexTables.size() - 1)).toString()
+ new Path(indexFolderPath,
validIndexTables.get(validIndexTables.size() - 1)).toString()
)
);
// Clean up all index tables (after creation of the new index)
tablesToCleanup.addAll(validIndexTables);
}
- // Persist new Z-index table
- finalZIndexDf
- .repartition(1)
- .write()
- .format("parquet")
- .save(newIndexTablePath.toString());
+ // Persist new col-stats-index table
+ finalColStatsIndexDf
+ .repartition(1)
+ .write()
+ .format("parquet")
+ .save(newIndexTablePath.toString());
Review comment:
Yeah, some of these is IDEA auto re-formatting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
