JingsongLi commented on code in PR #9046:
URL: https://github.com/apache/paimon/pull/9046#discussion_r3720437439
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java:
##########
@@ -388,6 +389,73 @@ static EvolutionStats evolutionStats(
return new EvolutionStats(groupRowCount, finalMin, finalMax,
finalNullCounts);
}
+ @VisibleForTesting
+ static class EvolutionStatsCache {
+
+ private final Map<Triple<Long, List<String>, List<String>>,
ProjectedFileSchema> cache =
+ new HashMap<>();
+
+ private ProjectedFileSchema get(
+ Function<Long, TableSchema> scanTableSchema, DataFileMeta
fileMeta) {
+ Triple<Long, List<String>, List<String>> key =
+ Triple.of(fileMeta.schemaId(), fileMeta.writeCols(),
fileMeta.valueStatsCols());
+ return cache.computeIfAbsent(key, ignored ->
projectFileSchema(scanTableSchema, key));
+ }
+
+ @VisibleForTesting
+ int size() {
+ return cache.size();
+ }
+
+ private static ProjectedFileSchema projectFileSchema(
+ Function<Long, TableSchema> scanTableSchema,
+ Triple<Long, List<String>, List<String>> key) {
+ TableSchema dataFileSchema =
scanTableSchema.apply(key.f0).project(key.f1);
+ TableSchema dataFileSchemaWithStats =
dataFileSchema.project(key.f2);
+ List<DataField> fields = dataFileSchema.fields();
+ Map<Integer, Integer> fieldIdToStatsIndex = new
HashMap<>(fields.size() * 2);
+ for (DataField field : fields) {
+ fieldIdToStatsIndex.put(field.id(), NO_STATS_FIELD_INDEX);
Review Comment:
Could we avoid encoding three domain states (null = field absent, -1 = field
present without stats, and >= 0 = stats position) in a Map<Integer, Integer>? A
typed value such as FileFieldStats.withoutStats() / withStats(index, type)
would preserve the single O(1) lookup, keep the stats index and type together,
and confine integer sentinels to the DataEvolutionRow / DataEvolutionArray
boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
