cshuo commented on code in PR #12967:
URL: https://github.com/apache/hudi/pull/12967#discussion_r2007214858
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java:
##########
@@ -267,35 +268,42 @@ public List<HoodieColumnRangeMetadata<Comparable>>
readColumnStatsFromMetadata(H
StoragePath filePath,
List<String> columnList) {
ParquetMetadata metadata = readMetadata(storage, filePath);
+ return readColumnStatsFromMetadata(metadata, filePath.getName(),
Option.of(columnList));
+ }
+ public List<HoodieColumnRangeMetadata<Comparable>>
readColumnStatsFromMetadata(ParquetMetadata metadata, String filePath,
Option<List<String>> columnList) {
// Collect stats from all individual Parquet blocks
Stream<HoodieColumnRangeMetadata<Comparable>>
hoodieColumnRangeMetadataStream =
metadata.getBlocks().stream().sequential().flatMap(blockMetaData ->
blockMetaData.getColumns().stream()
- .filter(f ->
columnList.contains(f.getPath().toDotString()))
- .map(columnChunkMetaData -> {
- Statistics stats = columnChunkMetaData.getStatistics();
- return (HoodieColumnRangeMetadata<Comparable>)
HoodieColumnRangeMetadata.<Comparable>create(
- filePath.getName(),
- columnChunkMetaData.getPath().toDotString(),
- convertToNativeJavaType(
- columnChunkMetaData.getPrimitiveType(),
- stats.genericGetMin()),
- convertToNativeJavaType(
- columnChunkMetaData.getPrimitiveType(),
- stats.genericGetMax()),
- // NOTE: In case when column contains only nulls
Parquet won't be creating
- // stats for it instead returning stubbed
(empty) object. In that case
- // we have to equate number of nulls to the
value count ourselves
- stats.isEmpty() ?
columnChunkMetaData.getValueCount() : stats.getNumNulls(),
- columnChunkMetaData.getValueCount(),
- columnChunkMetaData.getTotalSize(),
- columnChunkMetaData.getTotalUncompressedSize());
- })
+ .filter(f -> !columnList.isPresent() ||
columnList.get().contains(f.getPath().toDotString()))
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
