the-other-tim-brown commented on code in PR #951:
URL: https://github.com/apache/incubator-xtable/pull/951#discussion_r4117995865
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiFileStatsExtractor.java:
##########
@@ -201,8 +201,16 @@ private Optional<InternalDataFile>
tryEnrichWithMetadataStats(
fileStats.stream()
.map(pair -> getColumnStatFromHudiStat(pair.getLeft(),
pair.getRight()))
.collect(CustomCollectors.toList(fileStats.size()));
- long recordCount = getMaxFromColumnStats(columnStats).orElse(0L);
- return
Optional.of(file.toBuilder().columnStats(columnStats).recordCount(recordCount).build());
+ Optional<Long> recordCount = getMaxFromColumnStats(columnStats);
+ if (!recordCount.isPresent()) {
+ // The metadata table has an entry for this file, but every column's
value count is
+ // absent or non-positive (observed for files written before
column-stats indexing was
+ // enabled). Treat this the same as "no stats" instead of emitting
recordCount=0, so the
+ // caller falls back to reading the authoritative row count from the
Parquet footer.
+ return Optional.empty();
+ }
+ return Optional.of(
+
file.toBuilder().columnStats(columnStats).recordCount(recordCount.get()).build());
Review Comment:
Nitpick: you could simplify this to:
```
getMaxFromColumnStats(columnStats).map(recordCount ->
file.toBuilder().columnStats(columnStats).recordCount(recordCount).build());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
