laskoviymishka commented on code in PR #2597:
URL: https://github.com/apache/iceberg-go/pull/2597#discussion_r4596798759
##########
table/internal/parquet_files.go:
##########
@@ -1927,17 +1927,35 @@ func resolveParquetStatsColumns(meta
*metadata.FileMetaData, statsCols map[int]S
return columns
}
+// ParquetColumnStatsPresence reads the optional fields from the footer because
+// Arrow's typed statistics can report an absent null count as zero or a
+// one-sided bound as a min/max pair.
+func ParquetColumnStatsPresence(meta *metadata.FileMetaData, rowGroup, column
int) (nullCount, bounds bool) {
+ chunk := meta.RowGroups[rowGroup].Columns[column].MetaData
+ if chunk == nil || chunk.Statistics == nil {
+ return false, false
+ }
+ stats := chunk.Statistics
+ if meta.Schema.Column(column).ColumnOrder() ==
parquet.ColumnOrders.TypeDefinedOrder {
+ return stats.IsSetNullCount(), stats.IsSetMinValue() &&
stats.IsSetMaxValue()
+ }
+
+ return stats.IsSetNullCount(), stats.IsSetMin() && stats.IsSetMax()
Review Comment:
Could the import test also reach this deprecated `Min`/`Max` branch?
`writePartialMetricsParquet` writes through pqarrow, which sets `ColumnOrders`,
so only the `TypeDefinedOrder` branch runs
([parquet_import_metrics_test.go#L72-L90](https://github.com/apache/iceberg-go/blob/4ab989f69e894c7fce6fa9acd8d9c96c3d3c9708/table/parquet_import_metrics_test.go#L72-L90)).
A case with `ColumnOrders` cleared would cover it, and a string or binary
source column would help too, since every case uses the single Int32 `amount`.
##########
table/internal/utils.go:
##########
@@ -321,6 +329,10 @@ func (d *DataFileStatistics) ToDataFile(opts DataFileOpts)
iceberg.DataFile {
// prioritizing caller-supplied value.
fieldIDToPartitionData[field.FieldID] =
partitionVal
case field.Transform.PreservesOrder() &&
sourceFieldFound:
+ if _, incomplete :=
d.incompleteBounds[field.FirstSourceID()]; incomplete {
Review Comment:
Should the `!set` and `stats == nil` branches also put the column into
`incompleteBounds`, with a test next to
`TestPartialParquetBoundsPartitionInference`? Those branches only write
`invalidateCol`
([parquet_files.go#L2003-L2019](https://github.com/apache/iceberg-go/blob/4ab989f69e894c7fce6fa9acd8d9c96c3d3c9708/table/internal/parquet_files.go#L2003-L2019)),
so this guard never fires. With `Statistics = nil` on one row group or both,
`FileToDataFile` returns `partition[1000]=<nil>` and no error. `main` does the
same, but this PR is where partition inference starts requiring complete bounds.
##########
table/internal/utils.go:
##########
@@ -231,16 +231,24 @@ type DataFileStatistics struct {
NullValueCounts map[int]int64
NanValueCounts map[int]int64
ColAggs map[int]StatsAgg
+ incompleteBounds map[int]StatsAgg
Review Comment:
Could we document the two meanings this map carries? `ToDataFile` checks
only that the key is present, while `PartitionValue` reads the value, which is
nil when no row group had bounds
([parquet_files.go#L2079-L2083](https://github.com/apache/iceberg-go/blob/4ab989f69e894c7fce6fa9acd8d9c96c3d3c9708/table/internal/parquet_files.go#L2079-L2083)).
Switching the guard to a `!= nil` check would quietly drop it for those
columns.
```suggestion
// incompleteBounds holds source columns whose bounds are missing in at
least
// one row group. Key presence is what blocks partition inference; the
value
// is the partial aggregate from the other row groups and may be nil.
incompleteBounds map[int]StatsAgg
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]