[
https://issues.apache.org/jira/browse/HIVE-29737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-29737:
----------------------------------
Description:
Iceberg tables support writing Parquet bloom filters via table properties
(`write.parquet.bloom-filter-enabled.column.<col>`,
`write.parquet.bloom-filter-fpp.column.<col>`), and Hive inserts honor them —
the written files contain working bloom filters.
However, they are only consulted on the non-vectorized read path, where
`IcebergRecordReader` goes through Iceberg's `Parquet.read().filter(residual)`
and ReadConf applies `ParquetBloomRowGroupFilter`.
On
the vectorized path (the default, `hive.vectorized.execution.enabled=true`),
`HiveVectorizedReader#parquetRecordReader` hands the split to Hive's
`VectorizedParquetInputFormat,` bypassing Iceberg's reader. Row group pruning
then happens in ParquetRecordReaderBase#getSplit
{code:java}
filteredBlocks = RowGroupFilter.filterRowGroups(filter, splitGroup,
fileMetaData.getSchema());
{code}
was:
Iceberg tables support writing Parquet bloom filters via table properties
(`write.parquet.bloom-filter-enabled.column.<col>`,
`write.parquet.bloom-filter-fpp.column.<col>`), and Hive inserts honor them —
the written files contain working bloom filters.
However, they are only consulted on the non-vectorized read path, where
`IcebergRecordReader` goes through Iceberg's `Parquet.read().filter(residual)`
and ReadConf applies `ParquetBloomRowGroupFilter`.
On
the vectorized path (the default, `hive.vectorized.execution.enabled=true`),
`HiveVectorizedReader#parquetRecordReader` hands the split to Hive's
`VectorizedParquetInputFormat,` bypassing Iceberg's reader. Row group pruning
then happens in ParquetRecordReaderBase#getSplit
{code}
filteredBlocks = RowGroupFilter.filterRowGroups(filter, splitGroup,
fileMetaData.getSchema());
{code}
> Iceberg: Vectorized Parquet reader does not use bloom filters for row group
> pruning
> -----------------------------------------------------------------------------------
>
> Key: HIVE-29737
> URL: https://issues.apache.org/jira/browse/HIVE-29737
> Project: Hive
> Issue Type: Task
> Components: Iceberg integration
> Affects Versions: 4.2.0
> Reporter: Denys Kuzmenko
> Priority: Major
>
> Iceberg tables support writing Parquet bloom filters via table properties
> (`write.parquet.bloom-filter-enabled.column.<col>`,
> `write.parquet.bloom-filter-fpp.column.<col>`), and Hive inserts honor them —
> the written files contain working bloom filters.
> However, they are only consulted on the non-vectorized read path, where
> `IcebergRecordReader` goes through Iceberg's
> `Parquet.read().filter(residual)` and ReadConf applies
> `ParquetBloomRowGroupFilter`.
>
>
> On the vectorized path (the default,
> `hive.vectorized.execution.enabled=true`),
> `HiveVectorizedReader#parquetRecordReader` hands the split to Hive's
> `VectorizedParquetInputFormat,` bypassing Iceberg's reader. Row group pruning
> then happens in ParquetRecordReaderBase#getSplit
>
> {code:java}
> filteredBlocks = RowGroupFilter.filterRowGroups(filter, splitGroup,
> fileMetaData.getSchema());
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
