etseidl opened a new issue, #8818:
URL: https://github.com/apache/arrow-rs/issues/8818
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
The page indexes in `ParquetMetaData` are currently defined as
```rust
pub type ParquetColumnIndex = Vec<Vec<ColumnIndexMetaData>>;
pub type ParquetOffsetIndex = Vec<Vec<OffsetIndexMetaData>>;
```
The trouble is that the indexes are optional per column chunk.
```rust
pub struct ColumnChunkMetaData {
...
offset_index_offset: Option<i64>,
offset_index_length: Option<i32>,
column_index_offset: Option<i64>,
column_index_length: Option<i32>,
...
}
```
This often leads to issues when dealing with partially missing indexes. For
the column index, we currently have an enum variant to handle a missing index
(`ColumnIndexMetaData::NONE`). But the offset index has no such structure, so a
missing offset index can sometimes lead to a panic.
**Describe the solution you'd like**
I think the index structures should mirror the `ColumnChunkMetaData` and
allow for `None` at the leaf level. That is, redefine the page index structures
to be
```rust
pub type ParquetColumnIndex = Vec<Vec<Option<ColumnIndexMetaData>>>;
pub type ParquetOffsetIndex = Vec<Vec<Option<OffsetIndexMetaData>>>;
```
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features
you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
