Anthony Pessy created PARQUET-1602:
--------------------------------------
Summary: PageIndex not working as suggested ?
Key: PARQUET-1602
URL: https://issues.apache.org/jira/browse/PARQUET-1602
Project: Parquet
Issue Type: Improvement
Components: parquet-mr
Affects Versions: 1.11.0
Reporter: Anthony Pessy
I have a schema such as:
```
{{schema message pages {}}
{{ required binary url (STRING);}}
{{ optional binary content (STRING);}}
{{}}}
```
Where `url` is unique and ordered, the file is created in such a way that I
have ~600 pages of `content` for 1 page of `url`.
From [https://github.com/apache/parquet-format/blob/master/PageIndex.md] I saw:
> A single-row lookup in a rowgroup based on the sort column of that rowgroup
>will only read one data page per retrieved column.
I was expecting `ParquetReader` to find the matching row thanks to the
`FilterPredicate` on `url`, decoding only this column, then, using `offset
index`, directly seek to the appropriate page for `content` and decode it.
Instead, what I'm seeing, is that the reader fully reads & decode the ~600
pages of content (until it actually find the url).
Is there something I misunderstood or some step to ensure to make the reader
only consume the necessary pages?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
