[
https://issues.apache.org/jira/browse/PARQUET-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866645#comment-16866645
]
Anthony Pessy commented on PARQUET-1602:
----------------------------------------
{noformat}
Based on the offset index it is translated to the row range [0..19999].
Therefore, we need to load page-0 for the column url and all the pages are in
the row range [0..19999] for column content.{noformat}
Alright so it is indeed working as intended.
Could not find anywhere a basic algorithm on how the reader works but your
explanation was limpid, sorry for the lack of clarity on my part.
This is unfortunately not what I was hoping for because it loads lot's of
unnecessary data in my case.
{noformat}
I don't know why page-212 is loaded but page-213 is not. {noformat}
To be honest maybe I messed up the index, I tried to count the number of
'loaded pages' and/or check the 'pageIndex' in 'pageReader' but you clearly
explained how it works and it correlate what I was suspecting.
{noformat}
You may fine-tune it by decreasing the value of the property
parquet.page.row.count.limit from 20k so more pages will be created for the
column url.{noformat}
I'll try that!
{noformat}
Note: The (...) in the middle of the listed value is the mark of the truncation
at display time. Unfortunately, the tools display the min/max values with a
hardcoded maximum length of 40.{noformat}
Ok! May be nice to somehow see the full value (as opt-in maybe) since
apparently the value may be truncated.
Thanks for your time.
> PageIndex not working as suggested ?
> ------------------------------------
>
> Key: PARQUET-1602
> URL: https://issues.apache.org/jira/browse/PARQUET-1602
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.11.0
> Reporter: Anthony Pessy
> Priority: Major
>
> I have a schema such as:
> {code:java}
> schema message pages {
> required binary url (STRING);
> optional binary content (STRING);
> }
> {code}
> Where `url` is unique and ordered, the file is created in such a way that I
> have ~600 pages of `content` for 1 page of `url`.
>
> From [https://github.com/apache/parquet-format/blob/master/PageIndex.md] I
> saw:
> {noformat}
> A single-row lookup in a rowgroup based on the sort column of that rowgroup
> will only read one data page per retrieved column.{noformat}
> I was expecting `ParquetReader` to find the matching row thanks to the
> `FilterPredicate` on `url`, decoding only this column, then, using `offset
> index`, directly seek to the appropriate page for `content` and decode it.
> Instead, what I'm seeing, is that the reader fully reads & decode the ~600
> pages of content (until it actually find the url).
> Is there something I misunderstood or some step to ensure to make the reader
> only consume the necessary pages?
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
