[
https://issues.apache.org/jira/browse/IMPALA-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840447#comment-16840447
]
Csaba Ringhofer commented on IMPALA-6964:
-----------------------------------------
I have not noticed during the implementation of IMPALA-6433 that only data
pages are tracked, but not dictionary pages. The difference can be pretty large
for string columns or if column indexes filter out most data pages.
> Track stats about column and page sizes in Parquet reader
> ---------------------------------------------------------
>
> Key: IMPALA-6964
> URL: https://issues.apache.org/jira/browse/IMPALA-6964
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Sahil Takiar
> Priority: Major
> Labels: observability, parquet, ramp-up
> Fix For: Impala 3.2.0
>
>
> It would be good to have stats for scanned parquet data about page sizes. We
> currently can't tell much about the "shape" of the parquet pages from the
> profile. Some questions that are interesting:
> * How big is each column? I.e. total compressed and decompressed size read.
> * How big are pages on average? Either compressed or decompressed size
> * What is the compression ratio for pages? Could be inferred from the above
> two.
> I think storing all the stats in the profile per-column would be too much
> data, but we could probably infer most useful things from higher-level
> aggregates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
