[
https://issues.apache.org/jira/browse/PARQUET-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709932#comment-14709932
]
Ryan Blue commented on PARQUET-344:
-----------------------------------
I see the need to address the situation where way too many records fit in a
single row group and start causing individual tasks to take forever because all
of that data is can't be processed in parallel. But I'm not sure how we would
handle a few things:
* Would we expose a maximum rows per row-group setting? I could see that being
misused really easily.
* If we don't expose a maximum rows setting, how would we configure this?
* Because the row groups are getting great compression, it's safe to assume
that if we did break a row group at some number of records, we would still have
a single HDFS block. That conflicts with the current practice of not reading
the row group offsets from the file footer to avoid a huge job startup cost. It
would be best to stop the HDFS block after the first row group, so how can we
make that happen?
> Limit the number of rows per block and per split
> ------------------------------------------------
>
> Key: PARQUET-344
> URL: https://issues.apache.org/jira/browse/PARQUET-344
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Reporter: Quentin Francois
>
> We use Parquet to store raw metrics data and then query this data with
> Hadoop-Pig.
> The issue is that sometimes we end up with small Parquet files (~80mo) that
> contain more than 300 000 000 rows, usually because of a constant metric
> which results in a very good compression. Too good. As a result we have a
> very few number of maps that process up to 10x more rows than the other maps
> and we lose the benefits of the parallelization.
> The fix for that has two components I believe:
> 1. Be able to limit the number of rows per Parquet block (in addition to the
> size limit).
> 2. Be able to limit the number of rows per split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
