[
https://issues.apache.org/jira/browse/PARQUET-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941808#comment-16941808
]
Stefan Becker commented on PARQUET-1670:
----------------------------------------
[~gszadovszky]: thanks for your quick reply!
I am a colleague of Alex and wanted to share my thoughts as well :)
We have the current setup:
* Workstation with multicore 3GHZ CPU and 32GB RAM
* 100 small 1 mb parquet files
* We use 1.12-SNAPSHOT
* We run the following command:
{code:java}
java -Xmx26G -jar
/opt/apache/parquet-mr/parquet-tools/target/parquet-tools-1.12.0-SNAPSHOT.jar
merge -b big_part*.parquet big_total.parqet {code}
The results are:
* One core is 100% busy
* We consume up to 20GB of RAM, NO swapping
* The command above produces one ~80mb parquet file
So far so good. I understand what you say: the tool needs to read back all the
data, decodes it, encodes it and writes it back to disk. But still:
* We only need to read 100mb (100x1mb) of data from disk
* We only write 80mb to disk
What the heck is he really doing for 30 minutes? :) From my experience those
numbers do not fit at all. I do not know the solution but I think I know that
something is broken somewhere which does not have to do anything with the
way/logic things should work.
Either:
* we did something wrong
* there is a bug in the implemention that it does more work than it needs to do
* it uses some crazy algos where it is trying to compress/merge stuff in a
"multi step process" where it goes crazy
Currently we build our own system where we just use parquet files and some
other custom build software. We do not use Spark or Hive or anything like that.
I dont get it ...
> parquet-tools merge extremely slow with block-option
> ----------------------------------------------------
>
> Key: PARQUET-1670
> URL: https://issues.apache.org/jira/browse/PARQUET-1670
> Project: Parquet
> Issue Type: Bug
> Reporter: Alexander Gunkel
> Priority: Major
>
> parquet-tools merge is extremely time- and memory-consuming when used with
> block-option.
>
> The merge function builds a bigger file out of several smaller parquet-files.
> Used without the block-option it just concatenates the files into a bigger
> one without building larger row-groups. That doesn't help with
> query-performance-issues. With block-option, parquet-tools build bigger
> row-groups which improves the query-performance, but the merge-process itself
> is extremely slow and memory-consuming.
>
> Consider a case in which you have many small parquet files, e.g. 1000 files
> with a size of 100kb. Merging them into one file fails on my machine because
> even 20GB of memory are not enough for the process (the total amount of data
> as well as the resulting file should be smaller than 100MB).
>
> Different situation: Consider having 100 files of size 1MB. Then merging them
> is possible with 20GB of RAM, but it takes almoust half an hour to process,
> which is to much for many use-cases.
>
> Is there any possibility to accelerate the merge and reduce the need of
> memory?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)