Hi, dev: Recently I found the bug in compressing sort temp file and tried to
fix this bug in PR#1632 (https://github.com/apache/carbondata/pull/1632). In
this PR, Carbondata will compress the records in batch and write the compressed
content to file if we turn on this feature. However, I found that the GC
performance is terrible. In my scenario, about half of the time were wasted in
GC. And the overall performance is worse than before. I think the problem may
lie in compressing the records by batch. Instead of this, I propose to compress
the sort temp file in file level, not in record-batch level. 1. Compared with
uncompressed ones, compressing the file in record-batch level leads to
different layout of file. And it also affects the reading/writing behavior.
(The compressed:
|total_entry_number|batch_entry_numer|compressed_length|compressed_content|batch_entry_numer|compressed_length|compressed_content|...;
The uncompressed: |total_entry_number|record|record|...;) 2. During
compressing/uncompressing the record-batch, we have to store the bytes in
temporary memory. If the size is big, it directly goes into JVM old generation,
which will cause FULL GC frequently. I also tried to reuse this temporary
memory, but it can only be reusable in file level -- We need to allocate the
memory for each file. If the number of intermediate files are big, frequent
FULL GC is still inevitable. If the size is small, we will need to store more
`batch_entry_numer`(described in point1). Note that, the size is
rowSize*batchSize. In previous implementation, Carbondata use 2MB bytes to
store one row. 3. Using file level compression will simply the code since
CompressedStream is also an Stream, which will not affect the behavior in
reading/writing compressed/uncompressed files. 4. After I used file level
compression, the GC problem disappeared. Since my cluster has crashed, I didn't
get the actual performace enhanced. But seeing from the Carbondata maven tests,
the most time consuming module `Spark Common Test` takes less time to complete
comparing with uncompressed. Time consumed in `Spark Common Test` module: |
Compressor | Time Consumed | | --- | --- | | None | 19:25min | | SNAPPY |
18:38min | | LZ4 | 19:12min | | GZIP | 20:32min | | BZIP2 | 21:10min | In
conclusion, I think file level compression is better and I plan to remove the
record-batch leve compression related code in Carbondata. 发自网易邮箱手机版
