fengYu created KYLIN-2929:
-----------------------------
Summary: speed up Dump file performance
Key: KYLIN-2929
URL: https://issues.apache.org/jira/browse/KYLIN-2929
Project: Kylin
Issue Type: Bug
Reporter: fengYu
Assignee: fengYu
when I work on KYLIN-2926, I find coprocessor will dump to disk once
estimatedMemSize is bigger than spillThreshold, and found that spill data size
is extraordinary smaller that estimatedMemSize, in my case dump file size is
about 8MB and spillThreshold is setting to 3GB.
So, I try to keep the spill data in memory rather than write the file to disk
immediately, and when those in-memory spill data reach the threshold, write all
spill files together.
In my case, the coprocessor process cost time drop from 22s to 16s, it is about
30% upgrade。
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
