Liu Shaohui created KYLIN-4048:
----------------------------------
Summary: Too long spark cube building time for too many eviction
and loading for dict slices
Key: KYLIN-4048
URL: https://issues.apache.org/jira/browse/KYLIN-4048
Project: Kylin
Issue Type: Improvement
Affects Versions: v2.5.2
Reporter: Liu Shaohui
In our cluster, a cube building costs too long. In the log for spark, we found
there are too many eviction and loading for dict slices in AppendTrieDictionary.
{code:java}
$ grep "read slice from" spark.log | wc -l
119721
$ grep "Evict slice with key" spark.log| wc -l
119634
{code}
The reason is that the memory of spark executor(4G) is not enough to hold all
the slices of dict(3.3G in hdfs) in memory, which cause the bad performance of
cube building.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
