Liu Shaohui created KYLIN-4048:
----------------------------------

             Summary: Too long spark cube building time for too many eviction 
and loading for dict slices
                 Key: KYLIN-4048
                 URL: https://issues.apache.org/jira/browse/KYLIN-4048
             Project: Kylin
          Issue Type: Improvement
    Affects Versions: v2.5.2
            Reporter: Liu Shaohui


In our cluster, a cube building costs too long. In the log for spark, we found 
there are too many eviction and loading for dict slices in AppendTrieDictionary.
{code:java}
$ grep "read slice from" spark.log | wc -l
119721
$ grep "Evict slice with key" spark.log| wc -l
119634
{code}
The reason is that the memory of spark executor(4G) is not enough to hold all 
the slices of dict(3.3G in hdfs) in memory, which cause the bad performance of 
cube building.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to