Dictionary will be loaded into Kylin's memory, so based on your JVM heap configuration, the dictionary couldn't be very big.
Usually, the dictionary can only support dimension whose cardinality is less than 5 million (smaller is better). For larger cardinality, please use "fixed_length" encoding or "integer" encoding (if the column is in int format). Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Email: [email protected] Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: [email protected] Join Kylin dev mail group: [email protected] maoxiaomao <[email protected]> 于2019年8月29日周四 下午3:02写道： > Hi, > I building a MERGE job, and it got the following error when Step1. > Merge Dictionary. I merge 3 segment ,and my dict file size is 185.7M*3 > (557M)，it's really larger than normal. > I wonder: > Is there any size limit for dictionary？ > How can I make it's size smaller? > env: > Kylin 2.6.1 > > Error: java.lang.OutOfMemoryError: Java heap space > at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:341) > at > org.apache.kylin.dict.TrieDictionaryForest.readFields(TrieDictionaryForest.java:234) > at > org.apache.kylin.dict.DictionaryInfoSerializer.deserialize(DictionaryInfoSerializer.java:80) > at > org.apache.kylin.dict.DictionaryInfoSerializer.deserialize(DictionaryInfoSerializer.java:35) > at > org.apache.kylin.common.persistence.ContentReader.readContent(ContentReader.java:40) > at > org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:269) > at > org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256) > at > org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397) > at > org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80) > at > org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77) > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) > at com.google.common.cache.LocalCache.get(LocalCache.java:4000) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874) > at > org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryMapper.doMap(MergeDictionaryMapper.java:103) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryMapper.doMap(MergeDictionaryMapper.java:66) > at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > > > > > > -- > [image: 点击参加抽奖] > <http://pepsi2009.163.com/validsign.php?valide=0ba9e7477bf1486aad82350cf9504362&[email protected]> > > > >
