There are a couple options for this: 1) use a Hive view to shade your wide lookup table, picking up only interested columns, and then use this view as lookup table in Cube. From 1.5.x Kylin starts to support view as lookup;
2) Kylin 2.0 supports using a big table as lookup; When create the model, you have an option of "not take snapshot", then Kylin will not load it to memory. 2017-07-07 9:04 GMT+08:00 flycshi <[email protected]>: > in DictionaryGeneratorCLI.class for method processSegment > to get lookup table by column > // snapshot > Set<String> toSnapshot = Sets.newHashSet(); > Set<TableRef> toCheckLookup = Sets.newHashSet(); > for (DimensionDesc dim : cubeSeg.getCubeDesc().getDimensions()) { > TableRef table = dim.getTableRef(); > if (*cubeSeg.getModel().isLookupTable(table)*) { > toSnapshot.add(table.getTableIdentity()); > toCheckLookup.add(table); > } > } > > when lookup table is larger, this step easily to failed to load lookup > table > > the judgement of here , whether can consider to add more judgement > for example, if the column is not a derived dimension and even if the > column > belong to a lookup table, and the lookup table will not to load in memory > > otherwise， every lookup table will load in memory，this lead to big lookup > table unable to use in kylin completely。 > > Looking forward to your reply，thanks. > > > -- > View this message in context: http://apache-kylin.74782.x6. > nabble.com/Lookup-Table-Enumerator-high-memory-tp1397p8384.html > Sent from the Apache Kylin mailing list archive at Nabble.com. > -- Best regards, Shaofeng Shi 史少锋
