Thank you, ShaoFeng for response ! I have learned many internals of Kylin with this mail thread. I appreciate your help.
Regards, Shrikant Bang. On Tue, Aug 28, 2018 at 7:14 PM ShaoFeng Shi <[email protected]> wrote: > Hi Shrikant, > > Do you mean reusing the lookup table snapshot across cubes? As you know, > Kylin took snapshots for lookup table and load them to memory during at > query time. > > When Kylin took a snapshot, it will check whether the lookup table was > changed since last time. If no change, it will reuse the snapshot. The > detail logic is in SnapshotManager.buildSnapshot(); > > So, from this point of view, if your calendar lookup table is stable, it > will be reused by multiple cubes. > > Hope this helps; > > 2018-08-28 17:04 GMT+08:00 Shrikant Bang <[email protected]>: > >> Thank you, ShaoFeng for response! >> >> Apart from UHC, we have other dimension which will be used by multiple >> cubes. >> >> e.g. calendar_dimension ( date, day, week, week, month, quarter .... etc >> etc ) which immutable. >> >> Few of calendar's dimension become part of cube and few become derived >> columns. >> >> Is there any way I can cache in Kylin's node and keep using it every >> other cube? It will be kind of global cache for all cubes under a project. >> >> Thank You, >> Shrikant Bang. >> >> >> >> On Tue, Aug 28, 2018 at 9:05 AM ShaoFeng Shi <[email protected]> >> wrote: >> >>> >>> Will you recommend using "integer" type for UHC (3+ millions) dimension >>> and then have derived columns for relative dimensions (look-ups) where type >>> is not "integer"? >>> >> This depends on the cardinality of the two columns. For example, >>> "user_id" and "email", they are close to 1:1, so this derivation is good. >>> But "user_id" and "sex" is not good because "sex"'s cardinality is much >>> smaller than "user_id", which means lots of post-aggregation will happen >>> after the derivation. Usually, we suggest the relationship is less or >>> around 10:1, but this is not fixed, you can select depends on the >>> performance requirement. >>> >>> Is derived column's aggregation happens at HBase Co-Processor side? Any >>> JIRA/doc for my learnings? >>> >> No, derivation calculation only happens in Kylin node, won't be >>> pushed down. Because Lookup table's snapshot is only loaded in Kylin node. >>> >>> 2018-08-27 19:00 GMT+08:00 Shrikant Bang <[email protected]>: >>> >>>> Thanks, ShaoFeng for response! >>>> >>>> I have started using memory 2G (default cluster setting) and OOM got >>>> solved when memory increased to 4G. >>>> >>>> Will you recommend using "integer" type for UHC (3+ millions) dimension >>>> and then have derived columns for relative dimensions (look-ups) where type >>>> is not "integer"? >>>> >>>> Is derived column's aggregation happens at HBase Co-Processor side? Any >>>> JIRA/doc for my learnings? >>>> >>>> please suggest. >>>> >>>> Thank You, >>>> Shrikant Bang >>>> >>>> On Tue, Aug 21, 2018 at 6:36 PM ShaoFeng Shi <[email protected]> >>>> wrote: >>>> >>>>> Hi Shrikant, >>>>> >>>>> How much memory are you allocating to Reducer? Please consider to >>>>> allocate more mem to reducer, as Kylin builds the dictionary in the >>>>> reducers. >>>>> >>>>> You can also disable this, then Kylin will build dict in its own JVM. >>>>> This may cause your Kylin process OOM if there is an ultra high >>>>> cardinality >>>>> (UHC) column. >>>>> >>>>> kylin.engine.mr.build-dict-in-reducer=false >>>>> >>>>> >>>>> Do you know how high the cardinality of that dimension? For UHC which >>>>> cardinality > 3 millions, we don't recommend to use dictionary as the >>>>> encoding. You may need to use "fixed_length" or "integer"(if it is in >>>>> type of integer). >>>>> >>>>> >>>>> 2018-08-16 16:50 GMT+08:00 Ashish Singhi <[email protected]>: >>>>> >>>>>> Hi Shrikant, >>>>>> >>>>>> Refer http://kylin.apache.org/blog/2015/08/13/kylin-dictionary/ >>>>>> You might find it useful. >>>>>> >>>>>> Regards, >>>>>> Ashish >>>>>> >>>>>> On Thu, Aug 16, 2018 at 10:33 AM, Shrikant Bang < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Thank you, ShaoFeng & Billy for responses. >>>>>>> >>>>>>> I could able to set hierarchies in dimension. >>>>>>> >>>>>>> While building cube, step "fact distinct column" job is failing in a >>>>>>> reducer with Out Of Memory exception. >>>>>>> >>>>>>> java.lang.OutOfMemoryError: Java heap space >>>>>>> at java.util.IdentityHashMap.resize(IdentityHashMap.java:471) >>>>>>> at java.util.IdentityHashMap.put(IdentityHashMap.java:440) >>>>>>> at >>>>>>> org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:476) >>>>>>> at >>>>>>> org.apache.kylin.dict.TrieDictionaryBuilder.build(TrieDictionaryBuilder.java:418) >>>>>>> at >>>>>>> org.apache.kylin.dict.TrieDictionaryForestBuilder.build(TrieDictionaryForestBuilder.java:109) >>>>>>> at >>>>>>> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.build(DictionaryGenerator.java:220) >>>>>>> at org.apache.kylin.engine.mr >>>>>>> .steps.FactDistinctColumnsReducer.doCleanup(FactDistinctColumnsReducer.java:216) >>>>>>> at org.apache.kylin.engine.mr >>>>>>> .KylinReducer.cleanup(KylinReducer.java:103) >>>>>>> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179) >>>>>>> at >>>>>>> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) >>>>>>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) >>>>>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>> at >>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >>>>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >>>>>>> >>>>>>> >>>>>>> I tried debugging and understood that dictionary is getting built in >>>>>>> reducer's clean up method. >>>>>>> >>>>>>> I am curious to learn internals. Can you please help me in below : >>>>>>> >>>>>>> 1. Any pointer/reference/JIRA for understanding how TRIE >>>>>>> (dictionary) of dimension's value getting used in next steps? >>>>>>> >>>>>>> 2. Any best practice/references in tuning "fact distinct column" >>>>>>> job for those reducer which have high cardinality. I am trying with >>>>>>> increasing memory as of now as partitioning and number of reducers are >>>>>>> depends on cuboids number. >>>>>>> >>>>>>> >>>>>>> P.S. I am using v2.4 of Kylin with HBase 1.x >>>>>>> >>>>>>> Thank You, >>>>>>> Shrikant Bang >>>>>>> >>>>>>> On Tue, Aug 14, 2018 at 8:33 PM ShaoFeng Shi <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> For question 1), in Cube's "advanced setting" step, you can specify >>>>>>>> the cuboid whitelist to build. >>>>>>>> >>>>>>>> 2018-08-13 22:26 GMT+08:00 Billy Liu <[email protected]>: >>>>>>>> >>>>>>>>> Hello Shrikant, >>>>>>>>> >>>>>>>>> For 1, seems the 4 dimensions are hierarchy structure. You could >>>>>>>>> define them as hierarchy dimensions in Cube, and leave A as >>>>>>>>> mandatory >>>>>>>>> dimension. >>>>>>>>> >>>>>>>>> For 2, select 'user_activity' as partition column in model design. >>>>>>>>> There are a few built-in formats, most date types are supported. >>>>>>>>> >>>>>>>>> With Warm regards >>>>>>>>> >>>>>>>>> Billy Liu >>>>>>>>> Shrikant Bang <[email protected]> 于2018年8月13日周一 下午5:39写道： >>>>>>>>> > >>>>>>>>> > Hi Team, >>>>>>>>> > >>>>>>>>> > We are doing a PoC on building OLAP cubes. Could you please >>>>>>>>> help me to get answer of below queries? >>>>>>>>> > >>>>>>>>> > Selective Cuboids: >>>>>>>>> > We need to have selective cuboids as part of OLAP cubes. >>>>>>>>> > Let say if we have 4 dimensions : A, B, C, D then we need just >>>>>>>>> (A,B,C,D) , (A,B,C), (A,B) and (A) >>>>>>>>> > >>>>>>>>> > Refresh Settings: >>>>>>>>> > How to specify partition column and format while building cube >>>>>>>>> for fact table. >>>>>>>>> > e.g. user_activity is partitioned by date 'yyyy-MM-dd' and cube >>>>>>>>> should be refreshed everyday with previous day's computation. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Thank You, >>>>>>>>> > Shrikant Bang >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Shaofeng Shi 史少锋 >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> >>>>> Shaofeng Shi 史少锋 >>>>> >>>>> >>> >>> >>> -- >>> Best regards, >>> >>> Shaofeng Shi 史少锋 >>> >>> > > > -- > Best regards, > > Shaofeng Shi 史少锋 > >
