+1. Very insight answer. With Warm regards
Billy Liu Ma Gang <[email protected]> 于2018年5月30日周三 下午3:58写道： > > Hi, > > The number of reducers in 'fact distinct columns' step is calculated as: > {number of normal columns need to build dict} + {UHC columns number * UHC > reducer count} + {number of cuboid row counters} + 1 > the UHC reducer number can be configured by > "kylin.engine.mr.uhc-reducer-count". > > You need to identify which reducer is OOM, and find log in the reducer side > to identify the responsibility of the reducer. Usually the OOM caused by > column's cardinality is too high, and use dict encoding for that column. > Currently there are two solutions for this issue: > 1. use other encoding method instead of dict for ultra high cardinality > column. > 2. increase the memory of reducer(only help when the cardinality is not very > high...) > > > At 2018-05-30 15:09:03, "陈星宇" <[email protected]> wrote: > >hi, > >when i use kylin 2.3.1 build huge cube, got error at 'Extract Fact Table > >Distinct Columns', > >reduce job fail because of Java heap space, i suspect the num of reducer is > >too less, so tried many parameter to increase reducer, but is not working, > >see my parameter as below: > > > > > > > >kylin.engine.mr.mapper-input-rows 200000 > >kylin.engine.mr.min-reducer-number 26 > >kylin.storage.hbase.hfile-size-gb 0.5 > >kylin.storage.hbase.min-region-count 28 > >kylin.storage.hbase.region-cut-gb 0.5 > >kylin.query.max-return-rows 7000000 > >kylin.engine.mr.uhc-reducer-count 30 > >kylin.storage.hbase.max-region-count 500 > > > > > > > >is any suggestion for this? > > > > > >thanks > > > > > >chenxingyu
