Hi Luke, OK, I'll test kylin with Snappy and write a document if get results worth recording.
Best Regards, George/倪春恩 Software Engineer/软件工程师 Mobile:+86-13501723787| Fax:+8610-56842040 北京明略软件系统有限公司（www.mininglamp.com） 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层 F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping District,Beijing,102218 ---------------------------------------------------------------------------------------------------------------------------- From: Luke Han Date: 2015-09-06 14:33 To: [email protected] Subject: Re: Tests with lzo compression in kylin Hi George, Could you also please try to test with Snappy like Meng's comments? Would like to see more detail comparison here with different compression library. BTW, could you please draft a blog for this? Refer to website/_posts for more detail. Thanks. Best Regards! --------------------- Luke Han On Sun, Sep 6, 2015 at 9:56 AM, [email protected] <[email protected]> wrote: > i use snappy instead of lzo, and found that the time of cube building > increased, without snappy: less than 4hours, with snappy: about 5hours(more > time spent on converting results to hfiles), and with compression, the cube > size decrease about 25%-30%, i hadn't test the query performance yet, the > table include about 6 billions records and 8 dimensions; > > > > 中国移动广东有限公司 网管中心 梁猛 > [email protected] > > 发件人： nichunen > 发送时间： 2015-09-06 09:36 > 收件人： dev > 主题： Tests with lzo compression in kylin > Hi, > > I have made some tests on our cluster after hadoop lzo installed and lzo > enabled in kylin. Kylin has better performance with LZO. > > I build cubes with two tables, small one with 10,000 records(table called > Small_Table), and large one with 4,000,000 records(table called > Large_Table). > > The cube sizes are reduced obviously. > > Large_Table > Small_Table > No LZO > 776.33m > 16.15m > LZO > 571.49m > 7.53m > > For the query duration time is not quite stable, I made comparation with a > time-consuming query on kylin with and without lzo. The query seems like > "SELECT A,B from Large_Table where A<'5000000000' and B>'5000000000' group > by A,B order by A;" > On Kylin with lzo, I queried for 10 times, the time durings were: > 4.80s,5.74s,5.98s,4.95s,4.86s,7.24s,4.72s,6.80s,6.42s,7.08s > The mean time was 5.859s. > > On Kylin without lzo, I queried for 10 times, the time durings were: > 11.66s,6.31s,7.17s,6.37s,6.78s,6.43s,7.47s,5.62s,7.60s,6.47s > The mean time was 7.188s. > > For the time of cube building, I didn't see much improvement, maybe this > is because I didn't build many times and do not have more accurate > comparations. > Could you please share your experience about Kylin with lzo? > > Tnanks > > > > Best Regards, > > George/倪春恩 > Software Engineer/软件工程师 > Mobile:+86-13501723787| Fax:+8610-56842040 > 北京明略软件系统有限公司（www.mininglamp.com） > 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层 > F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping > District,Beijing,102218 > > ---------------------------------------------------------------------------------------------------------------------------- >
