Thanks George:)
Best Regards! --------------------- Luke Han On Sun, Sep 6, 2015 at 5:08 PM, nichunen <[email protected]> wrote: > Hi Luke, > > OK, I'll test kylin with Snappy and write a document if get results > worth recording. > > ------------------------------ > > Best Regards, > > > > George/倪春恩 > > Software Engineer/软件工程师 > > Mobile:+86-13501723787| Fax:+8610-56842040 > > 北京明略软件系统有限公司（www <http://www.semidata.com/>.mininglamp.com） > > 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层 > > F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping > District,Beijing,102218 > > > ---------------------------------------------------------------------------------------------------------------------------- > > [image: cid:[email protected]] > > > *From:* Luke Han <[email protected]> > *Date:* 2015-09-06 14:33 > *To:* [email protected] > *Subject:* Re: Tests with lzo compression in kylin > Hi George, > Could you also please try to test with Snappy like Meng's comments? > Would like to see more detail comparison here with different > compression library. > > BTW, could you please draft a blog for this? Refer to website/_posts > for more detail. > Thanks. > > > Best Regards! > --------------------- > > Luke Han > > On Sun, Sep 6, 2015 at 9:56 AM, [email protected] <[email protected]> > wrote: > > > i use snappy instead of lzo, and found that the time of cube building > > increased, without snappy: less than 4hours, with snappy: about > 5hours(more > > time spent on converting results to hfiles), and with compression, the > cube > > size decrease about 25%-30%, i hadn't test the query performance yet, the > > table include about 6 billions records and 8 dimensions; > > > > > > > > 中国移动广东有限公司 网管中心 梁猛 > > [email protected] > > > > 发件人： nichunen > > 发送时间： 2015-09-06 09:36 > > 收件人： dev > > 主题： Tests with lzo compression in kylin > > Hi, > > > > I have made some tests on our cluster after hadoop lzo installed and lzo > > enabled in kylin. Kylin has better performance with LZO. > > > > I build cubes with two tables, small one with 10,000 records(table called > > Small_Table), and large one with 4,000,000 records(table called > > Large_Table). > > > > The cube sizes are reduced obviously. > > > > Large_Table > > Small_Table > > No LZO > > 776.33m > > 16.15m > > LZO > > 571.49m > > 7.53m > > > > For the query duration time is not quite stable, I made comparation with > a > > time-consuming query on kylin with and without lzo. The query seems like > > "SELECT A,B from Large_Table where A<'5000000000' and B>'5000000000' > group > > by A,B order by A;" > > On Kylin with lzo, I queried for 10 times, the time durings were: > > 4.80s,5.74s,5.98s,4.95s,4.86s,7.24s,4.72s,6.80s,6.42s,7.08s > > The mean time was 5.859s. > > > > On Kylin without lzo, I queried for 10 times, the time durings were: > > 11.66s,6.31s,7.17s,6.37s,6.78s,6.43s,7.47s,5.62s,7.60s,6.47s > > The mean time was 7.188s. > > > > For the time of cube building, I didn't see much improvement, maybe this > > is because I didn't build many times and do not have more accurate > > comparations. > > Could you please share your experience about Kylin with lzo? > > > > Tnanks > > > > > > > > Best Regards, > > > > George/倪春恩 > > Software Engineer/软件工程师 > > Mobile:+86-13501723787| Fax:+8610-56842040 > > 北京明略软件系统有限公司（www.mininglamp.com） > > 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层 > > F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping > > District,Beijing,102218 > > > > > ---------------------------------------------------------------------------------------------------------------------------- > > > >
