----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10795/#review19770 -----------------------------------------------------------
Ship it! Ship It! - Ashutosh Chauhan On April 26, 2013, 11:25 a.m., Gopal V wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/10795/ > ----------------------------------------------------------- > > (Updated April 26, 2013, 11:25 a.m.) > > > Review request for hive, Ashutosh Chauhan and Gunther Hagleitner. > > > Description > ------- > > Speed up RCFile::sync() by reading large blocks of data from HDFS rather than > using readByte() on the input stream. > > This improves the loop behaviour and reduces the number of calls on the > synchronized read() methods within HDFS, resulting in a 10x performance boost > to this function. > > In real time, it converts a call that takes upto a second and brings it below > 100ms, by reading 512 byte chunks instead of reading data 1 byte at a time. > > > This addresses bug HIVE-4423. > https://issues.apache.org/jira/browse/HIVE-4423 > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java d3d98d0 > > Diff: https://reviews.apache.org/r/10795/diff/ > > > Testing > ------- > > ant test -Dtestcase=TestRCFile -Dmodule=ql > ant test -Dtestcase=TestCliDriver -Dqfile_regex=.*rcfile.* -Dmodule=ql > > And benchmarking with count(1) on the store_sales rcfile table at scale=10 > > before: 43.8, after: 39.5 > > > Thanks, > > Gopal V > >
