> On 2010-07-07 13:58:43, Ryan Rawson wrote: > > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueSkipListSet.java, > > line 55 > > <http://review.hbase.org/r/257/diff/2/?file=2159#file2159line55> > > > > I think this information should be maintained in MemStore not inside > > this data structure. We might get rid of this data structure type and > > change to another one day. This makes it too hard to do that. > > Pranav Khaitan wrote: > When we are flushing the memstore to a storefile, we are passing an > object of KeyValueSkipListSet. This variable goes through several functions > before reaching Store. If we don't have TimeRangeTracker inside > KeyValueSkipListSet, we will have to change all flush related functions to > take an extra argument as input. This way, in future, if we decide to send > another piece of information, we will have to add more arguments. Having > TimeRangeTracker inside KeyValueSkipListSet lets us pass the information > without changing all flush related functions. Would it still be better to > pass TimeRangeTracker as an additional argument?
this totally makes sense, the only issue is that historically we have KeyValueSkipListSet because we couldnt use SkipListSet with the particular implementation of incrementColumnValue we had. Now that the implementation of ICV is changing (in an unrelated JIRA), we no longer need a specialized SkipListSet and we could use the standard one instead. We have the StoreFlusherImpl inside Store which exists to capture this kind of metadata and carry it along, so doing the other thing might not be too painful or bogus, what do you think? - Ryan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/257/#review314 ----------------------------------------------------------- On 2010-07-07 14:22:41, Pranav Khaitan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > http://review.hbase.org/r/257/ > ----------------------------------------------------------- > > (Updated 2010-07-07 14:22:41) > > > Review request for hbase, Nicolas, Jonathan Gray, Karthik Ranganathan, and > Kannan Muthukkaruppan. > > > Summary > ------- > > Every memstore and store file will have a minimum and maximum timestamp > associated with it. If the range of timestamps we are searching for doesn't > overlap with the range for a particular file, we can skip searching it and > save time. > > Would significantly improve the performance for timestamp range queries. > Particularly useful when most of the reads are for recent entries and the > older files can be safely skipped. > > Addresses HBASE-2265 JIRA. > > This diff includes fixing some minor bugs like KeyValueHeap used to throw an > uncaught exception when size of scanner set was zero. > > Internal review done by Jonathan and Kannan. > > > This addresses bug HBASE-2265. > http://issues.apache.org/jira/browse/HBASE-2265 > > > Diffs > ----- > > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java > 959782 > > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueSkipListSet.java > 959782 > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java > 959782 > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 959782 > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java > 959782 > > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java > 959782 > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > 959782 > > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java > PRE-CREATION > trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java > 960082 > trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java > 959782 > trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java > 959782 > > Diff: http://review.hbase.org/r/257/diff > > > Testing > ------- > > All existing JUnit tests run successfully. More JUnit tests for Memstore, > StoreFile and Store added to test correctness with multiple timestamps. > > Conducted a test to measure the extra time required to keep track of min and > max timestamps while writing KeyValues. The comparison was done by entering > 1 Million KeyValues into memstore ten times with and without timestamp > tracking and then taking the average time for each of them. WAL was disabled > and no flushing was done during this test to minimize overheads. The average > time taken for entering 1M KeyValues into memstore without keeping track of > timestamp was 13.44 seconds while the average time when keeping track of > timestamps was 13.45 seconds. This shows that no significant overhead has > been added while keeping track of timestamps. > > > Thanks, > > Pranav > >
