> -----Original Message----- > From: Stefan Groschupf [mailto:[EMAIL PROTECTED] > Sent: Wednesday, January 02, 2008 3:46 AM > To: hadoop-user@lucene.apache.org > Subject: Re: HBase implementation question > > Hi, > > Reads are probably a bit more complicated than writes. A read > > operation first checks the cache and may satisfy the > request directly > > from the cache. If not, the operation checks the newest MapFile for > > the data, then the next to newest, ..., to the oldest stopping when > > the requested data has been retrieved. Because a random > read (or even > > a sequential read that is not a scan) can end up checking multiple > > files for data they are considerably slower than either writes and > > sequential scans (think of a scan as working with a cursor in a > > traditional database). > > Sorry, just to double check I understand it correctly. The > number of files need to be checked for a read is related to > the compaction threshold, since all files are merged into one > big sorted file after a given time by the compaction thread?
Correct. > Any idea how many files usually need to checked in average? Not yet. We have been focused on getting things to work right and increasing reliability before we focus on performance. > Would it make any sense here to work with key-spaces similar > to the map/reduce partitioner to keep the number of files > that need to be read smaller? Ok, well this gets into the part of the story that I left out of my last message for the sake of simplicity. Each table is broken into a series of row ranges called regions. A HRegion object manages a single row range. Initially there is only one region that spans the entire table. As a region grows, it is split in half when it exceeds a configurable threshold in size. We optimize splits so that each child region shares one half of the parent until the child does a compaction. When both children have done a compaction, the parent is garbage collected. Each region has all the columns for its row range. Columns are stored separately and each column for a row range is managed by an HStore object. So a lookup for a particular row/column value is limited by only checking the row range that contains the row and then by only checking for the requested columns. Negative lookups can be sped up by adding bloom filters. Bloom filters are configured on a per column basis, so not every column need have a bloom filter. If you haven't already read Google's Bigtable paper, I think you would find it informative as many of the concepts employed by Bigtable have similar analogues in HBase. You can find the Bigtable paper here: http://labs.google.com/papers/bigtable.html a c > Thanks, > Stefan > > > No virus found in this incoming message. > Checked by AVG Free Edition. > Version: 7.5.516 / Virus Database: 269.17.13/1206 - Release > Date: 1/1/2008 12:09 PM > > No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date: 1/2/2008 11:29 AM
