Some context on how I observed bloom filters being loaded constantly. I
added the following logging statements to HFileReaderV2.java:
}
if (!useLock) {
// check cache again with lock
useLock = true;
continue;
}
// Load block from filesystem.
long startTimeNs = System.nanoTime();
HFileBlock hfileBlock = fsBlockReader.readBlockData(dataBlockOffset,
onDiskBlockSize, -1, pread);
hfileBlock = dataBlockEncoder.diskToCacheFormat(hfileBlock,
isCompaction);
validateBlockType(hfileBlock, expectedBlockType);
passSchemaMetricsTo(hfileBlock);
BlockCategory blockCategory =
hfileBlock.getBlockType().getCategory();
// My logging statements ---->
if(blockCategory == BlockCategory.INDEX) {
LOG.info("index block miss, reading from disk " + cacheKey);
} else if (blockCategory == BlockCategory.BLOOM) {
LOG.info("bloom block miss, reading from disk " + cacheKey);
} else {
LOG.info("block miss other than index or bloom, reading from disk
" + cacheKey);
}
//-------------->
final long delta = System.nanoTime() - startTimeNs;
HFile.offerReadLatency(delta, pread);
getSchemaMetrics().updateOnCacheMiss(blockCategory, isCompaction,
delta);
// Cache the block if necessary
if (cacheBlock && cacheConf.shouldCacheBlockOnRead(
hfileBlock.getBlockType().getCategory())) {
cacheConf.getBlockCache().cacheBlock(cacheKey, hfileBlock,
cacheConf.isInMemory());
}
if (hfileBlock.getBlockType() == BlockType.DATA) {
HFile.dataBlockReadCnt.incrementAndGet();
}
With these in place I saw the following statements in log:
2013-06-05 01:04:55,281 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_30361506
2013-06-05 01:05:00,579 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_28779560
2013-06-05 01:07:41,335 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_4199735
2013-06-05 01:08:58,460 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_8519720
2013-06-05 01:11:01,545 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_12838948
2013-06-05 01:11:03,035 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_3973250
2013-06-05 01:11:36,339 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_17159812
2013-06-05 01:12:35,398 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_21478349
2013-06-05 01:13:02,572 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_25798003
2013-06-05 01:13:03,260 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_8068381
2013-06-05 01:13:20,265 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_30118048
2013-06-05 01:13:20,522 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_60833137
2013-06-05 01:13:32,261 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_34545951
2013-06-05 01:13:48,504 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_38865311
2013-06-05 01:13:49,951 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_12161793
2013-06-05 01:14:02,073 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_43185677
2013-06-05 01:14:12,956 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_47506066
2013-06-05 01:14:25,132 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_51825831
2013-06-05 01:14:25,946 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_16257519
2013-06-05 01:14:34,478 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_56145793
2013-06-05 01:14:45,319 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_60466405
2013-06-05 01:14:45,998 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_91304775
2013-06-05 01:14:58,203 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_64893493
2013-06-05 01:14:58,463 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_20352561
2013-06-05 01:15:09,299 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_69214092
2013-06-05 01:15:32,944 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_73533616
2013-06-05 01:15:46,903 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_77865906
2013-06-05 01:15:47,273 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_24448138
2013-06-05 01:15:55,312 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_82185687
2013-06-05 01:16:07,591 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_86506129
2013-06-05 01:16:20,728 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_90825624
2013-06-05 01:16:22,551 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_28542144
2013-06-05 01:16:22,810 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_121777484
2013-06-05 01:16:23,035 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_57670002
2013-06-05 01:16:33,196 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_95253904
2013-06-05 01:16:48,187 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_99574899
2013-06-05 01:17:06,648 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_103895087
2013-06-05 01:17:10,526 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_32744846
2013-06-05 01:17:22,939 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_108214936
2013-06-05 01:17:36,010 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_112535209
2013-06-05 01:17:46,028 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_116855742
2013-06-05 01:17:47,029 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_36838416
2013-06-05 01:17:54,472 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_121174753
2013-06-05 01:17:55,491 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_152248177
2013-06-05 01:18:05,912 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_125601238
2013-06-05 01:18:15,417 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_129921797
2013-06-05 01:18:16,713 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_40933856
2013-06-05 01:18:29,521 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_134242324
2013-06-05 01:18:38,653 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_138561860
2013-06-05 01:18:49,280 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_142881436
2013-06-05 01:18:50,052 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 52cded0c399b48fdbccd8b3d4e25502f_45029905
2013-06-05 01:18:58,339 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_147201737
2013-06-05 01:19:06,371 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_151533253
2013-06-05 01:19:07,782 INFO
org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
from disk 11958ab7a4a1492e853743b02e1bd7b1_182719269
I kept seeing these statements appearing constantly over a long period,
this seemed to confirm to me that bloom filter blocks are being loaded over
a period time, which also matched what I read about HFileV2. May be I am
wrong about both. Would love to understand what's really going on.
Thanks in Advance,
Pankaj
On Tue, Jun 4, 2013 at 11:05 PM, ramkrishna vasudevan <
[email protected]> wrote:
> Whenever the region is opened all the bloom filter meta data are loaded
> into memory. I think his concern is every time all the store files are
> read and then we load it into memory and wants some faster ways of doing
> it.
> Asaf you are right.
>
> Regards
> Ram
>
>
> On Wed, Jun 5, 2013 at 11:22 AM, Asaf Mesika <[email protected]>
> wrote:
>
> > When you do the first read of this region, wouldn't this load all bloom
> > filters?
> >
> >
> >
> > On Wed, Jun 5, 2013 at 8:43 AM, ramkrishna vasudevan <
> > [email protected]> wrote:
> >
> > > for the question whether you will be able to do a warm up for the bloom
> > and
> > > block cache i don't think it is possible now.
> > >
> > > Regards
> > > Ram
> > >
> > >
> > > On Wed, Jun 5, 2013 at 10:57 AM, Asaf Mesika <[email protected]>
> > > wrote:
> > >
> > > > If you will read HFile v2 document on HBase site you will understand
> > > > completely how the search for a record works and why there is linear
> > > search
> > > > in the block but binary search to get to the right block.
> > > > Also bear in mind the amount of keys in a blocks is not big since a
> > block
> > > > in HFile by default is 65k, thus from a 10GB HFile you are only fully
> > > > scanning 65k out of it.
> > > >
> > > > On Wednesday, June 5, 2013, Pankaj Gupta wrote:
> > > >
> > > > > Thanks for the replies. I'll take a look at
> src/main/java/org/apache/
> > > > > hadoop/hbase/coprocessor/BaseRegionObserver.java.
> > > > >
> > > > > @ramkrishna: I do want to have bloom filter and block index all the
> > > time.
> > > > > For good read performance they're critical in my workflow. The
> worry
> > is
> > > > > that when HBase is restarted it will take a long time for them to
> get
> > > > > populated again and performance will suffer. If there was a way of
> > > > loading
> > > > > them quickly and warm up the table then we'll be able to restart
> > HBase
> > > > > without causing slow down in processing.
> > > > >
> > > > >
> > > > > On Tue, Jun 4, 2013 at 9:29 PM, Ted Yu <[email protected]>
> wrote:
> > > > >
> > > > > > bq. But i am not very sure if we can control the files getting
> > > selected
> > > > > for
> > > > > > compaction in the older verisons.
> > > > > >
> > > > > > Same mechanism is available in 0.94
> > > > > >
> > > > > > Take a look
> > > > > > at
> > > > > >
> > > >
> > src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> > > > > > where you would find the following methods (and more):
> > > > > >
> > > > > > public void preCompactSelection(final
> > > > > > ObserverContext<RegionCoprocessorEnvironment> c,
> > > > > > final Store store, final List<StoreFile> candidates, final
> > > > > > CompactionRequest request)
> > > > > > public InternalScanner
> > > > > > preCompact(ObserverContext<RegionCoprocessorEnvironment> e,
> > > > > > final Store store, final InternalScanner scanner) throws
> > > > > IOException
> > > > > > {
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Tue, Jun 4, 2013 at 8:14 PM, ramkrishna vasudevan <
> > > > > > [email protected]> wrote:
> > > > > >
> > > > > > > >>Does Minor compaction remove HFiles in which all entries are
> > out
> > > of
> > > > > > > TTL or does only Major compaction do that
> > > > > > > Yes it applies for Minor compactions.
> > > > > > > >>Is there a way of configuring major compaction to compact
> only
> > > > files
> > > > > > > older than a certain time or to compress all the files
> except
> > > the
> > > > > > latest
> > > > > > > few?
> > > > > > > In the latest trunk version the compaction algo itself can be
> > > > plugged.
> > > > > > > There are some coprocessor hooks that gives control on the
> > scanner
> > > > > that
> > > > > > > gets created for compaction with which we can control the KVs
> > being
> > > > > > > selected. But i am not very sure if we can control the files
> > > getting
> > > > > > > selected for compaction in the older verisons.
> > > > > > > >> The above excerpt seems to imply to me that the search for
> key
> > > > > inside
> > > > > > a
> > > > > > > block
> > > > > > > is linear and I feel I must be reading it wrong. I would expect
> > the
> > > > > scan
> > > > > > to
> > > > > > > be a binary search.
> > > > > > > Once the data block is identified for a key, we seek to the
> > > beginning
> > > > > of
> > > > > > > the block and then do a linear search until we reach the exact
> > key
> > > > that
> > > > > > we
> > > > > > > are looking out for. Because internally the data (KVs) are
> > stored
> > > as
> > > > > > byte
> > > > > > > buffers per block and it follows this pattern
> > > > > > > <keylength><valuelength><keybytearray><valuebytearray>
> > > > > > > >>Is there a way to warm up the bloom filter and block index
> > cache
> > > > for
> > > > > > > a table?
> > > > > > > You always want the bloom and block index to be in cache?
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jun 5, 2013 at 7:45 AM, Pankaj Gupta <
> > > [email protected]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have a few small questions regarding HBase. I've searched
> the
> > > > forum
> > > > > > but
> > > > > > > > couldn't find clear answers hence asking them here:
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. Does Minor compaction remove HFiles in which all
> entries
> > > are
> > > > > out
> > > > > > of
> > > > > > > > TTL or does only Major compaction do that? I found this
> > jira:
> > > > > > > > https://issues.apache.org/jira/browse/HBASE-5199 but I
> > dont'
> > > > know
> > > > > > if
> > > > > > > > the
> > > > > > > > compaction being talked about there is minor or major.
> > > > > > > > 2. Is there a way of configuring major compaction to
> compact
> > > > only
> > > > > > > files
> > > > > > > > older than a certain time or to compress all the files
> > except
> > > > the
> > > > > > > latest
> > > > > > > > few? We basically want to use the time based filtering
> > > > > optimization
> > > > > > in
> > > > > > > > HBase to get the latest additions to the table and since
> > major
> > > > > > > > compaction
> > > > > > > > bunches everything into one file, it would defeat the
> > > > > optimization.
> > > > > > > > 3. Is there a way to warm up the bloom filter and block
> > index
> > > > > cache
> > > > > > > for
> > > > > > > > a table? This is for a case where I always want the bloom
> > > > filters
> > > > > > and
> > > > > > > > index
> > > > > > > > to be all in memory, but not the
> > > >
> > >
> >
>
--
*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | [email protected]
Pankaj Gupta | Software Engineer
*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
United States | Canada | United Kingdom | Germany
We're
hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!
