Anil, It really depends on how many HFiles can be skipped entirely. In general, given that this is like full-table scan with filter, your time is good. Especially if it is satisfactory to you :). Glad that the idea with setting manually ts helped. This trick is overlooked too often :(
Alex Baranau ------ Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch - Solr On Wed, Aug 22, 2012 at 2:18 AM, anil gupta <[email protected]> wrote: > Hi Alex, > > Thanks for creating the JIRA. > On Monday, I completed testing the time range filtering using timestamps > and IMO the results seems satisfactory(if not great). The table has 34 > million records(average row size is 1.21 KB), in 136 seconds i get the > entire result of query which had 225 rows. > I am running a HBase 0.92, 8 node cluster on Vmware Hypervisor. Each node > had 3.2 GB of memory, and 500 GB HDFS space. Each Hard Drive in my set-up > is hosting 2 Slaves Instance(2 VM's running Datanode, > NodeManager,RegionServer). I have only allocated 1200MB for RS's. I haven't > done any modification in the block size of HDFS or HBase. Considering the > below-par hardware configuration of cluster, does the performance sounds OK > for timestamp filtering? > > Thanks, > Anil > > On Mon, Aug 20, 2012 at 1:07 PM, Alex Baranau <[email protected] > >wrote: > > > Created: https://issues.apache.org/jira/browse/HBASE-6618 > > > > Alex Baranau > > ------ > > Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch - > Solr > > > > On Sat, Aug 18, 2012 at 5:02 PM, anil gupta <[email protected]> > wrote: > > > > > Hi Alex, > > > > > > Apart from the query which i mentioned in last email. Till now, i have > > > implemented the following queries using filters and coprocessors: > > > > > > 1. Getting all the records for a customer. > > > 2. Perform min,max,avg,sum aggregation for a customer using > > coprocessors. I > > > am storing some of the data as BigDecimal also to do accurate floating > > > point calculations. > > > 3. Perform min,max,avg,sum aggregation for a customer within a given > > > time-range using coprocessors. > > > 4. Filter that data for a customer within a given time-range on the > basis > > > of column values. The filtering on column values can be matching a > string > > > value or it can be doing range based numerical comparison. > > > > > > Basically, as per our current requirement all the queries have > customerid > > > and most of the queries have timerange also. We are not in prod yet. > All > > of > > > this effort is part of a POC. > > > > > > 2. Can you set timestamp on Puts the same as timestamp "assigned" to > your > > > record by app logic? > > > Anil: Wow! This sounds like an awesome idea. Actually, my data is > > > non-mutable so at present i was putting 0 as the timestamp for all the > > > data. I will definitely try this stuff. Currently, i run bulkloader to > > load > > > the data so i think its gonna be a small change. > > > > > > Yes, i would love to give a try from my side for developing a range > based > > > FuzzyRowFilter. However, first i am going to try putting in the > > timestamp. > > > > > > Thanks for a very helpful discussion. Let me know when you create the > > JIRA > > > for range-based FuzzyRowFilter. > > > > > > Thanks, > > > Anil Gupta > > > > > > On Sat, Aug 18, 2012 at 12:13 PM, Alex Baranau < > [email protected] > > > >wrote: > > > > > > > @Michael, > > > > > > > > This is not a simple partial key scan. Take this example of rows: > > > > > > > > aaaaa_100001_20120801 > > > > aaaaa_100001_20120802 > > > > aaaaa_100001_20120802 > > > > aaaaa_100001_20120803 > > > > aaaaa_100001_20120804 > > > > aaaaa_100001_20120805 > > > > aaaaa_100002_20120801 > > > > aaaaa_100002_20120802 > > > > aaaaa_100002_20120802 > > > > aaaaa_100002_20120803 > > > > aaaaa_100002_20120804 > > > > aaaaa_100002_20120805 > > > > > > > > where aaaaa is userId, 10000x is actionId and 201208xx is a > timestamp. > > If > > > > the query is to select actions in the range 20120803-20120805 (in > this > > > case > > > > last 3 days), then when scan encounters row: > > > > > > > > aaaaa_100001_20120801 > > > > > > > > it "knows" it can fast forward scanning to "aaaaa_100001_20120803", > and > > > > skip some records (in practice, this may mean skipping really a LOT > of > > > > recrods). > > > > > > > > > > > > @Anil, > > > > > > > > > Sample Query: I want to get all the event which happened in last > > month. > > > > > > > > 1. What other queries do you do? Just trying to understand why this > row > > > key > > > > format was chosen. > > > > > > > > 2. Can you set timestamp on Puts the same as timestamp "assigned" to > > your > > > > record by app logic? If you can, then this is the first thing to try > > and > > > > perform scan with the help of scan.setTimeRange(startTs, stopTs). > > > Depending > > > > on how you write the data this may help a lot with the reading speed > by > > > ts, > > > > because that way you may skip the whole HFiles from reading based on > > ts. > > > I > > > > don't know about your data a lot to judge, but: > > > > * in case you have not a lot of users most of which are with long > > > history > > > > of interaction with you system (i.e. there are a lot of records for > > > > specific "userX_actionY") and > > > > * if you write data with monotonically increasing timestamp > > > > * your regions are not too big > > > > then this might help you, as it will increase the chance that some of > > the > > > > HFiles will contain data *all of which* doesn't fell into the time > > > interval > > > > you select by. Otherwise, if written data items with different > > timestamps > > > > are very well spread across the HFiles the chance that some HFiles > are > > > > skipped from reading is very small. I believe Lars George has > > illustrated > > > > this in one of his presentations, but couldn't find it quickly. > > > > > > > > > something like FuzzyRowFilter with range > > > > > > > > Yes, smth like this looks like would be very valuable. It would be > > > > interesting to implement too. Let's see if I find the time for that > in > > my > > > > work plan. If you want to try it by yourself, go for it! Let me know > if > > > you > > > > need a help in that case ;) > > > > > > > > Alex Baranau > > > > ------ > > > > Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch - > > > Solr > > > > > > > > On Sat, Aug 18, 2012 at 6:56 AM, Michael Segel < > > > [email protected] > > > > >wrote: > > > > > > > > > What row keys are you skipping? > > > > > > > > > > Using your example... > > > > > You have a start row of 00000000200, and an end key of > > > > > xFFxFFxFFxFFxFFxFF00350. > > > > > Note that you could also write that end key as xFF(1..6) 01 since > it > > > > looks > > > > > like you're trying to match the 00 in positons 7 and 8 of your > > numeric > > > > > string. > > > > > > > > > > Assuming that when you say ? you mean that you expect to have a > > > character > > > > > in that spot and that your row key is exactly 11 characters in > > length. > > > > > > > > > > While you may not return all the rows in that range, you do have to > > > still > > > > > check the row key, unless I am missing something. > > > > > > > > > > So what am I missing? > > > > > > > > > > On Aug 17, 2012, at 3:42 PM, Alex Baranau < > [email protected]> > > > > > wrote: > > > > > > > > > > > There was a question [1] in > > > > > > https://issues.apache.org/jira/browse/HBASE-6509JIRA comment, it > > > makes > > > > > > more sense to answer it here. > > > > > > > > > > > > With the current FuzzyRowFilter I believe the only way to > approach > > > the > > > > > > problem is to add 150 fuzzy rules to the filter: ??????00200, > > > > > ??????00201, > > > > > > ..., ??????00350. > > > > > > > > > > > > As for performance of this approach I can say the following: > > > > > > * there are two "checks" happening for each processed row key > (i.e. > > > > those > > > > > > row keys we don't skip) > > > > > > * first one performs simple check if the given row key satisfies > > the > > > > > fuzzy > > > > > > rule and also determines if there's next row key to advance to > (if > > > this > > > > > one > > > > > > doesn't satisfy). The check takes up at max O(n), where n is the > > > length > > > > > of > > > > > > fuzzy rule. I.e. this is done in one simple loop, which can be > > broken > > > > > > before all bytes are checked. For m rules this will be O(m*n). > > > > > > * second piece calculates the next row key to provide it as a > hint > > > for > > > > > > fast-forwarding. We again check all rules and finding the > smallest > > > > hint. > > > > > > Operation is also done in one loop, i.e. O(m*n) here as well. > > > > > > > > > > > > With 150 fuzzy rules of length 11, the applying filter is > > equivalent > > > to > > > > > the > > > > > > loop with simple checks thru 150*11*2 ~ 3000 elements. This might > > > look > > > > a > > > > > > lot, but can work quite fast. So I'd just try it. > > > > > > > > > > > > As for extension which will be more efficient, it makes sense to > > > > consider > > > > > > implementing it. Let me think more about it and get back with the > > > JIRA > > > > > > Issue to you :). But I'd suggest you trying existing > FuzzyRowFilter > > > > > first. > > > > > > The output (performance) would give us some food for thinking, or > > may > > > > be > > > > > > even turns out to be acceptable for you (hopefully). > > > > > > > > > > > >> Can i run this kind of filter on HBase0.92 without doing any > > > > significant > > > > > > update to the cluster > > > > > > > > > > > > Until the next release, you'll have to use the FuzzyRowFilter as > > any > > > > > other > > > > > > custom filter. Just grab the patch from HBASE-6509 and copy the > > > filter. > > > > > No > > > > > > need to patch & rebuild HBase. > > > > > > > > > > > > Alex Baranau > > > > > > ------ > > > > > > Sematext :: http://sematext.com/ :: Hadoop - HBase - > > ElasticSearch - > > > > > Solr > > > > > > > > > > > > [1] > > > > > > > > > > > > Anil Gupta added a comment - 18/Aug/12 04:37 > > > > > > Hi Alex, > > > > > > I have a question related to this filter. I have a similar > > filtering > > > > > > requirement which will be an extension to FuzzyFilterRow. > > > > > > Suppose, i have the following structure of rowkeys: > > userid_actionid, > > > > > where > > > > > > userid is of 6 digit and then actionid is 5 digit. I would like > to > > > get > > > > > all > > > > > > the rows with actionid between 00200 to 00350. With current > > > > > FuzzyRowFilter > > > > > > i can search for all the rows a particular actionid. Instead of > > > > searching > > > > > > for a particular actionid i would like to search for a range of > > > > actionid. > > > > > > Does this use case sounds like an extension to current > > > FuzzyRowFilter? > > > > > Can > > > > > > i run this kind of filter on HBase0.92 without doing any > > significant > > > > > update > > > > > > to the cluster. If i develop this kind of filter then what is > > needed > > > to > > > > > run > > > > > > it on all the RS's? > > > > > > Thanks, > > > > > > Anil > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks & Regards, > > > Anil Gupta > > > > > > > > > -- > Thanks & Regards, > Anil Gupta >
