bq. the column we are applying filter against is in content family I assume content family would always be returned in your query (after filtering is done).
HBASE-5416 would help if you can specify metadata as essential column. This would greatly reduce the amount of data loaded into memory. Cheers On Fri, Mar 29, 2013 at 6:49 AM, Agarwal, Saurabh <[email protected]>wrote: > Thanks Marcos for reference. **** > > ** ** > > We are using filter on only one column. And we further narrow down filter > and results using ->**** > > ** ** > > scan.addColumn(family, qualifier);**** > > ** ** > > PageFilter pageFilter = *new* PageFilter(size);**** > > mainFilterList.addFilter(pageFilter);**** > > ** ** > > we have two columns family – content ( most of the data) and metadata > (only metadata). And the column we are applying filter against is in > content family. In that case, will HBASE-5416 help?**** > > ** ** > > We are also using ROW bloomfilter. Will ROWCOlUMN bloomfilter help in our > scenario? **** > > ** ** > > ** ** > > Saurabh. **** > > ** ** > > **** > > *From:* Marcos Luis Ortiz Valmaseda [mailto:[email protected]] > *Sent:* Thursday, March 28, 2013 10:36 PM > *To:* [email protected]; Agarwal, Saurabh [CCC-OT_IT]; > [email protected] > > *Subject:* Re: coprocessor is timing out in 0.94**** > > ** ** > > Regards, Saurabh.**** > > I see that you are using SingleColumnValueFilter. Look for these links:*** > * > > http://gbif.blogspot.com/2012/05/optimizing-hbase-mapreduce-scans-for.html > **** > > > http://mapredit.blogspot.com/2012/05/using-filters-in-hbase-to-match-two.html > **** > > ** ** > > Take a look later to this link, about the working to improve scans:**** > > https://issues.apache.org/jira/browse/HBASE-5416**** > > ** ** > > ** ** > > 2013/3/28 Agarwal, Saurabh <[email protected]>**** > > Ted, > > Thanks for response. > > Here is the filter we are using - > SingleColumnValueFilter(Bytes.toBytes(columnFamily), > Bytes.toBytes(columnQualifier), CompareFilter.CompareOp.EQUAL, new > RegexStringComparator("(?i)"+"keyword")); > > The thread dump at different points show that coprocessor is getting > called. Also logs showed it keep processing. But the speed is much slower > compare to 0.92. > > Regards, > Saurabh.**** > > > -----Original Message----- > From: Ted Yu [mailto:[email protected]] > Sent: Thursday, March 28, 2013 6:57 PM > To: [email protected] > Subject: Re: coprocessor is timing out in 0.94 > > bq. I checked thread dump > > If there was no exception in region server logs, thread dump of region > server when your coprocessor was running would reveal where it got stuck. > > From your description below, looks like you can utilize HBASE-5416 Improve > performance of scans with some kind of filters. > > bq. to apply the filter on one of the column > > Basically this column is the essential column. > > Cheers > > On Thu, Mar 28, 2013 at 3:22 PM, Ted Yu <[email protected]> wrote: > > > bq. when I removed the filter, it ran fine in 0.94 > > > > Can you disclose more information about your filter ? > > > > BTW 0.94.6 was just released which is fully compatible with 0.94.2 > > > > Cheers > > > > On Thu, Mar 28, 2013 at 3:18 PM, Agarwal, Saurabh < > > [email protected]> wrote: > > > >> Hi, > >> > >> We are in process of migrating from 0.92.1 to 0.94.2. A coprocessor > >> was running fine in 0.92. After migrating to 0.94, the client is > >> timing out (java.net.SocketTimeoutException). We are using > >> coprocessor to apply the filter on one of the column and return the > >> columns that match with that filter criteria. I checked thread dump, > >> region server, web UI, logs. There is no error or exception. One > >> thing I noticed that when I removed the filter, it ran fine in 0.94 as > well. > >> > >> Please advise if there is any specific setting we need to make in 0.94. > >> > >> Thanks, > >> Saurabh. > >> > > > >**** > > > > **** > > ** ** > > -- **** > > Marcos Ortiz Valmaseda, > *Data-Driven Product Manager* at PDVSA > *Blog*: http://dataddict.wordpress.com/ > *LinkedIn: *http://www.linkedin.com/in/marcosluis2186 > *Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>**** >
