I suggest you Phoenix: 1. It does all these salting things automatically 2. Does not require M/R 3. Faster than any M/R job 4. Allows you to use SQL to access your data.
Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: [email protected] ________________________________________ From: Tao Xiao [[email protected]] Sent: Wednesday, April 23, 2014 8:09 AM To: user Subject: Re: How to get specified rows and avoid full table scanning? Hi all, Thank you all for your replies. After examining HBaseWD, OpenRSDB and Phoenix, I feel HBaseWD should meet my requirements. My business is as follows: Tens of millions of rows are appended to a table and each row has a date property, say 2014-04-01. I will submit a MapReduce job, whose input is some days of rows from that table, so I need to filter out rows of days other than what I specify. If the date is stored as part of the row key, I hope I can use a scan specifying the start and end key. At the same time measures should be taken to prevent the hot spot problem from happening, because naturally time-series row keys tend to be stored contiguously. HBaseWD avoids the problem of hot spot by decorating the original row key with a prefix. HBaseWD also makes it possibly for a MapReduce job to process data of a specified range (by creating a scan instance and pass it a *startKey* and a *stopKey*), *but I'm not sure whether this would trigger a full table scan*. 2014-04-22 2:05 GMT+08:00 James Taylor <[email protected]>: > Tao, > Just wanted to give you a couple of relevant pointers to Apache Phoenix for > your particular problem: > - Preventing hotspotting by salting your table: > http://phoenix.incubator.apache.org/salted.html > - Pig Integration for your map/reduce job: > http://phoenix.incubator.apache.org/pig_integration.html > > What kind of processing will you be doing in your map-reduce job? FWIW, > Phoenix will allow you to run SQL queries directly over your data, so that > might be an alternative for some of the processing you need to do. > > Thanks, > James > > > On Mon, Apr 21, 2014 at 9:20 AM, Jean-Marc Spaggiari < > [email protected]> wrote: > > > Hi Tao, > > > > also, if you are thinking about time series, you can take a look at TSBD > > http://opentsdb.net/ > > > > JM > > > > > > 2014-04-21 11:56 GMT-04:00 Ted Yu <[email protected]>: > > > > > There're several alternatives. > > > One of which is HBaseWD : > > > > > > > > > http://blog.sematext.com/2012/04/09/hbasewd-avoid-regionserver-hotspotting-despite-writing-records-with-sequential-keys/ > > > > > > You can also take a look at Phoenix. > > > > > > Cheers > > > > > > > > > On Mon, Apr 21, 2014 at 8:04 AM, Tao Xiao <[email protected]> > > > wrote: > > > > > > > I have a big table and rows will be added to this table each day. I > > wanna > > > > run a MapReduce job over this table and select rows of several days > as > > > the > > > > job's input data. How can I achieve this? > > > > > > > > If I prefix the rowkey with the date, I can easily select one day's > > data > > > as > > > > the job's input, but this will involve hot spot problem because > > hundreds > > > of > > > > millions of rows will be added to this table each day and the data > will > > > > probably go to a single region server. Secondary index would be good > > for > > > > query but not good for a batch processing job. > > > > > > > > Are there any other ways? > > > > > > > > Are there any other frameworks which can achieve this goal > easieruser? > > > > Shark? Stinger？HSearch? > > > > > > > > > > Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or [email protected] and delete or destroy any copy of this message and its attachments.
