Hi, What is the reason not to use JDs replication WLA tracking (using ZK) and add another "scope" like attribute to the coldefs to define the indexing? Just curious because when I read the questions on the Wiki it mentions issues that JD I think had to solve already, being "here did we left off?", "no aggressive reapplying edits" etc.
Lars On Tue, Mar 1, 2011 at 6:24 AM, Dhruba Borthakur <dhr...@gmail.com> wrote: > Hi Jonathan, > > Nice wiki. You mention that >> 2. Generate a new, special kind of WALEdit for secondary table update > > is it possible to store this secondary-wal-edits as the contents on another > hbase table(say indexTable)? The advantage is that then this can be > implemented as a pure layer wrapping the regionserver (via co-processors and > such). The disadvantage is that a non-remote log (for indexTable) might need > to be updated. Is there a easy way to enhance hbase to co-locate regions on > the same machine? > > thanks, > dhruba > > On Mon, Feb 28, 2011 at 12:05 PM, Jonathan Gray <jg...@fb.com> wrote: > >> I've started a wiki page: >> http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing >> >> I gave a basic description of the idea I had and the open questions. >> >> Let's get all our thoughts in there. >> >> > -----Original Message----- >> > From: saint....@gmail.com [mailto:saint....@gmail.com] On Behalf Of >> Stack >> > Sent: Friday, February 25, 2011 4:07 PM >> > To: dev@hbase.apache.org >> > Subject: Re: what's the roadmap of secondary index of hbase? >> > >> > The MegaStore paper, >> > http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf, in section >> > 3.2.2, lists secondary indexing options MegaStore provides on top of >> > BigTable. For example, MS allows specifying secondary index on protobuf >> > cell content or duplicating data into secondary index so you have the >> data to >> > hand to satisfy first query and only if the client wants more do you go >> dig in >> > the primary table. It also talks about how secondary indices can be >> described >> > using their schema which might be of use. Might be worth a gander. >> > >> > St.Ack >> > >> > On Fri, Feb 25, 2011 at 3:32 PM, Stack <st...@duboce.net> wrote: >> > > On Fri, Feb 25, 2011 at 1:47 PM, Eugene Koontz <ekoo...@hiro-tan.org> >> > wrote: >> > >> I'm thinking that we could use a coprocessor that watches the >> > >> Write-Ahead Log (using the WAL-edit operations >> > >> https://issues.apache.org/jira/browse/HBASE-3257 "Coprocessors: >> > >> Extend server side integration API to include HLog operations"). This >> > >> coprocessor would write these edits, perhaps filtering or >> > >> transforming them, and enqueing the results in a global queue. A >> > >> separate process would be responsible for pulling operations off the >> > >> queue and using HBase client operations to do the insert into a >> > >> secondary index table appropriate for that operation. >> > >> Perhaps we could use some of the work that the Lily people have >> > >> done with HBase indexing (see >> > >> http://www.lilyproject.org/lily/about/playground/hbaseindexes.html) >> > >> in order to do the edit->hbase operation transformations and the >> > >> secondary index table creation. >> > > >> > > This sounds good as first approach (including lily part). >> > > >> > > St.Ack >> > > >> > > > > -- > Connect to me at http://www.facebook.com/dhruba >
