Hi, I also looked at your implementation. I have some issues with it but maybe someone can correct me if I'm wrong. What you're doing is writing the secondary index from the memstore in the preFlush hook. However there is no mutual exclusive lock (write lock) that is taken on the whole memstore for both the preFlush hook and the internal flush operation. The only time a write lock is taken on the whole memstore is when taking a snapshot and that comes after the preFlush hook. So after completion of the preFlush hook, a couple more elements can be added to the memstore until it is actually flushed to disk. This means you lose consistency between your primary and your secondary index. Can this issue be solved somehow? I am also interested in building a secondary index from the preflush hook.
Thanks, Sever On Fri, Jul 6, 2012 at 6:04 AM, fding hbase <[email protected]> wrote: > Sorry, I don't have much experience on unit test. The performance may not > be good > since this is my leisure time work, nor do I have good understanding for > consistency. > I really wanna help but for now it's maybe beyond my ability. I'm still > digging the code > of hbase too... :-) > > HBase is quite HUGE to me... > > On Fri, Jul 6, 2012 at 10:32 AM, Ted Yu <[email protected]> wrote: > > > The following action allows me to compile your code: > > > > 551 cp ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/*.java > > src/main/java/org/apache/hadoop/hbase/regionserver/ > > 552 mkdir src/main/java/org/apache/hadoop/hbase/regionserver/indexed > > 553 cp > > ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/indexed/* > > src/main/java/org/apache/hadoop/hbase/regionserver/indexed/ > > > > There is no unit test. Can you add some ? > > > > Do you have performance data: overhead for writing index, speedup in > query > > ? > > > > The most recent motion on this subject is HBASE-5723. > > I still need to dig into your code. If you can comment on the level of > > consistency guarantees for your design, that would be nice. > > > > Cheers > > > > On Thu, Jul 5, 2012 at 6:50 PM, fding hbase <[email protected]> > wrote: > > > > > Hi all, > > > > > > I've implemented an indexing system. It needs refinement of course > > > but I have my own work to do, so I decided to open it for everyone to > > > access, modify and any improvement or ideas would be great. > > > > > > https://github.com/danix800/hbase-indexed > > > > > > -- > > > > > > Best Regards! > > > > > > Fei Ding > > > [email protected] > > > > > > > > > -- > > Best Regards! > > Fei Ding > [email protected] > -- Sever Fundatureanu Vrije Universiteit Amsterdam E-mail: [email protected]
