Jake, I wonder how often people need reliable transactions for realtime search? Maybe Mysql's t-log could be used sans the database part?
The created_at column for near realtime seems like it could hurt the database due to excessive polling? Has anyone tried it yet? > I wrote up a simple file-based indexing event log in an afternoon Right, however it's probably a long perilous leap from this to a t-log that's production ready. I'm waiting for someone to dive in and mess with Bookkeeper http://wiki.apache.org/hadoop/BookKeeper and report back! Jason On Tue, Jan 12, 2010 at 9:49 PM, Jake Mannix <jake.man...@gmail.com> wrote: > On Tue, Jan 12, 2010 at 8:55 PM, Jason Rutherglen > <jason.rutherg...@gmail.com> wrote: >> >> > Zoie keeps track of an "index version" on disk alongside the Lucene >> > index which it uses to decide where it must reindex from to "catch up" if >> > it >> > there have been incoming indexing events while the server was out of >> > commission. >> >> This begs a little more clarity... Sounds like a transaction log. Oh >> right, with Zoie there's the assumption of an external transaction log >> however it doesn't provide one out of the box? > > The index versioning scheme Zoie uses is independent of what mechanism you > use to implement it. If your indexing technique is to talk to a database > directly, you don't need a transaction log, something as simple as a > "created_at" column will suffice in many situations. I gave a short talk to > demo zoie yesterday, and for it I wrote up a simple file-based indexing > event log in an afternoon. Similarly if you listen on a JMS queue or > basically any other message-queue based system that not "push only", you'll > have some notion of "replay since [timestamp / version / incrementing > counter]", but they're all vendor dependent. > It's not the kind of thing you can just provide out of the box due to this > vendor dependence. On the other hand, if someone came along and said they > wanted to use zoie with RabbitMQ or whatever, we'd certainly accept a patch > for a StreamDataProvider implementation which does that (and maybe one of > the zoie committers would even write it themself it it seemed like a common > enough use case). > -jake > >> >> On Tue, Jan 12, 2010 at 8:43 PM, Jake Mannix <jake.man...@gmail.com> >> wrote: >> > On Tue, Jan 12, 2010 at 8:15 PM, Otis Gospodnetic >> > <otis_gospodne...@yahoo.com> wrote: >> >> >> >> John, you should have a look at Zoie. I just finished adding >> >> LinkedIn's >> >> case study about Zoie to Lucene in Action 2, so this is fresh in my >> >> mind. >> >> >> >> :) >> > >> > Yep, Zoie ( http://zoie.googlecode.com ) will handle the server restart >> > part, in that while yes, you lose what is in RAM, Zoie keeps track of an >> > "index version" on disk alongside the Lucene index which it uses to >> > decide >> > where it must reindex from to "catch up" if it there have been incoming >> > indexing events while the server was out of commission. >> > Zoie does not support multiple servers using the same index, because >> > each >> > zoie instance has IndexWriter instances, and you'll get locking problems >> > trying to do that. You could have one Zoie instance effectively as the >> > "master/writer/realtime reader", and a bunch of raw Lucene "slaves" >> > which >> > could read off of that index, but as you say, could not get access to >> > the >> > RAMDirectory information until it was flushed to disk. >> > Why do you need a "cluster" of servers hitting the same index? Are they >> > different applications (with different search logic, so they need to be >> > different instances), or is it just to try and utilize your hardware >> > efficiently? If it's for performance reasons, you might find you get >> > better >> > use of your CPU cores by just sharding your one index into smaller ones, >> > each having their own Zoie instance, and putting a "broker" on top of >> > them >> > searching across all and mergesorting the results. Often even this >> > isn't >> > necessary, because Zoie will be opening the disk-backed IndexReader in >> > readonly mode, and thus all the synchronized blocks are gone, and one >> > single >> > Zoie instance will easily saturate your cpu cores by simple >> > multi-threading >> > by your appserver. >> > If you really needed to do many different kinds of writes (from >> > different >> > applications) and also have applications not involved in the writing >> > also >> > seeing (in real-time) these writes, then you could still do it with >> > Zoie, >> > but it would take some interesting architectural juggling (write your >> > own >> > StreamDataProvider class which takes input from a variety of sources and >> > merges them together to feed to one Zoie instance, then a broker on top >> > of >> > zoie which serves out IndexReaders to different applications living on >> > top >> > which can wrap them up in their own business logic as they saw fit... as >> > long as it was ok to have all the applications in the same JVM, of >> > course). >> > -jake >> > >> >> >> >> Otis >> >> -- >> >> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch >> >> >> >> >> >> >> >> ----- Original Message ---- >> >> > From: jchang <jchangkihat...@gmail.com> >> >> > To: java-dev@lucene.apache.org >> >> > Sent: Tue, January 12, 2010 6:10:56 PM >> >> > Subject: Lucene 2.9.0 Near Real Time Indexing and Service >> >> > Crashes/restarts >> >> > >> >> > >> >> > Lucene 2.9.0 has near real time indexing, writing to a RAMDir which >> >> > gets >> >> > flushed to disk when you do a search. >> >> > >> >> > Does anybody know how this works out with service restarts (both >> >> > orderly >> >> > shutdown and a crash)? If the service goes down while indexed items >> >> > are >> >> > in >> >> > RAMDir but not on disk, are they lost? Or is there some kind of log >> >> > recovery? >> >> > >> >> > Also, does anybody know the impact of this which clustered lucene >> >> > servers? >> >> > If you have numerous servers running off one index, I assume there is >> >> > no >> >> > way >> >> > for the other services to pick up the newly indexed items until they >> >> > are >> >> > flushed to disk, correct? I'd be happy if that is not so, but I >> >> > suspect >> >> > it >> >> > is so. >> >> > >> >> > Thanks, >> >> > John >> >> > -- >> >> > View this message in context: >> >> > >> >> > >> >> > http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-Service-Crashes-restarts-tp27136539p27136539.html >> >> > Sent from the Lucene - Java Developer mailing list archive at >> >> > Nabble.com. >> >> > >> >> > >> >> > --------------------------------------------------------------------- >> >> > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> >> > For additional commands, e-mail: java-dev-h...@lucene.apache.org >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: java-dev-h...@lucene.apache.org >> >> >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-dev-h...@lucene.apache.org >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
