Hi guys!

My apology for irrelevant post there.

I don't wanna receive emails from @lucene.apache.org  on this account as I have 
another one for it.

I sent email to unsub@. But still receiving emails.

Please suggest!!



From: Ravikumar Govindarajan <ravikumar.govindara...@gmail.com>
Sent: Thursday, August 10, 2017 6:37 PM
To: java-user@lucene.apache.org
Subject: Re: Storing external transaction log-ids in lucene...

Many thanks for the help...

Just one more clarification..

I also see that a sequence number is returned from IW on all mutations in
latest code including prepareCommit() [Apologize, as we are still on an old
version of lucene 4.6]

So the approach will be to co-relate the seq.no returned from individual
operations of IW with the external queue offset. Background commit thread
can then call prepareCommit() & it will exactly know what got through. Have
I understood it correctly?

Btw, we have a TrackingIndexWriter that returns a sequence number too, in
4.6. Is the new implementation something different from this?


On Thu, Aug 10, 2017 at 6:09 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to store
> this.
> Mike McCandless
> http://blog.mikemccandless.com
Changing Bits<http://blog.mikemccandless.com/>
The Apache Lucene project will likely release its next major release, 7.0, in a 
few months! Remember that Lucene developers generally try hard to backport new 

> On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan <
> ravikumar.govindara...@gmail.com> wrote:
> > Every mutation (Add/Update/Delete) has a transaction-id (incremental
> long)
> > assigned by our Messaging Queue (Kafka)
> >
> > To index these mutations, an indexer thread pulls data from the queue,
> adds
> > & commits to IndexWriter, then updates the latest transaction-id in an
> > external system (ZooKeeper). During rollback/server-restarts, the threads
> > read the previous value from ZK & resume...
> >
> > Have now moved to NRT implementation, where commit thread runs in the
> > background & am unable to find the latest transaction-id that got
> committed
> >
> > Initially thought of adding transaction-id as a first-class field of the
> > index itself & then writing a custom-codec to harvest it, but this fails
> > when a mutation-set consists only of deletes [as there is no way to pass
> > these ids via IW delete APIs]
> >
> > Is there a way to store this as part of segment-info?. Can
> setCommitData()
> > of IW help me here in any way?
> >
> > --
> > Ravi
> >

Reply via email to