Many thanks for the help...

Just one more clarification..

I also see that a sequence number is returned from IW on all mutations in
latest code including prepareCommit() [Apologize, as we are still on an old
version of lucene 4.6]

So the approach will be to co-relate the seq.no returned from individual
operations of IW with the external queue offset. Background commit thread
can then call prepareCommit() & it will exactly know what got through. Have
I understood it correctly?

Btw, we have a TrackingIndexWriter that returns a sequence number too, in
4.6. Is the new implementation something different from this?

--
Ravi

On Thu, Aug 10, 2017 at 6:09 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to store
> this.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan <
> ravikumar.govindara...@gmail.com> wrote:
>
> > Every mutation (Add/Update/Delete) has a transaction-id (incremental
> long)
> > assigned by our Messaging Queue (Kafka)
> >
> > To index these mutations, an indexer thread pulls data from the queue,
> adds
> > & commits to IndexWriter, then updates the latest transaction-id in an
> > external system (ZooKeeper). During rollback/server-restarts, the threads
> > read the previous value from ZK & resume...
> >
> > Have now moved to NRT implementation, where commit thread runs in the
> > background & am unable to find the latest transaction-id that got
> committed
> >
> > Initially thought of adding transaction-id as a first-class field of the
> > index itself & then writing a custom-codec to harvest it, but this fails
> > when a mutation-set consists only of deletes [as there is no way to pass
> > these ids via IW delete APIs]
> >
> > Is there a way to store this as part of segment-info?. Can
> setCommitData()
> > of IW help me here in any way?
> >
> > --
> > Ravi
> >
>

Reply via email to