That's exactly right!  That is the purpose of the sequence numbers returned
by IndexWriter mutations.  You can know exactly which ops made it into your
commit and which didn't.

TrackingIndexWriter is replaced by the sequence numbers.

Mike McCandless

http://blog.mikemccandless.com

On Thu, Aug 10, 2017 at 9:37 AM, Ravikumar Govindarajan <
ravikumar.govindara...@gmail.com> wrote:

> Many thanks for the help...
>
> Just one more clarification..
>
> I also see that a sequence number is returned from IW on all mutations in
> latest code including prepareCommit() [Apologize, as we are still on an old
> version of lucene 4.6]
>
> So the approach will be to co-relate the seq.no returned from individual
> operations of IW with the external queue offset. Background commit thread
> can then call prepareCommit() & it will exactly know what got through. Have
> I understood it correctly?
>
> Btw, we have a TrackingIndexWriter that returns a sequence number too, in
> 4.6. Is the new implementation something different from this?
>
> --
> Ravi
>
> On Thu, Aug 10, 2017 at 6:09 PM, Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
> > IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to
> store
> > this.
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan <
> > ravikumar.govindara...@gmail.com> wrote:
> >
> > > Every mutation (Add/Update/Delete) has a transaction-id (incremental
> > long)
> > > assigned by our Messaging Queue (Kafka)
> > >
> > > To index these mutations, an indexer thread pulls data from the queue,
> > adds
> > > & commits to IndexWriter, then updates the latest transaction-id in an
> > > external system (ZooKeeper). During rollback/server-restarts, the
> threads
> > > read the previous value from ZK & resume...
> > >
> > > Have now moved to NRT implementation, where commit thread runs in the
> > > background & am unable to find the latest transaction-id that got
> > committed
> > >
> > > Initially thought of adding transaction-id as a first-class field of
> the
> > > index itself & then writing a custom-codec to harvest it, but this
> fails
> > > when a mutation-set consists only of deletes [as there is no way to
> pass
> > > these ids via IW delete APIs]
> > >
> > > Is there a way to store this as part of segment-info?. Can
> > setCommitData()
> > > of IW help me here in any way?
> > >
> > > --
> > > Ravi
> > >
> >
>

Reply via email to