Many thanks. This is a real cool feature & saves us a lot of time !!!

--
Ravi

On Thu, Aug 10, 2017 at 9:40 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> That's exactly right!  That is the purpose of the sequence numbers returned
> by IndexWriter mutations.  You can know exactly which ops made it into your
> commit and which didn't.
>
> TrackingIndexWriter is replaced by the sequence numbers.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Aug 10, 2017 at 9:37 AM, Ravikumar Govindarajan <
> ravikumar.govindara...@gmail.com> wrote:
>
> > Many thanks for the help...
> >
> > Just one more clarification..
> >
> > I also see that a sequence number is returned from IW on all mutations in
> > latest code including prepareCommit() [Apologize, as we are still on an
> old
> > version of lucene 4.6]
> >
> > So the approach will be to co-relate the seq.no returned from individual
> > operations of IW with the external queue offset. Background commit thread
> > can then call prepareCommit() & it will exactly know what got through.
> Have
> > I understood it correctly?
> >
> > Btw, we have a TrackingIndexWriter that returns a sequence number too, in
> > 4.6. Is the new implementation something different from this?
> >
> > --
> > Ravi
> >
> > On Thu, Aug 10, 2017 at 6:09 PM, Michael McCandless <
> > luc...@mikemccandless.com> wrote:
> >
> > > IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to
> > store
> > > this.
> > >
> > > Mike McCandless
> > >
> > > http://blog.mikemccandless.com
> > >
> > > On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan <
> > > ravikumar.govindara...@gmail.com> wrote:
> > >
> > > > Every mutation (Add/Update/Delete) has a transaction-id (incremental
> > > long)
> > > > assigned by our Messaging Queue (Kafka)
> > > >
> > > > To index these mutations, an indexer thread pulls data from the
> queue,
> > > adds
> > > > & commits to IndexWriter, then updates the latest transaction-id in
> an
> > > > external system (ZooKeeper). During rollback/server-restarts, the
> > threads
> > > > read the previous value from ZK & resume...
> > > >
> > > > Have now moved to NRT implementation, where commit thread runs in the
> > > > background & am unable to find the latest transaction-id that got
> > > committed
> > > >
> > > > Initially thought of adding transaction-id as a first-class field of
> > the
> > > > index itself & then writing a custom-codec to harvest it, but this
> > fails
> > > > when a mutation-set consists only of deletes [as there is no way to
> > pass
> > > > these ids via IW delete APIs]
> > > >
> > > > Is there a way to store this as part of segment-info?. Can
> > > setCommitData()
> > > > of IW help me here in any way?
> > > >
> > > > --
> > > > Ravi
> > > >
> > >
> >
>

Reply via email to