Every mutation (Add/Update/Delete) has a transaction-id (incremental long) assigned by our Messaging Queue (Kafka)
Advertising
To index these mutations, an indexer thread pulls data from the queue, adds & commits to IndexWriter, then updates the latest transaction-id in an external system (ZooKeeper). During rollback/server-restarts, the threads read the previous value from ZK & resume... Have now moved to NRT implementation, where commit thread runs in the background & am unable to find the latest transaction-id that got committed Initially thought of adding transaction-id as a first-class field of the index itself & then writing a custom-codec to harvest it, but this fails when a mutation-set consists only of deletes [as there is no way to pass these ids via IW delete APIs] Is there a way to store this as part of segment-info?. Can setCommitData() of IW help me here in any way? -- Ravi
