Hi Mike, Robert Thanks for replying, the system is almost like what Mike has described: one writer is primary, and the other is trying to catch up and wait, but in our internal discussion we found there might be small chances where the secondary mistakenly think itself as primary (due to errors of other component) while primary is still alive and thus goes into the situation I described. And because we want to tolerate the error in case we can't prevent it from happening, we're looking for customizing filenames.
Thanks again for discussing this with me and I've learnt that playing with filenames can become quite troublesome, but still, even out of my own curiosity, I want to understand whether we're able to control the segment names in some way? Best Patrick On Fri, Dec 16, 2022 at 6:36 AM Michael Sokolov <msoko...@gmail.com> wrote: > +1 trying to coordinate multiple writers running independently will > not work. My 2c for availability: you can have a single primary active > writer with a backup one waiting, receiving all the segments from the > primary. Then if the primary goes down, the secondary one has the most > recent commit replicated from the primary (identical commit, same > segments etc) and can pick up from there. You would need a mechanism > to replay the writes the primary never had a chance to commit. > > On Fri, Dec 16, 2022 at 5:41 AM Robert Muir <rcm...@gmail.com> wrote: > > > > You are still talking "Multiple writers". Like i said, going down this > > path (playing tricks with filenames) isn't going to work out well. > > > > On Fri, Dec 16, 2022 at 2:48 AM Patrick Zhai <zhai7...@gmail.com> wrote: > > > > > > Hi Robert, > > > > > > Maybe I didn't explain it clearly but we're not going to constantly > switch > > > between writers or share effort between writers, it's purely for > > > availability: the second writer only kicks in when the first writer is > not > > > available for some reason. > > > And as far as I know the replicator/nrt module has not provided a > solution > > > on when the primary node (main indexer) is down, how would we recover > with > > > a back up indexer? > > > > > > Thanks > > > Patrick > > > > > > > > > On Thu, Dec 15, 2022 at 7:16 PM Robert Muir <rcm...@gmail.com> wrote: > > > > > > > This multiple-writer isn't going to work and customizing names won't > > > > allow it anyway. Each file also contains a unique identifier tied to > > > > its commit so that we know everything is intact. > > > > > > > > I would look at the segment replication in lucene/replicator and not > > > > try to play games with files and mixing multiple writers. > > > > > > > > On Thu, Dec 15, 2022 at 5:45 PM Patrick Zhai <zhai7...@gmail.com> > wrote: > > > > > > > > > > Hi Folks, > > > > > > > > > > We're trying to build a search architecture using segment > replication > > > > (indexer and searcher are separated and indexer shipping new > segments to > > > > searchers) right now and one of the problems we're facing is: for > > > > availability reason we need to have multiple indexers running, and > when the > > > > searcher is switching from consuming one indexer to another, there > are > > > > chances where the segment names collide with each other (because > segment > > > > names are count based) and the searcher have to reload the whole > index. > > > > > To avoid that we're looking for a way to name the segments so that > > > > Lucene is able to tell the difference and load only the difference > (by > > > > calling `openIfChanged`). I've checked the IndexWriter and the > > > > DocumentsWriter and it seems it is controlled by a private final > method > > > > `newSegmentName()` so likely not possible there. So I wonder whether > > > > there's any other ways people are aware of that can help control the > > > > segment names? > > > > > > > > > > A example of the situation described above: > > > > > Searcher previously consuming from indexer 1, and have following > > > > segments: _1, _2, _3, _4 > > > > > Indexer 2 previously sync'd from indexer 1, sharing the first 3 > > > > segments, and produced its own 4th segments (notioned as _4', but it > shares > > > > the same "_4" name): _1, _2, _3, _4' > > > > > Suddenly Indexer 1 dies and searcher switched from Indexer 1 to > Indexer > > > > 2, then when it finished downloading the segments and trying to > refresh the > > > > reader, it will likely hit the exception here, and seems all we can > do > > > > right now is to reload the whole index and that could be potentially > a high > > > > cost. > > > > > > > > > > Sorry for the long email and thank you in advance for any replies! > > > > > > > > > > Best > > > > > Patrick > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
