In this context I would also note that while copying is progress, an attempt to open the directory might report a corrupted index due to a commit not having all of its files available or due to partially copied files. For example, if your query servers are calling `DirectoryReader.openIfChanged()` in regular intervals, sooner or later they will crash. So you should coordinate reader refresh with the synchronization. Viliam
On Mon, Aug 25, 2025 at 6:05 PM Steven Schlansker < stevenschlans...@gmail.com> wrote: > Hi, we use Lucene NRT replication in production. > > For consistent snapshots, we use SnapshotDeletionPolicy to open a > snapshot, and then copy the snapshot'ed > files with your tool of choice like rsync. Without a snapshot, I don't > think such tools work reliably - you can copy > commit metadata without the data it refers to, for example. > > We've run Lucene NRT replication in production for about 4 years now. > Overall very happy with it. We intentionally run a simple configuration > (one indexer, N searchers) > to avoid difficult problems around leader election and network > partitioning. > > Definitely took a bit of learning, but overall we would choose this path > again. > It is more "hands on" than just spinning up an Elasticsearch cluster - but > on the other hand > our previous Elasticsearch cluster took a team just just for "care and > feeding" as the cluster had > split-brain issues etc. Compare to Lucene NRT replication, where a single > dev and no ops people > can easily manage the entire stack, since all state is on a single master > node. > > > On Aug 25, 2025, at 8:36 AM, Adrien Grand <jpou...@gmail.com> wrote: > > > > Hello, > > > > If you can guarantee no concurrent updates while you are doing the copy, > > then 3rd-party tools that replicate files such as fsync are fine. But > > otherwise or if you have the choice, I would recommend to use Lucene's > > built-in support for replication, which can also work if there are > > concurrent updates to the index. > > > > What is NRTLuceneReplication? I cannot find references to it. Lucene's > > replicator module does have NRT (near-realtime) support through: > > > https://lucene.apache.org/core/10_2_0/replicator/org/apache/lucene/replicator/nrt/package-summary.html > > . > > > > On Mon, Aug 25, 2025 at 5:18 PM sandy A <santhoshmaddy1...@gmail.com> > wrote: > > > >> Hi Lucene Community Team, > >> > >> I have a couple of queries related to Lucene replication and would > >> appreciate your guidance: > >> > >> *Query 1:* > >> Is it safe to use tools like *rsync* (on Linux) or *robocopy* (on > Windows) > >> for copying Lucene segment files from one server to another? > >> I want to understand if there are any potential risks (e.g., partial > >> copies, consistency issues) or best practices recommended for such > >> approaches. > >> > >> *Query 2:* > >> I noticed that *NRTLuceneReplication* is still marked as "in > development." > >> Could you please clarify why it is not yet considered production-ready? > Are > >> there known limitations or stability concerns that we should be aware of > >> before experimenting with it? > >> > >> Any insights or suggestions would be really helpful. > >> > >> > >> With Regards, > >> SanthoshKumar A > >> > > > > > > -- > > Adrien > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
