Re: Lucene cluster with NFS or synchronization tool such as rsync

2016-07-04 Thread Desteny Child
Hi Mike, Thanks you very much for your response. I would be really grateful if you can please provide me with an information where I can read(may be with examples) about new near-real-time replication ? Thanks, Alex 2016-07-04 12:57 GMT+03:00 Michael McCandless : > NFS is dangerous if differen

Re: lucene index reader performance

2016-07-04 Thread Tarun Kumar
Hey Michael, docIds from multiple indices (from multiple machines) need to be aggregated, sorted and first few thousand new to be queried. These few thousand docs can be distributed among multiple machines. Each machine will search the docs which are there in their own indices. So, pulling sorting

Re: Lucene cluster with NFS or synchronization tool such as rsync

2016-07-04 Thread Michael McCandless
NFS is dangerous if different nodes may take turns writing to the shared index. Locking sometimes doesn't work correctly, client-side metadata caching (e.g. the directory entry) can cause problems, NFS doesn't support "delete on final close" semantics that Lucene relies on. rsync-like behavior ca

Re: lucene index reader performance

2016-07-04 Thread Michael McCandless
Why not ask Lucene to do the sort on your time field, instead of pulling millions of docids to the client and having it sort. You could even do index-time sorting by time field if you want, which makes early termination possible (faster sorted searches). But if even on having Lucene do the sort y