Hi Sorry for sending the below..what I meant to say was is there any documentation that I can be pointed to with using lucene and rsync? I been up since 2am so brain slowing down really quickly...
Cheers Amin On Tue, Jun 23, 2009 at 10:13 AM, Amin Mohammed-Coleman <ami...@gmail.com>wrote: > Hi > > Thanks for your replies. Is there any documentation that I can look at for > using rsync? I am thinking of creating my own solution (early days) I might > come back to Solr. > > > Cheers > Amin > > > On Mon, Jun 22, 2009 at 10:24 AM, Michael McCandless < > luc...@mikemccandless.com> wrote: > >> Solr (as of 1.4) is moving to Java only implementation for replication >> (http://wiki.apache.org/solr/SolrReplication). >> >> If you roll your own, make sure to only run rsync at "proper" times >> according to the master, eg, it's probably simplest to close the >> writer on the master before running rsync. If you don't want to pause >> indexing when copying, then you should use SnapshotDeletionPolicy to >> get the list of files that need copying and pass that list to rsync. >> If you simply rsync whenever you want to, and a writer is open on the >> index, it's easy to get a corrupt copy. >> >> Mike >> >> On Mon, Jun 22, 2009 at 5:09 AM, Ian Lea<ian....@gmail.com> wrote: >> > Or if you don't want to use Solr, rsync is very good for this. Only >> takes >> > changes, takes care of deletions, very robust. That is what Solr uses. >> > >> > See http://wiki.apache.org/solr/CollectionDistribution for details or >> ideas. >> > >> > >> > -- >> > Ian. >> > >> > On Sun, Jun 21, 2009 at 4:32 AM, Otis Gospodnetic < >> > otis_gospodne...@yahoo.com> wrote: >> > >> >> >> >> Hi Amin, >> >> >> >> Have a look at Solr, it may be what you are after: >> >> http://lucene.apache.org/solr/ >> >> >> >> Otis >> >> -- >> >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >> >> >> >> >> >> >> ----- Original Message ---- >> >> > From: Amin Mohammed-Coleman <ami...@gmail.com> >> >> > To: java-user@lucene.apache.org >> >> > Sent: Saturday, June 20, 2009 1:32:18 PM >> >> > Subject: Lucene directory copy - master copy to local index >> >> > >> >> > Hi >> >> > I am prototyping the following situation: >> >> > >> >> > 1) Multiple nodes in a cluster >> >> > 2) Each node has a local index >> >> > 3) Search requests are maded against the local index >> >> > 4) Index updates are sent to a JMS where a master process adds >> document >> >> to >> >> > index >> >> > 5) Each node is configured to check whether the local index is out of >> >> date >> >> > and needs to copy from master index. >> >> > >> >> > I have created a class that will be configured (quartz job) to copy >> >> master >> >> > directory files to the local directory. I was wondering if I could >> get >> >> > advice on whether this is the best way to copy master directory >> changes >> >> to >> >> > the local. >> >> > >> >> > public class LocalIndexProvider { >> >> > >> >> > private Directory masterDirectory; >> >> > >> >> > private Directory localDirectory; >> >> > >> >> > private final Logger LOGGER = Logger.getLogger(getClass()); >> >> > >> >> > public LocalIndexProvider(Directory masterDirectory, Directory >> >> > localDirectory) { >> >> > >> >> > this.masterDirectory = masterDirectory; >> >> > >> >> > this.localDirectory = localDirectory; >> >> > >> >> > } >> >> > >> >> > public void updateLocalIndex() throws Exception { >> >> > >> >> > StopWatch stopWatch = new StopWatch("copy-time"); >> >> > >> >> > stopWatch.start(); >> >> > >> >> > String[] masterFiles = masterDirectory.list(); >> >> > >> >> > String[] localFiles = localDirectory.list(); >> >> > >> >> > Lock lock = localDirectory.getLockFactory().makeLock("test"); >> >> > >> >> > lock.obtain(); >> >> > >> >> > try { >> >> > >> >> > if (localFiles.length != masterFiles.length) { >> >> > >> >> > Directory.copy(masterDirectory, localDirectory, false); >> >> > >> >> > } >> >> > >> >> > }finally { >> >> > >> >> > lock.release(); >> >> > >> >> > } >> >> > >> >> > stopWatch.stop(); >> >> > >> >> > LOGGER.debug("total time taken :" + stopWatch.getTotalTimeMillis()); >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > >> >> > >> >> > The above is just a proof of concept and I am open to any comments >> about >> >> it. >> >> > It may be inefficient or it may be. I have looked at the source code >> >> from >> >> > hibernate search and it seems as though they copy each file from the >> >> > directory (master to local). I'm not sure whether this is something >> I >> >> need >> >> > to do or the above (with more refinement) would be enough. >> >> > >> >> > Any help would be highly appreciated. >> >> > >> >> > Cheers >> >> > Amin >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> >> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >
