Re: Replication and ignored fields
Not sure about that. I have read that the replication handler actually issues a commit() on itself once the index is downloaded. But probably a better way for Markus' case is to hook the prune job on the master, writing to another core (myIndexPruned). Then you replicate from that core instead, and you also get the benefit of transferring a smaller index across the network. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com On 8. nov. 2010, at 23.50, Shalin Shekhar Mangar wrote: On Fri, Nov 5, 2010 at 2:30 PM, Jan Høydahl / Cominvent jan@cominvent.com wrote: How about hooking in Andrzej's pruning tool at the postCommit event, literally removing unused fields. I believe a commit is fired on the slave by itself after every successful replication, to put the index live. You could execute a script which prunes away the dead meat and then call a new commit? Well, I don't think it will work because a new commit will cause the index version on the slave to be ahead of the master which will cause Solr replication to download a full index from the master and it'd go in an infinite loop. -- Regards, Shalin Shekhar Mangar.
Re: Replication and ignored fields
On Tue, Nov 9, 2010 at 12:33 AM, Jan Høydahl / Cominvent jan@cominvent.com wrote: Not sure about that. I have read that the replication handler actually issues a commit() on itself once the index is downloaded. That was true with the old replication scripts. The Java based replication just re-opens the IndexReader after all the files are downloaded so the index version on the slave remains in sync with the one on the master. But probably a better way for Markus' case is to hook the prune job on the master, writing to another core (myIndexPruned). Then you replicate from that core instead, and you also get the benefit of transferring a smaller index across the network. I agree, that is a good idea. -- Regards, Shalin Shekhar Mangar.
Re: Replication and ignored fields
Cool, thanks for the clarification, Shalin. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com On 9. nov. 2010, at 15.12, Shalin Shekhar Mangar wrote: On Tue, Nov 9, 2010 at 12:33 AM, Jan Høydahl / Cominvent jan@cominvent.com wrote: Not sure about that. I have read that the replication handler actually issues a commit() on itself once the index is downloaded. That was true with the old replication scripts. The Java based replication just re-opens the IndexReader after all the files are downloaded so the index version on the slave remains in sync with the one on the master. But probably a better way for Markus' case is to hook the prune job on the master, writing to another core (myIndexPruned). Then you replicate from that core instead, and you also get the benefit of transferring a smaller index across the network. I agree, that is a good idea. -- Regards, Shalin Shekhar Mangar.
Replication and ignored fields
Hi, I've got an ordinary master/slave replication set up. The master contains several fields that are not used by the slaves but are used by processes that interact with the master. Removing the fields from the master is not an option. Well, to save disk space i'd figure i create an `ignored` fieldType and set the fields that are unused on the slaves to use the ignored fieldType. ..it doesn't work and makes perfectly sense because it's just the index files that get copied over. The question, how to ignore fields with replication? Cheers, -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536600 / 06-50258350
Re: Replication and ignored fields
How about hooking in Andrzej's pruning tool at the postCommit event, literally removing unused fields. I believe a commit is fired on the slave by itself after every successful replication, to put the index live. You could execute a script which prunes away the dead meat and then call a new commit? http://wiki.apache.org/solr/SolrConfigXml#A.22Update.22_Related_Event_Listeners http://www.lucidimagination.com/solutions/webinars/mastering-the-lucene-index -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com On 5. nov. 2010, at 16.11, Markus Jelsma wrote: Hi, I've got an ordinary master/slave replication set up. The master contains several fields that are not used by the slaves but are used by processes that interact with the master. Removing the fields from the master is not an option. Well, to save disk space i'd figure i create an `ignored` fieldType and set the fields that are unused on the slaves to use the ignored fieldType. ..it doesn't work and makes perfectly sense because it's just the index files that get copied over. The question, how to ignore fields with replication? Cheers, -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536600 / 06-50258350
Re: Replication and ignored fields
Thanks for the pointer! How about hooking in Andrzej's pruning tool at the postCommit event, literally removing unused fields. I believe a commit is fired on the slave by itself after every successful replication, to put the index live. You could execute a script which prunes away the dead meat and then call a new commit? http://wiki.apache.org/solr/SolrConfigXml#A.22Update.22_Related_Event_Liste ners http://www.lucidimagination.com/solutions/webinars/mastering-the-lucene-in dex -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com On 5. nov. 2010, at 16.11, Markus Jelsma wrote: Hi, I've got an ordinary master/slave replication set up. The master contains several fields that are not used by the slaves but are used by processes that interact with the master. Removing the fields from the master is not an option. Well, to save disk space i'd figure i create an `ignored` fieldType and set the fields that are unused on the slaves to use the ignored fieldType. ..it doesn't work and makes perfectly sense because it's just the index files that get copied over. The question, how to ignore fields with replication? Cheers,
