RE: Embedded for write, HTTP for read - cache aging

2009-12-07 Thread Peter 4U

Hi Erik,

 

Thanks for your answer.

 

Yes, I've done an /update to the http server, which certainly works as far as 
the 'reading' goes.

This sends the update to the back-end index though, which essentially defeats 
the purpose of having the embedded instance do the write (as writes are always 
local, but reads might be remote, the goal is for super-fast writes, at the 
potential cost of slower reads). Maybe the http server can be set as 
'Read-only' (redirected /update handler) so that it doesn't hit the back-end 
indexer, but still tells it to check the index on the next read?

 

The main performance bottleneck isn't Solr itself, but the HTTP 
wrapping/transmission.

At low traffic rates, it really makes no difference at all.

But when you get into 1000's writes/sec the http wrapping and transmission 
becomes more and more significant as the traffic rate rises. On average, we've 
seen ~3-8% efficiency increase at very high rates (using a typical Windows TCP 
stack). This might not seem like much, but at really high screaming input 
rates, it does make a difference.

The EmbeddedSolr instance itself wraps each request into an XML request, so I 
believe the performance of the EmbeddedSolr instance could be increased if it 
handled requests without any wrapping at all (NamedList).

 

Thanks,

Peter

 


 
 From: erik.hatc...@gmail.com
 To: solr-user@lucene.apache.org
 Subject: Re: Embedded for write, HTTP for read - cache aging
 Date: Mon, 7 Dec 2009 05:49:01 +0100
 
 
 On Dec 5, 2009, at 12:56 PM, Peter 4U wrote:
  Does anyone know of a way to tell an http SolrServer to reload its 
  back-end index (mark cache as dirty) periodically?
 
 Send a commit/ to the HTTP SolrServer.
 
  I have a scenario where an EmbeddedSolrServer is used for writing 
  (for fast indexing), and an
 
  CommonsHttpSolrServer for reading (for remote access).
 
 I'm curious, now much faster is it in your situation?
 
 Erik
 
  
_
Have more than one Hotmail account? Link them together to easily access both
 http://clk.atdmt.com/UKM/go/186394591/direct/01/

Re: Embedded for write, HTTP for read - cache aging

2009-12-06 Thread Erik Hatcher


On Dec 5, 2009, at 12:56 PM, Peter 4U wrote:
Does anyone know of a way to tell an http SolrServer to reload its  
back-end index (mark cache as dirty) periodically?


Send a commit/ to the HTTP SolrServer.

I have a scenario where an EmbeddedSolrServer is used for writing  
(for fast indexing), and an


CommonsHttpSolrServer for reading (for remote access).


I'm curious, now much faster is it in your situation?

Erik



Embedded for write, HTTP for read - cache aging

2009-12-05 Thread Peter 4U

Hello,

 

Does anyone know of a way to tell an http SolrServer to reload its back-end 
index (mark cache as dirty) periodically?

 

I have a scenario where an EmbeddedSolrServer is used for writing (for fast 
indexing), and an

CommonsHttpSolrServer for reading (for remote access).

 

If the http server is used for writing, reading clients pick up any updates, as 
the /update has gone 'through' the http server.

For very high indexing rates, I'd rather not have to build an http request for 
every write (or group of writes), since the writer is always on the same 
machine as the index.

 

Any help on this is much appreciated.

 

Thanks,

Peter

 
  
_
View your other email accounts from your Hotmail inbox. Add them now.
http://clk.atdmt.com/UKM/go/186394592/direct/01/