Re: Varnish

2013-06-29 Thread William Bell 
OK.

Here is the answer for us. Here is a sample default.vcl. We are validating
the LastModified ( if (!beresp.http.last-modified) )
is changing when the core is indexed and the version changes of the index.

This does 10 minutes caching and a 1hr grace period (if solr is down, it
will deliver results up to 1 hr).

This uses the URL for caching.

You can also do:

http://localhost?PURGEME

To clear varnish if your IP is in the ACL list.


backend server1 {
.host = XXX.domain.com;
.port = 8983;
.probe = {
.url = /solr/pingall/select/?q=*%3A*;
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend server2{
.host = XXX1.domain.com;
.port = 8983;
.probe = {
.url = /solr/pingall/select/?q=*%3A*;
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend server3{
.host = XXX2.domain.com;
.port = 8983;
.probe = {
.url = /solr/pingall/select/?q=*%3A*;
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend server4{
.host = XXX3.domain.com;
.port = 8983;
.probe = {
.url = /solr/pingall/select/?q=*%3A*;
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}

director default round-robin {
  {
.backend = server1;
  }
  {
.backend = server2;
  }
  {
.backend = server3;
  }
  {
.backend = server4;
  }
}

acl purge {
localhost;
10.0.1.0/24;
10.0.3.0/24;
}


sub vcl_recv {
   if (req.url ~ \?PURGEME$) {
if (!client.ip ~ purge) {
error 405 Not allowed.  + client.ip;
}
ban(req.url ~ /);
error 200 Cached Cleared;
   }
   remove req.http.Cookie;
   if (req.backend.healthy) {
 set req.grace = 15s;
   } else {
 set req.grace = 1h;
   }
   return (lookup);
}

sub vcl_fetch {
  set beresp.grace = 1h;
  if (!beresp.http.last-modified) {
set beresp.ttl = 600s;
  }
  if (beresp.ttl  600s) {
set beresp.ttl = 600s;
  }
  unset beresp.http.Set-Cookie;
}

sub vcl_deliver {
if (obj.hits  0) {
set resp.http.X-Cache = HIT;
} else {
set resp.http.X-Cache = MISS;
}
}

sub vcl_hash {
hash_data(req.url);
return (hash);
}






On Tue, Jun 25, 2013 at 4:44 PM, Learner bbar...@gmail.com wrote:

 Check this link..
 http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Varnish

2013-06-29 Thread William Bell 
On a large website, by putting 1 varnish in front of all 4 SOLR boxes we
were able to trim 25% off the load time (TTFB) of the page.

Our hit ratio was between 55 and 75%. We gave varnish 24GB of RAM, and was
not able to fill it under full load with a 10 minute cache timeout.

We get about 2.4M SOLR calls every 15 to 20 minutes.

One varnish was able to handle it with almost no lingering connections, and
load average of  1.

Varnish is very optimized and worth trying.



On Sat, Jun 29, 2013 at 6:47 PM, William Bell billnb...@gmail.com wrote:

 OK.

 Here is the answer for us. Here is a sample default.vcl. We are validating
 the LastModified ( if (!beresp.http.last-modified) )
 is changing when the core is indexed and the version changes of the index.

 This does 10 minutes caching and a 1hr grace period (if solr is down, it
 will deliver results up to 1 hr).

 This uses the URL for caching.

 You can also do:

 http://localhost?PURGEME

 To clear varnish if your IP is in the ACL list.


 backend server1 {
 .host = XXX.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
 }
 backend server2{
 .host = XXX1.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
 }
 backend server3{
 .host = XXX2.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
 }
 backend server4{
 .host = XXX3.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
 }

 director default round-robin {
   {
 .backend = server1;
   }
   {
 .backend = server2;
   }
   {
 .backend = server3;
   }
   {
 .backend = server4;
   }
 }

 acl purge {
 localhost;
 10.0.1.0/24;
 10.0.3.0/24;
 }


 sub vcl_recv {
if (req.url ~ \?PURGEME$) {
 if (!client.ip ~ purge) {
 error 405 Not allowed.  + client.ip;
 }
 ban(req.url ~ /);
 error 200 Cached Cleared;
}
remove req.http.Cookie;
if (req.backend.healthy) {
  set req.grace = 15s;
} else {
  set req.grace = 1h;
}
return (lookup);
 }

 sub vcl_fetch {
   set beresp.grace = 1h;
   if (!beresp.http.last-modified) {
 set beresp.ttl = 600s;
   }
   if (beresp.ttl  600s) {
 set beresp.ttl = 600s;
   }
   unset beresp.http.Set-Cookie;
 }

 sub vcl_deliver {
 if (obj.hits  0) {
 set resp.http.X-Cache = HIT;
 } else {
 set resp.http.X-Cache = MISS;
 }
 }

 sub vcl_hash {
 hash_data(req.url);
 return (hash);
 }






 On Tue, Jun 25, 2013 at 4:44 PM, Learner bbar...@gmail.com wrote:

 Check this link..
 http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
 Sent from the Solr - User mailing list archive at Nabble.com.




 --
 Bill Bell
 billnb...@gmail.com
 cell 720-256-8076




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Varnish

2013-06-29 Thread Lance Norskog 
Solr HTTP caching also support e-tags. These are unique keys for the 
output of a query. If you send a query twice, and the index has not 
changed, the return will be the same. The e-tag is generated from the 
query string and the index generation number.


If Varnish supports e-tags, you can keep some queries cached longer than 
your timeout.


Lance

On 06/29/2013 05:51 PM, William Bell wrote:

On a large website, by putting 1 varnish in front of all 4 SOLR boxes we
were able to trim 25% off the load time (TTFB) of the page.

Our hit ratio was between 55 and 75%. We gave varnish 24GB of RAM, and was
not able to fill it under full load with a 10 minute cache timeout.

We get about 2.4M SOLR calls every 15 to 20 minutes.

One varnish was able to handle it with almost no lingering connections, and
load average of  1.

Varnish is very optimized and worth trying.



On Sat, Jun 29, 2013 at 6:47 PM, William Bell billnb...@gmail.com wrote:


OK.

Here is the answer for us. Here is a sample default.vcl. We are validating
the LastModified ( if (!beresp.http.last-modified) )
is changing when the core is indexed and the version changes of the index.

This does 10 minutes caching and a 1hr grace period (if solr is down, it
will deliver results up to 1 hr).

This uses the URL for caching.

You can also do:

http://localhost?PURGEME

To clear varnish if your IP is in the ACL list.


backend server1 {
 .host = XXX.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
}
backend server2{
 .host = XXX1.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
}
backend server3{
 .host = XXX2.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
}
backend server4{
 .host = XXX3.domain.com;
 .port = 8983;
 .probe = {
 .url = /solr/pingall/select/?q=*%3A*;
 .interval = 5s;
 .timeout = 1s;
 .window = 5;
 .threshold = 3;
 }
}

director default round-robin {
   {
 .backend = server1;
   }
   {
 .backend = server2;
   }
   {
 .backend = server3;
   }
   {
 .backend = server4;
   }
}

acl purge {
 localhost;
 10.0.1.0/24;
 10.0.3.0/24;
}


sub vcl_recv {
if (req.url ~ \?PURGEME$) {
 if (!client.ip ~ purge) {
 error 405 Not allowed.  + client.ip;
 }
 ban(req.url ~ /);
 error 200 Cached Cleared;
}
remove req.http.Cookie;
if (req.backend.healthy) {
  set req.grace = 15s;
} else {
  set req.grace = 1h;
}
return (lookup);
}

sub vcl_fetch {
   set beresp.grace = 1h;
   if (!beresp.http.last-modified) {
 set beresp.ttl = 600s;
   }
   if (beresp.ttl  600s) {
 set beresp.ttl = 600s;
   }
   unset beresp.http.Set-Cookie;
}

sub vcl_deliver {
 if (obj.hits  0) {
 set resp.http.X-Cache = HIT;
 } else {
 set resp.http.X-Cache = MISS;
 }
}

sub vcl_hash {
 hash_data(req.url);
 return (hash);
}






On Tue, Jun 25, 2013 at 4:44 PM, Learner bbar...@gmail.com wrote:


Check this link..
http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html



--
View this message in context:
http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
Sent from the Solr - User mailing list archive at Nabble.com.




--
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Varnish

2013-06-25 Thread Learner 
Check this link..
http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Varnish

2013-06-24 Thread William Bell 
Yes. A lot of people are using it in front of SOLR to speed up and reduce
the Garbase Collections.



On Fri, Jun 21, 2013 at 9:44 AM, Jack Park jackp...@topicquests.org wrote:

 I presume you mean https://www.varnish-cache.org/
 That's the first I'd heard of it.

 Thanks
 Jack

 On Thu, Jun 20, 2013 at 10:48 PM, William Bell billnb...@gmail.com
 wrote:
  Who is using varnish in front of SOLR?
 
  Anyone have any configs that work with the cache control headers of SOLR?
 
  --
  Bill Bell
  billnb...@gmail.com
  cell 720-256-8076




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Varnish

2013-06-21 Thread Jack Park 
I presume you mean https://www.varnish-cache.org/
That's the first I'd heard of it.

Thanks
Jack

On Thu, Jun 20, 2013 at 10:48 PM, William Bell billnb...@gmail.com wrote:
 Who is using varnish in front of SOLR?

 Anyone have any configs that work with the cache control headers of SOLR?

 --
 Bill Bell
 billnb...@gmail.com
 cell 720-256-8076

Varnish

2013-06-20 Thread William Bell 
Who is using varnish in front of SOLR?

Anyone have any configs that work with the cache control headers of SOLR?

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076