Re: 2.1 plans
]] Ingvar Hagelund | * Tollef Fog Heen | > a short while before Christmas, I wrote up a small document pointing to | > what I would like to get into 2.1 (...) | > Varnish 2.1 release plan | > (...) | | Will 2.1 be backwards compatible with vcl and configuration settings | from 2.0? | | ie, will I be able to just upgrade a binary pre-built package and | restart the server? Yes, I don't see any need to break backwards compatibility for 2.1. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
On Jan 9, 2009, at 1:59 AM, Tollef Fog Heen wrote: > | What about CARP-like cache routing (i.e., where multiple cache > servers > | themselves are hash buckets)? This would go a LONG way towards > | scalability. > > http://varnish.projects.linpro.no/wiki/PostTwoShoppingList second item > sounds like what you want? Yup, sounds like what I proposed about a year ago :-) --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
]] "Michael S. Fischer" | What about CARP-like cache routing (i.e., where multiple cache servers | themselves are hash buckets)? This would go a LONG way towards | scalability. http://varnish.projects.linpro.no/wiki/PostTwoShoppingList second item sounds like what you want? -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
]] "BUSTARRET, Jean-francois" | What about more ESI features (ie : cookie support), backend | revalidation with conditional GETs or streaming fetches | (http://varnish.projects.linpro.no/wiki/PostTwoShoppingList) ? More ESI support would be interesting too. However, we have a limited amount of manpower and in order for 2.1 to actually release sometime this summer rather than next summer, I'd like us to limit the feature list somewhat. If patches show up, I'm sure we can find the time to integrate those, though. :-) -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
+1. This is a very good idea for optimizing RAM utilization. --Michael On Jan 8, 2009, at 11:25 AM, Jeff Anderson wrote: > Thanks for the response. > > I think inline page compression would be great too. Store gzipped > objects in the persistent cache and unzip if uncompressed objects are > requested. > > > > > On Jan 8, 2009, at 10:54 AM, Per Buer wrote: > >> Jeff Anderson wrote: >>> I'd like to see individual object request statistics and a method to >>> prefetch objects from the backend that are most frequently >>> requested. >>> Perhaps also a way to prioritize objects into cache tiers based on >>> frequency of requests. So, for example, highly requested objects >>> are >>> maintained in RAM and less frequently requested objects are cached >>> to >>> disk. >> >> Your operating system already does this today with Varnish. Squid >> tries >> to maintain a two tier cache hierarchy without success. >> >>> If persistent storage is on its way maybe a method to assign >>> priority to large disk cache volumes versus memory regions. >> >> Noted. >> >>> It might be nice to have a distributed and/or tiered cache model >>> where a single >>> master has a very large cache and potentially very long grace >>> ability >>> where objects can exist even if stale. That master in turn could >>> host >>> frontend caches that communicate efficiently to the master cache >>> and >>> also have a tiered internal object priority. >> >> I believe most of this can be achieved today. Stale objects will >> hopefully reach the 2.0 series before the 2.1 revolutions - at least >> as >> a patch, I hope. >> >>> >>> Thanks, >>> --Jeff >>> >>> >>> On Jan 8, 2009, at 2:29 AM, Tollef Fog Heen wrote: >>> Hi, a short while before Christmas, I wrote up a small document pointing to what I would like to get into 2.1 and when I'd like milestones to happen. This is a suggestion, I'm open to ideas and comments on both feature set as well as if my guesstimates for dates is completely off: Varnish 2.1 release plan The theme for Varnish 2.1 is "scalability", particularly trying to address the needs of sites like finn.no which has a lot of objects and where priming the cache takes a long time, leading to long periods of higher load on the backend servers. The main feature is persistent storage, see http://varnish.projects.linpro.no/wiki/ ArchitecturePersistentStorage for design notes. Another important scalability feature is a new lockless hash algorithm which scales much better than the current one. Poul-Henning already has an implementation of this in the tree, but it's still fresh. Minor features which would be nice to get in are: * Web UI, showing pretty graphs as well as allowing easy configuration of a cluster of Varnish machines. * Expiry randomisation. This reduces the "lemmings" effect where you end up with a many objects with almost the same TTL (typically on startup) which then expire at the same time. The feature will allow you to set the TTL to plus/minus X %. * Dynamic, user-defined counters that can be read and written from VCL * Forced purges, where a thread walks the list of purged objects and removes them. The schedule Alphas: - 2009-01-15: New hash algorithm working - 2009-02-15: Web UI - 2009-03-15: Persistent storage Beta: - 2009-04-01: Feature complete Release - 2009-05-20: Release candidate - 2009-05-01: No release critical bugs left - 2009-05-10: Release -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc >>> >>> --Jeff >>> j...@funnyordie.com >>> >>> >>> >>> ___ >>> varnish-misc mailing list >>> varnish-misc@projects.linpro.no >>> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> >> -- >> Per Buer - Leder Infrastruktur og Drift - Redpill Linpro >> Telefon: 21 54 41 21 - Mobil: 958 39 117 >> http://linpro.no/ | http://redpill.se/ >> >> ___ >> varnish-misc mailing list >> varnish-misc@projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc > > --Jeff > j...@funnyordie.com > > > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
Thanks for the response. I think inline page compression would be great too. Store gzipped objects in the persistent cache and unzip if uncompressed objects are requested. On Jan 8, 2009, at 10:54 AM, Per Buer wrote: > Jeff Anderson wrote: >> I'd like to see individual object request statistics and a method to >> prefetch objects from the backend that are most frequently requested. >> Perhaps also a way to prioritize objects into cache tiers based on >> frequency of requests. So, for example, highly requested objects are >> maintained in RAM and less frequently requested objects are cached to >> disk. > > Your operating system already does this today with Varnish. Squid > tries > to maintain a two tier cache hierarchy without success. > >> If persistent storage is on its way maybe a method to assign >> priority to large disk cache volumes versus memory regions. > > Noted. > >> It might be nice to have a distributed and/or tiered cache model >> where a single >> master has a very large cache and potentially very long grace ability >> where objects can exist even if stale. That master in turn could >> host >> frontend caches that communicate efficiently to the master cache and >> also have a tiered internal object priority. > > I believe most of this can be achieved today. Stale objects will > hopefully reach the 2.0 series before the 2.1 revolutions - at least > as > a patch, I hope. > >> >> Thanks, >> --Jeff >> >> >> On Jan 8, 2009, at 2:29 AM, Tollef Fog Heen wrote: >> >>> Hi, >>> >>> a short while before Christmas, I wrote up a small document pointing >>> to >>> what I would like to get into 2.1 and when I'd like milestones to >>> happen. This is a suggestion, I'm open to ideas and comments on >>> both >>> feature set as well as if my guesstimates for dates is completely >>> off: >>> >>> Varnish 2.1 release plan >>> >>> The theme for Varnish 2.1 is "scalability", particularly trying to >>> address the needs of sites like finn.no which has a lot of objects >>> and >>> where priming the cache takes a long time, leading to long periods >>> of >>> higher load on the backend servers. >>> >>> The main feature is persistent storage, see >>> http://varnish.projects.linpro.no/wiki/ArchitecturePersistentStorage >>> for design notes. Another important scalability feature is a new >>> lockless hash algorithm which scales much better than the current >>> one. Poul-Henning already has an implementation of this in the >>> tree, >>> but it's still fresh. >>> >>> Minor features which would be nice to get in are: >>> >>> * Web UI, showing pretty graphs as well as allowing easy >>> configuration >>> of a cluster of Varnish machines. >>> >>> * Expiry randomisation. This reduces the "lemmings" effect where >>> you >>> end up with a many objects with almost the same TTL (typically on >>> startup) which then expire at the same time. The feature will allow >>> you to set the TTL to plus/minus X %. >>> >>> * Dynamic, user-defined counters that can be read and written from >>> VCL >>> >>> * Forced purges, where a thread walks the list of purged objects and >>> removes them. >>> >>> The schedule >>> >>> >>> Alphas: >>> - 2009-01-15: New hash algorithm working >>> - 2009-02-15: Web UI >>> - 2009-03-15: Persistent storage >>> Beta: >>> - 2009-04-01: Feature complete >>> Release >>> - 2009-05-20: Release candidate >>> - 2009-05-01: No release critical bugs left >>> - 2009-05-10: Release >>> >>> -- >>> Tollef Fog Heen >>> Redpill Linpro -- Changing the game! >>> t: +47 21 54 41 73 >>> ___ >>> varnish-misc mailing list >>> varnish-misc@projects.linpro.no >>> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> --Jeff >> j...@funnyordie.com >> >> >> >> ___ >> varnish-misc mailing list >> varnish-misc@projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc > > > -- > Per Buer - Leder Infrastruktur og Drift - Redpill Linpro > Telefon: 21 54 41 21 - Mobil: 958 39 117 > http://linpro.no/ | http://redpill.se/ > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc --Jeff j...@funnyordie.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
Hi, Tollef Fog Heen wrote: > Hi, > > a short while before Christmas, I wrote up a small document pointing to > what I would like to get into 2.1 and when I'd like milestones to > happen. This is a suggestion, I'm open to ideas and comments on both > feature set as well as if my guesstimates for dates is completely off: > > Varnish 2.1 release plan > (..) As the money man in the project I would like to say that if anybody misses anything on the feature list we are happy to talk to sponsors about it. Send me or varn...@linpro.no an email and we'll discuss it. Your feature might not go into the mainline right away but we'll be happy to support a patched version of varnish until we are able to put the patch into a proper release. Regards Per. -- http://linpro.no/ | http://redpill.se/ signature.asc Description: OpenPGP digital signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
Jeff Anderson wrote: > I'd like to see individual object request statistics and a method to > prefetch objects from the backend that are most frequently requested. > Perhaps also a way to prioritize objects into cache tiers based on > frequency of requests. So, for example, highly requested objects are > maintained in RAM and less frequently requested objects are cached to > disk. Your operating system already does this today with Varnish. Squid tries to maintain a two tier cache hierarchy without success. > If persistent storage is on its way maybe a method to assign > priority to large disk cache volumes versus memory regions. Noted. > It might be nice to have a distributed and/or tiered cache model where a > single > master has a very large cache and potentially very long grace ability > where objects can exist even if stale. That master in turn could host > frontend caches that communicate efficiently to the master cache and > also have a tiered internal object priority. I believe most of this can be achieved today. Stale objects will hopefully reach the 2.0 series before the 2.1 revolutions - at least as a patch, I hope. > > Thanks, > --Jeff > > > On Jan 8, 2009, at 2:29 AM, Tollef Fog Heen wrote: > >> Hi, >> >> a short while before Christmas, I wrote up a small document pointing >> to >> what I would like to get into 2.1 and when I'd like milestones to >> happen. This is a suggestion, I'm open to ideas and comments on both >> feature set as well as if my guesstimates for dates is completely off: >> >> Varnish 2.1 release plan >> >> The theme for Varnish 2.1 is "scalability", particularly trying to >> address the needs of sites like finn.no which has a lot of objects and >> where priming the cache takes a long time, leading to long periods of >> higher load on the backend servers. >> >> The main feature is persistent storage, see >> http://varnish.projects.linpro.no/wiki/ArchitecturePersistentStorage >> for design notes. Another important scalability feature is a new >> lockless hash algorithm which scales much better than the current >> one. Poul-Henning already has an implementation of this in the tree, >> but it's still fresh. >> >> Minor features which would be nice to get in are: >> >> * Web UI, showing pretty graphs as well as allowing easy configuration >> of a cluster of Varnish machines. >> >> * Expiry randomisation. This reduces the "lemmings" effect where you >> end up with a many objects with almost the same TTL (typically on >> startup) which then expire at the same time. The feature will allow >> you to set the TTL to plus/minus X %. >> >> * Dynamic, user-defined counters that can be read and written from VCL >> >> * Forced purges, where a thread walks the list of purged objects and >> removes them. >> >> The schedule >> >> >> Alphas: >> - 2009-01-15: New hash algorithm working >> - 2009-02-15: Web UI >> - 2009-03-15: Persistent storage >> Beta: >> - 2009-04-01: Feature complete >> Release >> - 2009-05-20: Release candidate >> - 2009-05-01: No release critical bugs left >> - 2009-05-10: Release >> >> -- >> Tollef Fog Heen >> Redpill Linpro -- Changing the game! >> t: +47 21 54 41 73 >> ___ >> varnish-misc mailing list >> varnish-misc@projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc > > --Jeff > j...@funnyordie.com > > > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ signature.asc Description: OpenPGP digital signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
I'd like to see individual object request statistics and a method to prefetch objects from the backend that are most frequently requested. Perhaps also a way to prioritize objects into cache tiers based on frequency of requests. So, for example, highly requested objects are maintained in RAM and less frequently requested objects are cached to disk. If persistent storage is on its way maybe a method to assign priority to large disk cache volumes versus memory regions. It might be nice to have a distributed and/or tiered cache model where a single master has a very large cache and potentially very long grace ability where objects can exist even if stale. That master in turn could host frontend caches that communicate efficiently to the master cache and also have a tiered internal object priority. Thanks, --Jeff On Jan 8, 2009, at 2:29 AM, Tollef Fog Heen wrote: > > Hi, > > a short while before Christmas, I wrote up a small document pointing > to > what I would like to get into 2.1 and when I'd like milestones to > happen. This is a suggestion, I'm open to ideas and comments on both > feature set as well as if my guesstimates for dates is completely off: > > Varnish 2.1 release plan > > The theme for Varnish 2.1 is "scalability", particularly trying to > address the needs of sites like finn.no which has a lot of objects and > where priming the cache takes a long time, leading to long periods of > higher load on the backend servers. > > The main feature is persistent storage, see > http://varnish.projects.linpro.no/wiki/ArchitecturePersistentStorage > for design notes. Another important scalability feature is a new > lockless hash algorithm which scales much better than the current > one. Poul-Henning already has an implementation of this in the tree, > but it's still fresh. > > Minor features which would be nice to get in are: > > * Web UI, showing pretty graphs as well as allowing easy configuration > of a cluster of Varnish machines. > > * Expiry randomisation. This reduces the "lemmings" effect where you > end up with a many objects with almost the same TTL (typically on > startup) which then expire at the same time. The feature will allow > you to set the TTL to plus/minus X %. > > * Dynamic, user-defined counters that can be read and written from VCL > > * Forced purges, where a thread walks the list of purged objects and > removes them. > > The schedule > > > Alphas: > - 2009-01-15: New hash algorithm working > - 2009-02-15: Web UI > - 2009-03-15: Persistent storage > Beta: > - 2009-04-01: Feature complete > Release > - 2009-05-20: Release candidate > - 2009-05-01: No release critical bugs left > - 2009-05-10: Release > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc --Jeff j...@funnyordie.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.1 plans
What about CARP-like cache routing (i.e., where multiple cache servers themselves are hash buckets)? This would go a LONG way towards scalability. --Michael On Jan 8, 2009, at 2:29 AM, Tollef Fog Heen wrote: > > Hi, > > a short while before Christmas, I wrote up a small document pointing > to > what I would like to get into 2.1 and when I'd like milestones to > happen. This is a suggestion, I'm open to ideas and comments on both > feature set as well as if my guesstimates for dates is completely off: > > Varnish 2.1 release plan > > The theme for Varnish 2.1 is "scalability", particularly trying to > address the needs of sites like finn.no which has a lot of objects and > where priming the cache takes a long time, leading to long periods of > higher load on the backend servers. > > The main feature is persistent storage, see > http://varnish.projects.linpro.no/wiki/ArchitecturePersistentStorage > for design notes. Another important scalability feature is a new > lockless hash algorithm which scales much better than the current > one. Poul-Henning already has an implementation of this in the tree, > but it's still fresh. > > Minor features which would be nice to get in are: > > * Web UI, showing pretty graphs as well as allowing easy configuration > of a cluster of Varnish machines. > > * Expiry randomisation. This reduces the "lemmings" effect where you > end up with a many objects with almost the same TTL (typically on > startup) which then expire at the same time. The feature will allow > you to set the TTL to plus/minus X %. > > * Dynamic, user-defined counters that can be read and written from VCL > > * Forced purges, where a thread walks the list of purged objects and > removes them. > > The schedule > > > Alphas: > - 2009-01-15: New hash algorithm working > - 2009-02-15: Web UI > - 2009-03-15: Persistent storage > Beta: > - 2009-04-01: Feature complete > Release > - 2009-05-20: Release candidate > - 2009-05-01: No release critical bugs left > - 2009-05-10: Release > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: 2.1 plans
Great news ! What about more ESI features (ie : cookie support), backend revalidation with conditional GETs or streaming fetches (http://varnish.projects.linpro.no/wiki/PostTwoShoppingList) ? Jean-François > -Message d'origine- > De : varnish-misc-boun...@projects.linpro.no > [mailto:varnish-misc-boun...@projects.linpro.no] De la part > de Tollef Fog Heen > Envoyé : jeudi 8 janvier 2009 11:30 > À : varnish-misc@projects.linpro.no > Objet : 2.1 plans > > > Hi, > > a short while before Christmas, I wrote up a small document > pointing to > what I would like to get into 2.1 and when I'd like milestones to > happen. This is a suggestion, I'm open to ideas and comments on both > feature set as well as if my guesstimates for dates is completely off: > > Varnish 2.1 release plan > > The theme for Varnish 2.1 is "scalability", particularly trying to > address the needs of sites like finn.no which has a lot of objects and > where priming the cache takes a long time, leading to long periods of > higher load on the backend servers. > > The main feature is persistent storage, see > http://varnish.projects.linpro.no/wiki/ArchitecturePersistentStorage > for design notes. Another important scalability feature is a new > lockless hash algorithm which scales much better than the current > one. Poul-Henning already has an implementation of this in the tree, > but it's still fresh. > > Minor features which would be nice to get in are: > > * Web UI, showing pretty graphs as well as allowing easy configuration > of a cluster of Varnish machines. > > * Expiry randomisation. This reduces the "lemmings" effect where you > end up with a many objects with almost the same TTL (typically on > startup) which then expire at the same time. The feature will allow > you to set the TTL to plus/minus X %. > > * Dynamic, user-defined counters that can be read and written from VCL > > * Forced purges, where a thread walks the list of purged objects and > removes them. > > The schedule > > > Alphas: > - 2009-01-15: New hash algorithm working > - 2009-02-15: Web UI > - 2009-03-15: Persistent storage > Beta: > - 2009-04-01: Feature complete > Release > - 2009-05-20: Release candidate > - 2009-05-01: No release critical bugs left > - 2009-05-10: Release > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
