Re: What is The logic of Vary Headers cachiness?
On 25/07/2013 6:37 p.m., Henrik Nordström wrote: tor 2013-07-25 klockan 17:23 +1200 skrev Amos Jeffries: We think alike. See my other reply. Although the lookup per-match is not required. HTTP only requires that we identify a whole set of options and pull the most appropriate - with some requirements around defining "appropriate". newest match is hightly preferable. Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. It feels like this is kind of separate optimization. It can be restricted to Vary transactions, but does not have to be. I agree. It is not clear which cases of Vary handling this would be appropriate for. We do however need to fix bug #7 once and for all. The map used in squid-2 do not really work well in cache validations. The problem is that the full objects gets revalidated when we get a new 304 response, when actually it's only the response that matches this request that have been revalidated. There is also a problem with too much churn in the x-vary marker object. Which problem specifically? that churn exists? that it can grow big + churn? races between clients? or that letting it out to disk can cause churn to be slooow? I have been playing with the idea of locking these into memory cache, or using a dedicated memory area just for them to avoid the speed issues. A specialized store for them will also allow us to isolate the secondary-lookup logic in that stores lookup process - it can identify the variant and recurse down to other stores for the final selection using the extra key bits. I believe that they can be generated from a disk scan and if necessary we can add swap.state TLV entries for the missing x-vary meta details to be reloaded quickly. That would make them churn particularly badly on startup, but avoid the necessity to store anywhere long-term, and help detect obsolete variants undeleted from disk. Amos
[PATCH] Fix an assert with side effects in ServerStateData::handleAdaptedHeader
This patch simply moves the adaptedBodySource->setConsumerIfNotLate(this) call out of an assert() in ServerStateData::handleAdaptedHeader. setConsumerIfNotLate conditionally schedules some async calls, so the call definitely shouldn't be removed by the precompiler. -- Nathan Hoad Software Developer www.getoffmalawn.com assert-with-side-effects.patch Description: Binary data
Re: What is The logic of Vary Headers cachiness?
tor 2013-07-25 klockan 17:23 +1200 skrev Amos Jeffries: > We think alike. See my other reply. Although the lookup per-match is not > required. HTTP only requires that we identify a whole set of options and > pull the most appropriate - with some requirements around defining > "appropriate". newest match is hightly preferable. > >> Or better yet allow the same response body to be > >> referenced by multiple responses (simplifies validation logics and aging > >> considerably). Response headers is the original response updated with > >> header data from the 304 response in response to If-None-Match. > > It feels like this is kind of separate optimization. It can be > > restricted to Vary transactions, but does not have to be. > > I agree. It is not clear which cases of Vary handling this would be > appropriate for. We do however need to fix bug #7 once and for all. The map used in squid-2 do not really work well in cache validations. The problem is that the full objects gets revalidated when we get a new 304 response, when actually it's only the response that matches this request that have been revalidated. There is also a problem with too much churn in the x-vary marker object. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
ons 2013-07-24 klockan 18:46 -0600 skrev Alex Rousskov: > > > The full logics (still missing from squid-3) involves an n-m map of > > requests to response variants. > > I thought Squid already tries to support that. Or are you talking about > responses with varying Vary header values (discussed further below)? Squid-2 does, but Squid-3 only keeps a 1-1 map. > > This has direct impact on where Vary logics can be performed. > > Can you elaborate on this point? I think there are ways to support full > Vary logics with and without "all variants are in one store" restrictions. It has an impact on how many store api accesses is needed to resolve a request for a varying URL. > > * Add mapping of request headers to specific response variant > > (identified by ETag). > > AFAIK, this is already supported by adjusting the store key. No, there is no way to query the store which ETags are known for a URL. All it knows is that the URL response varies. > > * Lookup the response matching this specific request. > > This is already supported by looking up the computed key in the store index. Sort of. That lookup is today based on URL + response headers which is correct for objects with no ETag. > > To complicate matters further different responses MAY have different > > Vary header value. But it is not very likely. > > True. To support that, we would need to add a list of Vary header values > to the special Vary object and iterate it, looking up a cache object for > every listed Vary header value. It's a little more complicated than that. > > Or better yet allow the same response body to be > > referenced by multiple responses (simplifies validation logics and aging > > considerably). Response headers is the original response updated with > > header data from the 304 response in response to If-None-Match. > > It feels like this is kind of separate optimization. It can be > restricted to Vary transactions, but does not have to be. I don't think it's separate. It has direct impact on how you can store responses to a varying URL and how to handle cache validation. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
On 25/07/2013 12:46 p.m., Alex Rousskov wrote: On 07/24/2013 05:55 PM, Henrik Nordström wrote: ons 2013-07-24 klockan 10:01 -0600 skrev Alex Rousskov: That is what Squid does today, bugs notwithstanding. If the found store entry is the special "Vary" entry, then Squid does another lookup, with the appropriate header values added to the store key hash. Yes, but is only part of what is needed for correct Vary support. The full logics (still missing from squid-3) involves an n-m map of requests to response variants. I thought Squid already tries to support that. Or are you talking about responses with varying Vary header values (discussed further below)? Finally, as we are migrating to per-cache store indexes, more store lookups should be avoided when possible because the number of mandatory lookups has to be multiplied by (the number of cache_dirs plus one for the memory cache index) to check all the indexes. There is a design decision that needs to be taken here.. should it be possible to have different responses for the same Vary:ing URL to be stored in different stores, or should they all need to go into the same store? I think it should be possible for them to be in different stores -- the store selection decision should be done at a higher level. Multiple stores are usually a side effect of hardware layout or some size-related optimizations. They usually live on a lower level than "HTTP caching" and we should not add "all variants have to be in one area of the cache" restrictions unless there is a very good reason for them. This has direct impact on where Vary logics can be performed. Can you elaborate on this point? I think there are ways to support full Vary logics with and without "all variants are in one store" restrictions. Full Vary support requires the following store operations * Which responses (ETag values) is known for given URL? This list is needed to construct a If-None-Match validation request as needed to ask upstream which variant is the correct response. This can be done using the special Vary object updated whenever a new variant is cached or an old variant is purged. I do not know whether there is already code to support such updates. If you mean the x-vary-marker object linking to any variants like I keep going on about. No there is nothing stored in it at present. It just contains the last Vary header seen inits header set so the recipient code can get it out and generate a new lookup key. * Add mapping of request headers to specific response variant (identified by ETag). AFAIK, this is already supported by adjusting the store key. Yes, but the code doing that readjustment needs to be shifted and radically redesigned in line with any changes to the x-vary-marker object for better references. * Lookup the response matching this specific request. This is already supported by looking up the computed key in the store index. To complicate matters further different responses MAY have different Vary header value. But it is not very likely. True. To support that, we would need to add a list of Vary header values to the special Vary object and iterate it, looking up a cache object for every listed Vary header value. We think alike. See my other reply. Although the lookup per-match is not required. HTTP only requires that we identify a whole set of options and pull the most appropriate - with some requirements around defining "appropriate". Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. It feels like this is kind of separate optimization. It can be restricted to Vary transactions, but does not have to be. I agree. It is not clear which cases of Vary handling this would be appropriate for. We do however need to fix bug #7 once and for all. Amos
Re: What is The logic of Vary Headers cachiness?
On 25/07/2013 11:55 a.m., Henrik Nordström wrote: ons 2013-07-24 klockan 10:01 -0600 skrev Alex Rousskov: That is what Squid does today, bugs notwithstanding. If the found store entry is the special "Vary" entry, then Squid does another lookup, with the appropriate header values added to the store key hash. Yes, but is only part of what is needed for correct Vary support. The full logics (still missing from squid-3) involves an n-m map of requests to response variants. Instead of returning a special Vary object during this first lookup, Squid could return a regular cached object (one of the Vary variants), with some special Vary flag set, Yes, ideally the vary lookup logics should fit in the store, and not upper layers. Was not really an option when the Vary logics was originally designed in Squid-2. I was intending to work up a body payload for the x-vary-marker object which contained that N-M map you speak of above. With the pending Key header feature this makes even more sense. The redirected lookup can of course be done internally by the store overall, but it can't be all in one dir class due to size differences. From my point of view on the code and after coding StoreID I know that there are two lookups for HEAD and GET and they are not the same object So why not just use 3 object level checking?? GET and HEAD are handled as different objects due to Squid design. It's not meant to from HTTP point of view. I am afraid I do not know what you mean by a "3 object level checking". Neither do I. IIRC Squid does not lookup HEAD and GET for the same request under normal conditions. Lookups with multiple methods happen for HEAD requests and for purging. Both categories are relatively rare. There is some cross-magics between HEAD and GET to try to map Squid store semantics to what HTTP expects, but far from complete. And IMHO unecessary complexity. Finally, as we are migrating to per-cache store indexes, more store lookups should be avoided when possible because the number of mandatory lookups has to be multiplied by (the number of cache_dirs plus one for the memory cache index) to check all the indexes. There is a design decision that needs to be taken here.. should it be possible to have different responses for the same Vary:ing URL to be stored in different stores, or should they all need to go into the same store? Makign them all us the same store will in some cases violate the min-size/max-size limitations. It also restricts us from having something like a special index for looking up the x-vary-marker objects which does sub-lookup logics and protects from duplicated logics in any storage media with unusual index searching (ie the remote DB cloudy stores being played with nowdays). This has direct impact on where Vary logics can be performed. Full Vary support requires the following store operations * Which responses (ETag values) is known for given URL? This list is needed to construct a If-None-Match validation request as needed to ask upstream which variant is the correct response. The x-vary-marker payload being the N-M index voids this impact. ETag can be a meta data stored there and avoid sub-lookups entirely on many cases. * Add mapping of request headers to specific response variant (identified by ETag). Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. The x-vary-marker being an N-M index again voids this impact. Using the response field-values instead of the full request header field covers this. Again no sub-lookup required. * Lookup the response matching this specific request. To complicate matters further different responses MAY have different Vary header value. But it is not very likely. With Key header this becomes a whole lot more likely. In fact there is a relatively strong case for each response having both a Vary and Key header in most responses. Making four identifier patterns per response variant (ETag, Vary, Key, and Digest hash when we get there). Having the x-vary-marker payload as the N-M map in the form of a list of variant pattern matches *all* of which are applied to the request then selected from (in direct accordance with HTTP specs algorithm) we can fully identify the set of available variants before the second lookup pulls out a specific one. Working with sets like this is exactly how the specs are phrased. Amos
Re: [PATCH] update port protocol parameter
On 25/07/2013 4:08 p.m., Alex Rousskov wrote: On 07/22/2013 09:36 AM, Amos Jeffries wrote: How does this one suit you? " Update the http(s)_port directives protocol= parameter to use AnyP::ProtocolVersion for internal storage instead of opaque string text. This both limits the possible parameter values to one of HTTP, HTTP/1.1, HTTPS or HTTPS/1.1 and ensures that URLs generated from that protocol parameter value are http:// and https:// URL. " You may want to add a note that previously allowed but now illegal protocol values will result in Squid quitting. With that note and the constructor removal which I did shortly after posting the patch. Are you happy for this to go in? Amos
Re: [PATCH] update port protocol parameter
On 07/22/2013 09:36 AM, Amos Jeffries wrote: > How does this one suit you? > > " > Update the http(s)_port directives protocol= parameter to use > AnyP::ProtocolVersion for internal storage instead of opaque string text. > > This both limits the possible parameter values to one of HTTP, HTTP/1.1, > HTTPS or HTTPS/1.1 and ensures that URLs generated from that protocol > parameter value are http:// and https:// URL. > " You may want to add a note that previously allowed but now illegal protocol values will result in Squid quitting. Thank you, Alex.
Re: What is The logic of Vary Headers cachiness?
On 07/24/2013 05:55 PM, Henrik Nordström wrote: > ons 2013-07-24 klockan 10:01 -0600 skrev Alex Rousskov: > >> That is what Squid does today, bugs notwithstanding. If the found store >> entry is the special "Vary" entry, then Squid does another lookup, with >> the appropriate header values added to the store key hash. > > Yes, but is only part of what is needed for correct Vary support. > > The full logics (still missing from squid-3) involves an n-m map of > requests to response variants. I thought Squid already tries to support that. Or are you talking about responses with varying Vary header values (discussed further below)? >> Finally, as we are migrating to per-cache store indexes, more store >> lookups should be avoided when possible because the number of mandatory >> lookups has to be multiplied by (the number of cache_dirs plus one for >> the memory cache index) to check all the indexes. > > There is a design decision that needs to be taken here.. should it be > possible to have different responses for the same Vary:ing URL to be > stored in different stores, or should they all need to go into the same > store? I think it should be possible for them to be in different stores -- the store selection decision should be done at a higher level. Multiple stores are usually a side effect of hardware layout or some size-related optimizations. They usually live on a lower level than "HTTP caching" and we should not add "all variants have to be in one area of the cache" restrictions unless there is a very good reason for them. > This has direct impact on where Vary logics can be performed. Can you elaborate on this point? I think there are ways to support full Vary logics with and without "all variants are in one store" restrictions. > Full Vary support requires the following store operations > > * Which responses (ETag values) is known for given URL? This list is > needed to construct a If-None-Match validation request as needed to ask > upstream which variant is the correct response. This can be done using the special Vary object updated whenever a new variant is cached or an old variant is purged. I do not know whether there is already code to support such updates. > * Add mapping of request headers to specific response variant > (identified by ETag). AFAIK, this is already supported by adjusting the store key. > * Lookup the response matching this specific request. This is already supported by looking up the computed key in the store index. > To complicate matters further different responses MAY have different > Vary header value. But it is not very likely. True. To support that, we would need to add a list of Vary header values to the special Vary object and iterate it, looking up a cache object for every listed Vary header value. > Or better yet allow the same response body to be > referenced by multiple responses (simplifies validation logics and aging > considerably). Response headers is the original response updated with > header data from the 304 response in response to If-None-Match. It feels like this is kind of separate optimization. It can be restricted to Vary transactions, but does not have to be. Cheers, Alex.
Re: What is The logic of Vary Headers cachiness?
ons 2013-07-24 klockan 10:01 -0600 skrev Alex Rousskov: > That is what Squid does today, bugs notwithstanding. If the found store > entry is the special "Vary" entry, then Squid does another lookup, with > the appropriate header values added to the store key hash. Yes, but is only part of what is needed for correct Vary support. The full logics (still missing from squid-3) involves an n-m map of requests to response variants. > Instead of returning a special Vary object during this first lookup, > Squid could return a regular cached object (one of the Vary variants), > with some special Vary flag set, Yes, ideally the vary lookup logics should fit in the store, and not upper layers. Was not really an option when the Vary logics was originally designed in Squid-2. > > From my point of view on the code and after coding StoreID I know that > > there are two lookups for HEAD and GET and they are not the same object > > So why not just use 3 object level checking?? GET and HEAD are handled as different objects due to Squid design. It's not meant to from HTTP point of view. > I am afraid I do not know what you mean by a "3 object level checking". Neither do I. > IIRC Squid does not lookup HEAD and GET for the same request under > normal conditions. Lookups with multiple methods happen for HEAD > requests and for purging. Both categories are relatively rare. There is some cross-magics between HEAD and GET to try to map Squid store semantics to what HTTP expects, but far from complete. > Finally, as we are migrating to per-cache store indexes, more store > lookups should be avoided when possible because the number of mandatory > lookups has to be multiplied by (the number of cache_dirs plus one for > the memory cache index) to check all the indexes. There is a design decision that needs to be taken here.. should it be possible to have different responses for the same Vary:ing URL to be stored in different stores, or should they all need to go into the same store? This has direct impact on where Vary logics can be performed. Full Vary support requires the following store operations * Which responses (ETag values) is known for given URL? This list is needed to construct a If-None-Match validation request as needed to ask upstream which variant is the correct response. * Add mapping of request headers to specific response variant (identified by ETag). Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. * Lookup the response matching this specific request. To complicate matters further different responses MAY have different Vary header value. But it is not very likely. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
On 07/24/2013 03:43 AM, Eliezer Croitoru wrote: > On 07/17/2013 02:34 PM, Amos Jeffries wrote: >> On 17/07/2013 10:29 p.m., Eliezer Croitoru wrote: >>> As I have seen some issues that "indicate" that the Vary headers cause >>> some problems while caching.. >>> >>> I want to make sure I understand how a Vary headers should be treated >>> before diving into some code. >> >> Um, the "shodul be" rather than what Squid is currently doing with its >> bugs... >> >> On request: >> * lookup the URL / store-ID in the index >> ==> finds the x-vary-* object created by Squid for storeing the >> vary_headers detail > > Here or one step before it should only look for the hash of the url and > then open the file if exists and verify if it's a vary object at all. That is what Squid does today, bugs notwithstanding. If the found store entry is the special "Vary" entry, then Squid does another lookup, with the appropriate header values added to the store key hash. > If the server has vary then we can still serve non-vary objects but > first make sure that there is a reason to serve Vary like object from > origin rather then just invalidate any cache option exists by Bogus Vary > headers. You lost me here. The initial lookup described above is unavoidable. At the first lookup time, Squid does not yet know whether the cached response(s) have Vary or not, so Squid has to use a regular lookup (no Vary-controlled header values added to the store key). Instead of returning a special Vary object during this first lookup, Squid could return a regular cached object (one of the Vary variants), with some special Vary flag set, but that is not how the code was written. The current implementation is probably a lot simpler, albeit probably not the most efficient than the alternatives. > From my point of view on the code and after coding StoreID I know that > there are two lookups for HEAD and GET and they are not the same object > So why not just use 3 object level checking?? > This will add a little overhead on the cpu but not on memory since the > objects are already there. > As the example of in a 16 cores system there is more CPU then squid.. we > can afford this one lookup and forget about loosing performance. > Objections? I am afraid I do not know what you mean by a "3 object level checking". IIRC Squid does not lookup HEAD and GET for the same request under normal conditions. Lookups with multiple methods happen for HEAD requests and for purging. Both categories are relatively rare. If you are suggesting that Squid starts with a lookup using Vary-listed request header values, please note that Squid does not know which request headers are under Vary control. Squid gets that information from the special Vary object which is found during the first regular lookup. Finally, as we are migrating to per-cache store indexes, more store lookups should be avoided when possible because the number of mandatory lookups has to be multiplied by (the number of cache_dirs plus one for the memory cache index) to check all the indexes. HTH, Alex. >> * looks up the URL+vary / store_id+vary in the index >> ==> finds the real object >> ==> deal with caching using that variants response headers and the >> request headers, same as any non-variant object would be handled if the >> x-vary step had not taken place. >> >> Also, I think if the variant needs to be invalidated Squid currently >> coded to drop all variants and/or the main x-vary-* stub object during >> revalidation. The HTTP/1.1 specs need to be checked to see if that is >> right or if only the one variant object should be invalidated and the >> others left for later requests to alter. >> >> >>> >>> My assumption is that there are "Vary" headers that the servers might be >>> considering while answering the request. >>> >>> If the above is indeed right and my assumption of how squid caches vary >>> headers There is a problem either in finding the corresponded responses >>> or something else. >>> Please help me understand how is the logic of The Vary headers works. >>> >>> Let say I have a requeset: >>> >>> GET /resource.dll HTTP/1.1 >>> Host: example.net >>> Accept-Encoding: *.* >>> - >>> >>> the corresponding request will be: >>> >>> GET /resource.dll HTTP/1.1 >>> Host: example.net >>> - >>> >>> since there is no Vary field that indicates there is a Vary header >>> present in the request, or the existence of a so called "Vary" header >>> like "Accept-Encoding" is like stating "consider this vary header"? >> >> The request *never* contains any indication of a Vary header. This is >> why the stub x-vary-* object exists. Only after Squid looks up its cache >> index and finds that x-vary stub object does it know that variants exist >> on this object and the x-vary objects "vary_headers" details tells it >> how to look up the particular variant needed by this request. >> >>> if the request will be the same then OK. >>> if the response is not the same then we h
Re: What is The logic of Vary Headers cachiness?
On 07/17/2013 02:34 PM, Amos Jeffries wrote: > On 17/07/2013 10:29 p.m., Eliezer Croitoru wrote: >> As I have seen some issues that "indicate" that the Vary headers cause >> some problems while caching.. >> >> I want to make sure I understand how a Vary headers should be treated >> before diving into some code. > > Um, the "shodul be" rather than what Squid is currently doing with its > bugs... > > On request: > * lookup the URL / store-ID in the index > ==> finds the x-vary-* object created by Squid for storeing the > vary_headers detail Here or one step before it should only look for the hash of the url and then open the file if exists and verify if it's a vary object at all. If the server has vary then we can still serve non-vary objects but first make sure that there is a reason to serve Vary like object from origin rather then just invalidate any cache option exists by Bogus Vary headers. >From my point of view on the code and after coding StoreID I know that there are two lookups for HEAD and GET and they are not the same object So why not just use 3 object level checking?? This will add a little overhead on the cpu but not on memory since the objects are already there. As the example of in a 16 cores system there is more CPU then squid.. we can afford this one lookup and forget about loosing performance. Objections? Eliezer > > * looks up the URL+vary / store_id+vary in the index > ==> finds the real object > ==> deal with caching using that variants response headers and the > request headers, same as any non-variant object would be handled if the > x-vary step had not taken place. > > Also, I think if the variant needs to be invalidated Squid currently > coded to drop all variants and/or the main x-vary-* stub object during > revalidation. The HTTP/1.1 specs need to be checked to see if that is > right or if only the one variant object should be invalidated and the > others left for later requests to alter. > > >> >> My assumption is that there are "Vary" headers that the servers might be >> considering while answering the request. >> >> If the above is indeed right and my assumption of how squid caches vary >> headers There is a problem either in finding the corresponded responses >> or something else. >> Please help me understand how is the logic of The Vary headers works. >> >> Let say I have a requeset: >> >> GET /resource.dll HTTP/1.1 >> Host: example.net >> Accept-Encoding: *.* >> - >> >> the corresponding request will be: >> >> GET /resource.dll HTTP/1.1 >> Host: example.net >> - >> >> since there is no Vary field that indicates there is a Vary header >> present in the request, or the existence of a so called "Vary" header >> like "Accept-Encoding" is like stating "consider this vary header"? > > The request *never* contains any indication of a Vary header. This is > why the stub x-vary-* object exists. Only after Squid looks up its cache > index and finds that x-vary stub object does it know that variants exist > on this object and the x-vary objects "vary_headers" details tells it > how to look up the particular variant needed by this request. > >> if the request will be the same then OK. >> if the response is not the same then we have a problem. >> Would we calculate the Vary based on the request only or based on also >> the response?? and then when the response is not matching the request >> what will be the hash of the request? > > Calculated based on the request headers (only) using the key details > stored in x-vary fake response object. > > Amos >
Re: Dmitry Kurochkin
On Tue, Jul 23, 2013 at 7:18 PM, Alex Rousskov wrote: > It is with great sadness I inform you that Dmitry Kurochkin died in a > skydiving accident a few days ago. Dmitry was an avid skydiver, with > more than 360 jumps and some regional records behind him. He loved that > sport. > > Dmitry's recent contributions to Squid include code related to HTTP/1.1 > compliance, SMP scalability, SMP Rock Store, Large Rock, Collapsed > Forwarding, and FTP gateway features. He also worked on automating Squid > compliance testing with Co-Advisor and Squid performance testing with > Web Polygraph. > > Dmitry was a wonderful person, a talented developer, and a key member of > the Factory team. He was a pleasure to work with. We miss him badly. This is very sad news. Condolences to everyone at TMF, his family, and all the people affected by this tragic loss. -- /kinkie
Re: Dmitry Kurochkin
On 07/23/2013 08:18 PM, Alex Rousskov wrote: > It is with great sadness I inform you that Dmitry Kurochkin died in a > skydiving accident a few days ago. Dmitry was an avid skydiver, with > more than 360 jumps and some regional records behind him. He loved that > sport. > > Dmitry's recent contributions to Squid include code related to HTTP/1.1 > compliance, SMP scalability, SMP Rock Store, Large Rock, Collapsed > Forwarding, and FTP gateway features. He also worked on automating Squid > compliance testing with Co-Advisor and Squid performance testing with > Web Polygraph. > > Dmitry was a wonderful person, a talented developer, and a key member of > the Factory team. He was a pleasure to work with. We miss him badly. > > > Alex. > My consolations to his family and friends...
