Re: Should we remove ESI?
On 12/06/2013 8:52 a.m., Kinkie wrote: On Tue, Jun 11, 2013 at 5:00 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/11/2013 02:23 AM, Kinkie wrote: On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/09/2013 02:40 PM, Kinkie wrote: while attempting to increase portability to recent clang releases, I noticed that libTrie hasn't benefited from the portability work that was done in the past few years. I can see three ways to move forward: 1- replicate these changes into libTrie 2- change libTrie to piggyback squid's configuration variables 3- fully integrate libTrie into squid's build system. Unless Robert knows otherwise, squid is the only user of this library.. I cannot tell what libTrie does: The README file is empty and the commit message only implies that it is an ESI component. AFAICT, only ESI uses it today. From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. And there are other similar tools around -- we use one in Web Polygraph IIRC. It would be a good project to compare them and use the best for HTTP header lookup if (and only if) they show noticeable improvement. Then, depending on the winning tool, the best integration option can be found. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). Yes, but even bigger ESI-associated problems are, IMO: * ESI adaptations are not appropriate for a built-in Squid component. The content adaptation that ESI performs should be done using an optional module or service. If eCAP has (or can be enhanced to have) enough hooks, eCAP should be used for that. Otherwise, another API can be provided for all similar adaptations. * ESI may be the only big user of Client Streams, an unfinished API that is responsible for many client side problems. If there is no ESI, it would be much easier to restore client side code sanity. I agree that an inquiry on squid-users and a separate email to ESI sponsors is a good step forward _if_ there is no squid-dev consensus that ESI must stay as an integrated component. The consensus that ESI should ideally be turned into an eCAP module is overwhelming. As is the consensus that so far noone has the time to do it :( What is the second-best option? Second best option is to leave it alone for a while longer. It is currently in a mostly-working state with nobody either complaining or requesting new feature changes to it. I am getting along with re-architecting the client-side okay for now despite its existence. If it ever gets in the way too much it will get my attention at that time. The bigger problems for the re-arch effort right now are parser, SSL-bump, and tunnel.cc all playing with the client socket in special ways. NP: Alex and Kinkie if you two are able to focus any more effort on the StringNG project it would be great. I am nearly ready to start proposing parser design patches and I would much rather do that with SBuf capabilities. Amos
Re: Should we remove ESI?
On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/09/2013 02:40 PM, Kinkie wrote: while attempting to increase portability to recent clang releases, I noticed that libTrie hasn't benefited from the portability work that was done in the past few years. I can see three ways to move forward: 1- replicate these changes into libTrie 2- change libTrie to piggyback squid's configuration variables 3- fully integrate libTrie into squid's build system. Unless Robert knows otherwise, squid is the only user of this library.. I cannot tell what libTrie does: The README file is empty and the commit message only implies that it is an ESI component. AFAICT, only ESI uses it today. From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. I do not know much about ESI, but IMHO, if somebody has cycles to work on this, it would be best to spend them removing ESI (together with libtTrie) from Squid sources while converting ESI into an eCAP adapter. This will be a big step forward towards making client side code sane (but removing ESI itself does not require making complex changes to the client side code itself). Robert is the expert on this. My question right now is, is anyone using ESI? ESI requires a specifically-crafted mix of infrastructure and application; there are nowadays simpler ways to obtain similar results. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). -- /kinkie
Re: Should we remove ESI?
On 6/11/2013 11:23 AM, Kinkie wrote: On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/09/2013 02:40 PM, Kinkie wrote: while attempting to increase portability to recent clang releases, I noticed that libTrie hasn't benefited from the portability work that was done in the past few years. I can see three ways to move forward: 1- replicate these changes into libTrie 2- change libTrie to piggyback squid's configuration variables 3- fully integrate libTrie into squid's build system. Unless Robert knows otherwise, squid is the only user of this library.. I cannot tell what libTrie does: The README file is empty and the commit message only implies that it is an ESI component. AFAICT, only ESI uses it today. From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. I do not know much about ESI, but IMHO, if somebody has cycles to work on this, it would be best to spend them removing ESI (together with libtTrie) from Squid sources while converting ESI into an eCAP adapter. This will be a big step forward towards making client side code sane (but removing ESI itself does not require making complex changes to the client side code itself). Robert is the expert on this. My question right now is, is anyone using ESI? ESI requires a specifically-crafted mix of infrastructure and application; there are nowadays simpler ways to obtain similar results. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). -- /kinkie +1 from me to that. Eliezer
Re: Should we remove ESI?
On 11 June 2013 20:23, Kinkie gkin...@gmail.com wrote: On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov rouss...@measurement-factory.com wrote: From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. It is a generic trie implementation, it is very good at some forms of lookup, and it's used in ESI yes; I had planned to try it in the HTTP parser, but ETIME. I do not know much about ESI, but IMHO, if somebody has cycles to work on this, it would be best to spend them removing ESI (together with libtTrie) from Squid sources while converting ESI into an eCAP adapter. This will be a big step forward towards making client side code sane (but removing ESI itself does not require making complex changes to the client side code itself). Robert is the expert on this. My question right now is, is anyone using ESI? ESI requires a specifically-crafted mix of infrastructure and application; there are nowadays simpler ways to obtain similar results. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). We get occasional queries about it on IRC and the lists; I don't know if it's in use in production or not. I think it would be sad to remove working code, but if noone is using it, noone is using it. I think refactoring it to use eCap rather than clientStreams would be fine, but I can't volunteer to do that myself. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services
Re: Should we remove ESI?
On 11/06/2013 8:54 p.m., Robert Collins wrote: On 11 June 2013 20:23, Kinkie wrote: On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov wrote: From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. It is a generic trie implementation, it is very good at some forms of lookup, and it's used in ESI yes; I had planned to try it in the HTTP parser, but ETIME. I do not know much about ESI, but IMHO, if somebody has cycles to work on this, it would be best to spend them removing ESI (together with libtTrie) from Squid sources while converting ESI into an eCAP adapter. This will be a big step forward towards making client side code sane (but removing ESI itself does not require making complex changes to the client side code itself). Robert is the expert on this. My question right now is, is anyone using ESI? ESI requires a specifically-crafted mix of infrastructure and application; there are nowadays simpler ways to obtain similar results. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). We get occasional queries about it on IRC and the lists; I don't know if it's in use in production or not. I think it would be sad to remove working code, but if noone is using it, noone is using it. Yes. I know of a few installations actively using it as of about a year ago. One newspaper in .il, a blog installation, and some form of mapping/game system. I have tried experimenting with it a little myself, but run into problems with Apache and the surrogate pieces. The big lack of take-up has been that in the past it required a specialty build of Squid and we have not exactly promoted it much since it was polished up for default-enable in 3.1. The alternative XHR / jQuery / node.js mechanisms are quite well known, slightly more flexible and work without a proxy. I think refactoring it to use eCap rather than clientStreams would be fine, but I can't volunteer to do that myself. Ditto. Amos
Re: Should we remove ESI?
On 06/11/2013 02:23 AM, Kinkie wrote: On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/09/2013 02:40 PM, Kinkie wrote: while attempting to increase portability to recent clang releases, I noticed that libTrie hasn't benefited from the portability work that was done in the past few years. I can see three ways to move forward: 1- replicate these changes into libTrie 2- change libTrie to piggyback squid's configuration variables 3- fully integrate libTrie into squid's build system. Unless Robert knows otherwise, squid is the only user of this library.. I cannot tell what libTrie does: The README file is empty and the commit message only implies that it is an ESI component. AFAICT, only ESI uses it today. From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. And there are other similar tools around -- we use one in Web Polygraph IIRC. It would be a good project to compare them and use the best for HTTP header lookup if (and only if) they show noticeable improvement. Then, depending on the winning tool, the best integration option can be found. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). Yes, but even bigger ESI-associated problems are, IMO: * ESI adaptations are not appropriate for a built-in Squid component. The content adaptation that ESI performs should be done using an optional module or service. If eCAP has (or can be enhanced to have) enough hooks, eCAP should be used for that. Otherwise, another API can be provided for all similar adaptations. * ESI may be the only big user of Client Streams, an unfinished API that is responsible for many client side problems. If there is no ESI, it would be much easier to restore client side code sanity. I agree that an inquiry on squid-users and a separate email to ESI sponsors is a good step forward _if_ there is no squid-dev consensus that ESI must stay as an integrated component. Alex.
Re: Should we remove ESI?
On Tue, Jun 11, 2013 at 5:00 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/11/2013 02:23 AM, Kinkie wrote: On Mon, Jun 10, 2013 at 7:22 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 06/09/2013 02:40 PM, Kinkie wrote: while attempting to increase portability to recent clang releases, I noticed that libTrie hasn't benefited from the portability work that was done in the past few years. I can see three ways to move forward: 1- replicate these changes into libTrie 2- change libTrie to piggyback squid's configuration variables 3- fully integrate libTrie into squid's build system. Unless Robert knows otherwise, squid is the only user of this library.. I cannot tell what libTrie does: The README file is empty and the commit message only implies that it is an ESI component. AFAICT, only ESI uses it today. From what I understand (Robert, can you come to the rescue?) libTrie is a very optimized key- and prefix- lookup engine, trading memory useage for speed. It would be great to use in the Http parser to look up header keys, for instance. And there are other similar tools around -- we use one in Web Polygraph IIRC. It would be a good project to compare them and use the best for HTTP header lookup if (and only if) they show noticeable improvement. Then, depending on the winning tool, the best integration option can be found. For this reason I would launch an inquiry to our users and to the original ESI sponsors to understand whether to simply stop supporting ESI. It is ~10kLOC that noone really looks after, and they imply dependencies (e.g. on the xml libraries). Yes, but even bigger ESI-associated problems are, IMO: * ESI adaptations are not appropriate for a built-in Squid component. The content adaptation that ESI performs should be done using an optional module or service. If eCAP has (or can be enhanced to have) enough hooks, eCAP should be used for that. Otherwise, another API can be provided for all similar adaptations. * ESI may be the only big user of Client Streams, an unfinished API that is responsible for many client side problems. If there is no ESI, it would be much easier to restore client side code sanity. I agree that an inquiry on squid-users and a separate email to ESI sponsors is a good step forward _if_ there is no squid-dev consensus that ESI must stay as an integrated component. The consensus that ESI should ideally be turned into an eCAP module is overwhelming. As is the consensus that so far noone has the time to do it :( What is the second-best option? -- /kinkie
Re: Should we remove ESI?
On 06/09/2013 02:40 PM, Kinkie wrote: while attempting to increase portability to recent clang releases, I noticed that libTrie hasn't benefited from the portability work that was done in the past few years. I can see three ways to move forward: 1- replicate these changes into libTrie 2- change libTrie to piggyback squid's configuration variables 3- fully integrate libTrie into squid's build system. Unless Robert knows otherwise, squid is the only user of this library.. I cannot tell what libTrie does: The README file is empty and the commit message only implies that it is an ESI component. AFAICT, only ESI uses it today. I do not know much about ESI, but IMHO, if somebody has cycles to work on this, it would be best to spend them removing ESI (together with libtTrie) from Squid sources while converting ESI into an eCAP adapter. This will be a big step forward towards making client side code sane (but removing ESI itself does not require making complex changes to the client side code itself). I am willing to assist with the eCAP APIs necessary for the conversion, but I cannot volunteer to work on ESI removal or the ESI adapter, unfortunately. Perhaps somebody who uses ESI can help? $0.02, Alex.
