Re: eCAP: expose Squid or link with eCAP lib?
On Sat, 2008-02-23 at 15:03 +0100, Henrik Nordström wrote: fre 2008-02-15 klockan 09:07 +1100 skrev Robert Collins: Its more work both at code and at runtime. The only thing it really allows that 1) doesn't is non-GPL eCAP modules. I don't see how 2) can allow non-GPL eCAP modules. We can't add a linking excemption to the license even if there is a well defined API. The only way to link non-GPL code to Squid is by not linking it into the runtime binary, statically or dynamically. non-GPL code needs to run in it's own process image, and for that we have ICAP already. Split out runtime linking and compiling. Runtime linking is done on the users machine, no distribution is taking place and thus the GPL is not relevant. Compiling uses header information (and on some platforms information about the library (e.g. dll symbol name mappings) to cause the generated binary to be one that will link at runtime correctly. This can (but doesn't always) cause a copy of information to be placed into the binary that was compiled. Distributing that binary requires a copying licence for the copied information - which is where the GPL kicks in. Now, when you have a well defined interface (e.g. readline), its entirely possible to have *two* implementations that are binary compatible. Implementation one: GPL. Feature complete. Implementation two: BSD. Feature complete from an interface perspective (you can compile anything you can compile with the GPL implementation), but its runtime support is shockingly bad. And the loophole should now be obvious: - vendor A writes against the GPL version in private, to test. - they then create a binary module for platform X by compiling against the BSD library. - the binary module runs against the GPL version. -Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. signature.asc Description: This is a digitally signed message part
Re: eCAP: expose Squid or link with eCAP lib?
tor 2008-02-14 klockan 09:09 -0700 skrev Alex Rousskov: 1) Expose Squid internals: Publish/install Squid headers and libraries to give direct access to Squid resources. This approach will most likely require installing pretty much all headers because the module may need to use many Squid services (e.g., DNS lookups) and because of the dependencies between Squid headers. Don't expose much of the internals. A lot is going to need changes when we start the performance redesign of the internal core. The following will certainly change: - HTTP message representation - HTTP message referencing - Object data access The needs of eCAP is fairly limited. Focus on that and provide an interface for that only. Let it grow over time, but avoid dependencies on the internal APIs and structs unless absolutely needed. This should be published as a set of header files defining the API between an eCAP module and Squid, and possibly a small library of helper/support code of use to eCAP modules only. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: eCAP: expose Squid or link with eCAP lib?
fre 2008-02-15 klockan 09:07 +1100 skrev Robert Collins: Its more work both at code and at runtime. The only thing it really allows that 1) doesn't is non-GPL eCAP modules. I don't see how 2) can allow non-GPL eCAP modules. We can't add a linking excemption to the license even if there is a well defined API. The only way to link non-GPL code to Squid is by not linking it into the runtime binary, statically or dynamically. non-GPL code needs to run in it's own process image, and for that we have ICAP already. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: eCAP: expose Squid or link with eCAP lib?
Hi Adrian, Adrian Chadd wrote: On Sat, Feb 16, 2008, Tsantilas Christos wrote: . PS. I actually believe that it must exists a squid external utility library which will be used by both squid and modules. But OK this looks difficult right now... Its been talked about for Squid-3 for quite some time but noone seems to have properly untangled the mess. OK, but this is not a requirement, squid3 is enough good without it. And become better day by day. There is significant progress. I started working on this in my Squid-2 branch; this upset some people. :) It is not that it will upset some people... it is just that we would have better results if all developers are spending their squid development time working on the same general direction... I actually got quite far. Regards, Christos
Re: eCAP: expose Squid or link with eCAP lib?
On Sat, Feb 16, 2008, Tsantilas Christos wrote: OK, but this is not a requirement, squid3 is enough good without it. And become better day by day. There is significant progress. I started working on this in my Squid-2 branch; this upset some people. :) It is not that it will upset some people... it is just that we would have better results if all developers are spending their squid development time working on the same general direction... Hey, I'm happy if Squid development was aligned with what our diverse group of users wants. But thats a discussion going on the squid-core list; I won't pollute this list with it. :) Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: eCAP: expose Squid or link with eCAP lib?
Adrian Chadd wrote: Hey, I'm happy if Squid development was aligned with what our diverse group of users wants. But thats a discussion going on the squid-core list; OK men no problem, you can do yours discussions in squid-core list... I was just answered an email which has my mail address in the To: field ... I won't pollute this list with it. :) Regards, Christos
Re: eCAP: expose Squid or link with eCAP lib?
On Sat, 2008-02-16 at 12:20 +0200, Tsantilas Christos wrote: PS. I actually believe that it must exists a squid external utility library which will be used by both squid and modules. But OK this looks difficult right now... That is exactly what eCAP library will do, I think. Both Squid and a module will link with it, and the library will insulate one side from the other side internals. I hope to share some sketches very soon. Thanks, Alex.
Re: eCAP: expose Squid or link with eCAP lib?
Hi, Alex Rousskov wrote: 1) Expose Squid internals: Publish/install Squid headers and libraries to give direct access to Squid resources. This approach will most likely require installing pretty much all headers because the module may need to use many Squid services (e.g., DNS lookups) and because of the dependencies between Squid headers. 2) Link with an eCAP library: Implement a Squid-independent eCAP library that Squid and modules will build with to get connected to each other. This way, Squid does not have to publish any of its headers (the library does). This approach may simplify Squid header management and even allow integration with other proxies, but it is more work because it is a stand-alone library and because Squid would have to translate between internal Squid types and eCAP library types. Since the eCAP interface/library is released, it will not be easy to make changes on it any more. Since the eCAP interface/api is released, a number of projects will start using this interface (this is the goal of the eCAP ). Any change on eCAP interface will hurt these projects. This will cause problems in the main squid development. What will happen if the correction of a bug, requires an extra argument in an interface function? An other example is the http headers related code/api which already exists in squid3. All developers agree that the http parser must rewritten which maybe requires change the HttpHeaders api too The second path looks safer but the decision depends on the development time the squid developers (mainly Alex) can give. Also maybe has a performance penalty. Regards, Christos
Re: eCAP: expose Squid or link with eCAP lib?
Tsantilas Christos wrote: Hi, Alex Rousskov wrote: 1) Expose Squid internals: Publish/install Squid headers and libraries to give direct access to Squid resources. This approach will most likely require installing pretty much all headers because the module may need to use many Squid services (e.g., DNS lookups) and because of the dependencies between Squid headers. 2) Link with an eCAP library: Implement a Squid-independent eCAP library that Squid and modules will build with to get connected to each other. This way, Squid does not have to publish any of its headers (the library does). This approach may simplify Squid header management and even allow integration with other proxies, but it is more work because it is a stand-alone library and because Squid would have to translate between internal Squid types and eCAP library types. Since the eCAP interface/library is released, it will not be easy to make changes on it any more. Since the eCAP interface/api is released, a number of projects will start using this interface (this is the goal of the eCAP ). Any change on eCAP interface will hurt these projects. This will cause problems in the main squid development. What will happen if the correction of a bug, requires an extra argument in an interface function? This is not such a large problem as it seems. A published library has a version-number assigned to it is libsquidecap1.1 (for 1st release of squid ecap library) Major changes such as entirely cutting or adding features to the API require a new version. Minor changes such as parameters in a bug fix, only require that the old function remains for its users with some bug-compat wrapper to the new safe call (deprecated and often marked to be removed in next big release). C++ handles overloading with different parameters natively so that is a non-issue in most cases. An other example is the http headers related code/api which already exists in squid3. All developers agree that the http parser must rewritten which maybe requires change the HttpHeaders api too This is very close to being a blocker for eCAP. Unless the eCAP developer is willing to re-write v2 ecap after the Http* classes change. The second path looks safer but the decision depends on the development time the squid developers (mainly Alex) can give. Also maybe has a performance penalty. Wrapper calls most probably do have some penalties. With increasing delays on back-compat wrappers-on-wrappers as time (and changes) go by if they are not culled out. BUT, those penalties are born by the clients and only badly effect those who do not to maintain their programs well enough to move to the new API calls. Amos -- Please use Squid 2.6STABLE17+ or 3.0STABLE1+ There are serious security advisories out on all earlier releases.
Re: eCAP: expose Squid or link with eCAP lib?
On Fri, 2008-02-15 at 09:07 +1100, Robert Collins wrote: On Thu, 2008-02-14 at 09:09 -0700, Alex Rousskov wrote: An eCAP loadable module needs to plug into Squid message processing pipeline, similar to how ICAP servers do that now, but without ICAP/TCP overheads. I see two design choices for giving the module efficient access to Squid: 1) Expose Squid internals: Publish/install Squid headers and libraries to give direct access to Squid resources. This approach will most likely require installing pretty much all headers because the module may need to use many Squid services (e.g., DNS lookups) and because of the dependencies between Squid headers. I prefer this technically because it means that work done to make eCAP clean will naturally clean up squid too. I agree that a more thorough cleanup would be required to install headers. On the other hand, most of that cleanup effort is unrelated to Squid code quality as such (see the squid/src/squid/ issue for an example). The effort would be directed at satisfying external requirements placed on installable libraries and not on Squid code quality. In other words, this may add work that does not really benefit Squid directly. The amount and complexity of that extra work may even derail the cleanup! With option (2), that should not happen. 2) Link with an eCAP library: Implement a Squid-independent eCAP library that Squid and modules will build with to get connected to each other. This way, Squid does not have to publish any of its headers (the library does). This approach may simplify Squid header management and even allow integration with other proxies, but it is more work because it is a stand-alone library and because Squid would have to translate between internal Squid types and eCAP library types. Its more work both at code and at runtime. The only thing it really allows that 1) doesn't is non-GPL eCAP modules. Whether (1) requires GPL is debatable (and we should not debate that legal question here). (2) Is a cleaner solution from code quality point and module management point of view because you are forced to carefully encapsulate eCAP module capabilities; access to Squid internals is performed via well-known channels. With (1), anything within Squid is up for grabs as there are no API restrictions. Thank you, Alex.
Re: eCAP: expose Squid or link with eCAP lib?
Hello, The Squid eCAP branch already has limited support for loadable modules. I have also written some eCAP-specific code, but I am not satisfied with its design yet. This may be the last opportunity to change things without rewriting a lot of code so I would love to get your feedback and ideas. An eCAP loadable module needs to plug into Squid message processing pipeline, similar to how ICAP servers do that now, but without ICAP/TCP overheads. I see two design choices for giving the module efficient access to Squid: 1) Expose Squid internals: Publish/install Squid headers and libraries to give direct access to Squid resources. This approach will most likely require installing pretty much all headers because the module may need to use many Squid services (e.g., DNS lookups) and because of the dependencies between Squid headers. 2) Link with an eCAP library: Implement a Squid-independent eCAP library that Squid and modules will build with to get connected to each other. This way, Squid does not have to publish any of its headers (the library does). This approach may simplify Squid header management and even allow integration with other proxies, but it is more work because it is a stand-alone library and because Squid would have to translate between internal Squid types and eCAP library types. I like the straightforwardness of (1) but it is crude and messy. I like the elegance of (2) but it is more work. Currently, I favor (2) but I want to hear what others think. Which path would you choose? I'd choose (3). Some of the basic types have clean headers already. Is it possible to incrementally grow eCAP access to the internals of squid? Starting with, ie just a buffer of page text, and work the API up? I mean, we publish the available cleaned squid .h for its use starting at lowest levels needed to support eCAP and working the way up to full control paths. So the eCAP library never defines its own seperate types publicly, but uses the squid ones as they get cleaned up. I imagine the basic first-signup users of eCAP will be wanting to perform content-filters on pages as they arrive. That only needs a read-only buffer of the page data and a hook to allow/deny the transaction. Amos Does anybody have better ideas on how to support eCAP modules that do ICAP-like adaptations inside Squid? Thank you, Alex.
Re: eCAP: expose Squid or link with eCAP lib?
On Fri, 2008-02-15 at 12:15 +1300, Amos Jeffries wrote: I imagine the basic first-signup users of eCAP will be wanting to perform content-filters on pages as they arrive. That only needs a read-only buffer of the page data and a hook to allow/deny the transaction. FWIW, some of the users waiting for eCAP have full-functioning Traffic Spicer modules that do a lot more than allow/deny based on read-only page buffers. I agree that it is possible to gradually cleanup and publish polished Squid headers, but the first publication would still require a major cleanup to remove the current inter-header dependencies! And when you are done with that cleanup, you might as well expose pretty much everything :-). To do gradual exposure without the major initial cleanup, you need that eCAP library that provides its own clean types and hides Squid guts from the modules. That's pretty much the same as option #2, as nobody is going to start with a full-blown eCAP library anyway. I think the core question here is whether it is a good idea to expose [polished, cleaned, improved, etc.] Squid guts? Is it better to provide a well-defined eCAP-focused interface instead? I think an eCAP interface is a better idea even if we have polished guts. Besides other speculative code quality reasons already discussed here, I add this one: Guts change. Squid3 guts will change a lot. eCAP modules should not stop Squid from changing, and should not suffer too much when Squid guts change. I should have thought of that reason earlier, but that's why it helps to discuss these things :-). Thank you, Alex.
Re: eCAP: expose Squid or link with eCAP lib?
On Fri, 2008-02-15 at 12:15 +1300, Amos Jeffries wrote: I imagine the basic first-signup users of eCAP will be wanting to perform content-filters on pages as they arrive. That only needs a read-only buffer of the page data and a hook to allow/deny the transaction. FWIW, some of the users waiting for eCAP have full-functioning Traffic Spicer modules that do a lot more than allow/deny based on read-only page buffers. I agree that it is possible to gradually cleanup and publish polished Squid headers, but the first publication would still require a major cleanup to remove the current inter-header dependencies! And when you are done with that cleanup, you might as well expose pretty much everything :-). To do gradual exposure without the major initial cleanup, you need that eCAP library that provides its own clean types and hides Squid guts from the modules. That's pretty much the same as option #2, as nobody is going to start with a full-blown eCAP library anyway. I think the core question here is whether it is a good idea to expose [polished, cleaned, improved, etc.] Squid guts? Is it better to provide a well-defined eCAP-focused interface instead? I think an eCAP interface is a better idea even if we have polished guts. Besides other speculative code quality reasons already discussed here, I add this one: Guts change. Squid3 guts will change a lot. eCAP modules should not stop Squid from changing, and should not suffer too much when Squid guts change. I should have thought of that reason earlier, but that's why it helps to discuss these things :-). True on both counts. So what shall this interface look like? I was imagining an interface that pulled/set some squid data structures for client use. Even if it was gated through an API layer. Amos
Re: eCAP: expose Squid or link with eCAP lib?
On Fri, Feb 15, 2008, Amos Jeffries wrote: I'd choose (3). Some of the basic types have clean headers already. Is it possible to incrementally grow eCAP access to the internals of squid? Starting with, ie just a buffer of page text, and work the API up? I mean, we publish the available cleaned squid .h for its use starting at lowest levels needed to support eCAP and working the way up to full control paths. So the eCAP library never defines its own seperate types publicly, but uses the squid ones as they get cleaned up. I imagine the basic first-signup users of eCAP will be wanting to perform content-filters on pages as they arrive. That only needs a read-only buffer of the page data and a hook to allow/deny the transaction. If you don't tidy up stuff early on then you end up introducing more dependencies which will make restructuring more of a pain. Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
