Re: Ideas for an output filter for mod_lua
On 08/23/2012 11:32 PM, Tim Bannister wrote: On 23 Aug 2012, at 11:45, Daniel Gruno rum...@cord.dk wrote: On 08/23/2012 12:02 AM, Tim Bannister wrote: My patch is for implementing gzip compression by httpd, not decompression, but the code will look pretty similar. That's quite neat, then. I will try to make an actual implementation in Lua. The part I found difficult was the interaction with the second transfer-encoding, “chunked”. Using gzip Transfer-Encoding: implies using chunked, because we want to shorten the response and this means that the Content-Length definitely doesn't match the size of the HTTP response body. I took a shot at an actual compression filter in Lua today, that uses zlib to deflate buckets and sends it as chunked data to the client. The script I used can be found at http://apaste.info/OPHa and I've verified that it works well for compressing text (I compressed a 50kb html output from a Lua script down to a 5kb chunked message with 4 separate chunks due to calling r:flush() in the script) The documentation for these new filters can be temporarily found at http://www.humbedooh.com/mod_lua.html.en#modifying_buckets while I polish it off for final committing (I have a bad habit of committing stuff, then making 10 new revisions later on ;) ). I just need to finish off some final adjustments, and I'll be committing the Lua filter support to the trunk for review and nagging. With regards, Daniel.
Re: Ideas for an output filter for mod_lua
On 08/23/2012 12:02 AM, Tim Bannister wrote: On 22 Aug 2012, at 22:25, Daniel Gruno rum...@cord.dk wrote: Would your concept meaningfully generalise beyond application-level filters? I'm not entirely sure what you mean by this, could you elaborate? If you want some more sophisticated examples of what could be achieved with Lua filtering, I'd be happy to provide some more details on how this concept could be utilised. I don't know if this is another way of phrasing Nick's question or not, but would I be able to implement gzip Transfer-Encoding: just using Lua and this new directive? I found (bug 52860) it a bit tricky to achieve in C, so I think it could be harder still with the extra limitations of the Lua environment. My C code uses AP_FTYPE_TRANSCODE which I think is the right choice but few modules get involved at this filtering stage. The filter phases are as follows (as with any other filter mod): 1) mod_lua calls a setup function before buckets are sent, to allow for headers to be changed and data to be inserted before the actual content, fx. appending an advertisment a'la old Geocities or some such. 2) The Lua function yields to let mod_lua know it's ready to receive buckets 3) Buckets are sent, one by one to the function, which processes them, and again yields with the new content for each bucket. 4) once all buckets have passed through, mod_lua makes a final call with an empty (nil) buffer, allowing the Lua script to add any additional tail calls/data to the filter (it adds tail data by yielding with whatever should be added). Thus, an example Lua function could be: --- function filter(r) -- Initial phase r.content_type = text/plain -- set up some stuff coroutine.yield() -- End initial phase -- bucket phase while (buffer) do -- for each bucket sent, do... local output = mangle(buffer) -- transformation coroutine.yield(output) -- Send the output down the chain end -- Final phase once all buckets have been sent clean_up() coroutine.yield(This line will be appended to all files) end --- So yes, theoretically you should be able to implement decompression this way, by doing something along the lines of this (totally just making it up): - local zip = require zlib -- or something... function gzip_handle(r) r.headers_out['Transfer-Encoding'] = gzip -- or ? do_magic_header_stuff_here() -- add header data coroutine.yield() -- yield and wait for buckets while (buffer) do -- for each bucket, deflate it local deflated = zip.deflate(buffer) coroutine.yield(deflated) -- pass on new data end append_tail_to_output() -- pass on a tail if needed end - I haven't tried this out with gzip, but I have tried with base64_encoding a lot of files through a test filter that works similar to what I wrote above. I'm using AP_FTYPE_RESOURCE for the testing, but surely we could either change that or add some additional optional argument to the directive to control where it should place itself. As a 'fun' side effect of doing this in Lua, you can use the same filter functions for both input and output filtering, as they work the same way, although I don't know of any filters that would benefit from it ;) In any case, I'll get started on adding an input filter hook as well, and see if I can get it working just as well as the output filter I have in place, and then commit something snazzy for review. With regards, Daniel.
Re: Ideas for an output filter for mod_lua
On 23 Aug 2012, at 11:45, Daniel Gruno rum...@cord.dk wrote: On 08/23/2012 12:02 AM, Tim Bannister wrote: I don't know if this is another way of phrasing Nick's question or not, but would I be able to implement gzip Transfer-Encoding: just using Lua and this new directive? I found (bug 52860) it a bit tricky to achieve in C, so I think it could be harder still with the extra limitations of the Lua environment. My C code uses AP_FTYPE_TRANSCODE which I think is the right choice but few modules get involved at this filtering stage. … So yes, theoretically you should be able to implement decompression this way, by doing something along the lines of this (totally just making it up): - local zip = require zlib -- or something... function gzip_handle(r) r.headers_out['Transfer-Encoding'] = gzip -- or ? do_magic_header_stuff_here() -- add header data coroutine.yield() -- yield and wait for buckets while (buffer) do -- for each bucket, deflate it local deflated = zip.deflate(buffer) coroutine.yield(deflated) -- pass on new data end append_tail_to_output() -- pass on a tail if needed end - My patch is for implementing gzip compression by httpd, not decompression, but the code will look pretty similar. That's quite neat, then. I will try to make an actual implementation in Lua. The part I found difficult was the interaction with the second transfer-encoding, “chunked”. Using gzip Transfer-Encoding: implies using chunked, because we want to shorten the response and this means that the Content-Length definitely doesn't match the size of the HTTP response body. -- Tim Bannister – is...@jellybaby.net smime.p7s Description: S/MIME cryptographic signature
Re: Ideas for an output filter for mod_lua
On Thu, 23 Aug 2012 22:32:20 +0100 Tim Bannister is...@jellybaby.net wrote: That's quite neat, then. I will try to make an actual implementation in Lua. The part I found difficult was the interaction with the second transfer-encoding, “chunked”. Using gzip Transfer-Encoding: implies using chunked, because we want to shorten the response and this means that the Content-Length definitely doesn't match the size of the HTTP response body. gzip is a content-encoding, not a transfer-encoding. That makes it easy to implement: content filters and protocol filters work at different levels, so can coexist without trouble. -- Nick Kew
Re: Ideas for an output filter for mod_lua
On Aug 23, 2012, at 2:49 PM, Nick Kew wrote: On Thu, 23 Aug 2012 22:32:20 +0100 Tim Bannister is...@jellybaby.net wrote: That's quite neat, then. I will try to make an actual implementation in Lua. The part I found difficult was the interaction with the second transfer-encoding, “chunked”. Using gzip Transfer-Encoding: implies using chunked, because we want to shorten the response and this means that the Content-Length definitely doesn't match the size of the HTTP response body. gzip is a content-encoding, not a transfer-encoding. It is both. Transfer-Encoding is the correct way to implement it. That makes it easy to implement: content filters and protocol filters work at different levels, so can coexist without trouble. Except that they aren't standards-compliant because they don't handle etags correctly. Content-Encoding works with more clients but is much harder to implement correctly. Roy
Re: Ideas for an output filter for mod_lua
On 22 Aug 2012, at 12:00, Daniel Gruno wrote: So, any feedback, comments, thoughts on this? Basic concept looks fine. I guess we'd need more detail to say any more about it. Is the implementation 'clean' or does it involve hacks to core? If what you have is pure module then I'd see no reason not to drop mod_lua_filter (or is it mod_filter_lua?) into trunk on CTR. Meanwhile, some random thoughts: Is there a deep reason to limit it to output filters, or is that just what you happen to have implemented? Would your concept meaningfully generalise beyond application-level filters? -- Nick Kew
Re: Ideas for an output filter for mod_lua
On 08/22/2012 01:36 PM, Nick Kew wrote: Basic concept looks fine. I guess we'd need more detail to say any more about it. Is the implementation 'clean' or does it involve hacks to core? If what you have is pure module then I'd see no reason not to drop mod_lua_filter (or is it mod_filter_lua?) into trunk on CTR. I don't see why it should be made a separate module - it relies and operates on the stuff already inside mod_lua, and making it a part of it only requires adding some 50-60 lines of code (ymmv). But ideas/opinions are of course welcome. As for the 'cleanliness' of the code, there are no hacks, it works similar to how the other hook functions in mod_lua work. Each call to a Lua(In|Out)putFilter would register a new filter with a callback to fx ap_lua_output_filter, which will then traverse a list of known scripts/handles, find the one associated with the filter name in question and run it, making the Lua script yield whenever it's parsed a bucket and sending it further down the chain. Apart from the actual bucket handling, the function and the directives behave just as the other hooks and the LuaMapHandler does. When called, it would register a file/function in an array, and when a call is made to that handler, it checks to see what is being called, and finds the file/function associated with it. i.e. if you had two directives: LuaOutputFilter filterA /www/filters.lua filter_a_func LuaOutputFilter filterB /www/filters.lua filter_b_func then an output, with any of those two filters attached, would call ap_lua_output_filter, which would look up the filter currently being run and then run the file/function mapped to that filter. I guess what separates this from the other hooks the most is that it relies on co-routines (aka non-blocking/event-based or whatever the newest buzz word for it is) to get the job done. Meanwhile, some random thoughts: Is there a deep reason to limit it to output filters, or is that just what you happen to have implemented? The same method could be applied to input filters as well, I just had an interest in trying it out for output filters, so that's what I hacked together before making this proposal. I'll try implementing an input filter tomorrow. Would your concept meaningfully generalise beyond application-level filters? I'm not entirely sure what you mean by this, could you elaborate? If you want some more sophisticated examples of what could be achieved with Lua filtering, I'd be happy to provide some more details on how this concept could be utilised. With regards, Daniel.
Re: Ideas for an output filter for mod_lua
On 22 Aug 2012, at 22:25, Daniel Gruno rum...@cord.dk wrote: Would your concept meaningfully generalise beyond application-level filters? I'm not entirely sure what you mean by this, could you elaborate? If you want some more sophisticated examples of what could be achieved with Lua filtering, I'd be happy to provide some more details on how this concept could be utilised. I don't know if this is another way of phrasing Nick's question or not, but would I be able to implement gzip Transfer-Encoding: just using Lua and this new directive? I found (bug 52860) it a bit tricky to achieve in C, so I think it could be harder still with the extra limitations of the Lua environment. My C code uses AP_FTYPE_TRANSCODE which I think is the right choice but few modules get involved at this filtering stage. -- Tim Bannister – is...@jellybaby.net smime.p7s Description: S/MIME cryptographic signature
