Bringing this thread back to the mailing list so the useful info below can be archived :)
On 17/08/2012, at 6:36 PM, Shu Kit Chan <[email protected]> wrote: > Andrew, > > Sorry. I accidentally press the "Reply" instead of "Reply All" last time > > I took a longer look at your original email and I tried to simulate in > my own setup. I think there is at least two thing missing definitely. > > #1) In remap.config, you need to have > > map / http://dev1.test/~ho/proxied > map http://dev1.test/~ho/proxied http://dev1.test/~ho/proxied > > It's a bit counter-intuitive but that's just how it works for making > internal call with ATS (i.e. the use of TSFetchUrl). Yeh that seems a bit redundant to me too. > > #2) Unfortunately, not everything in esi spec works fine in this > plugin. So a bit of documentation is needed. Specifically, "onerror" > is not working fine. So with the problem in #1, the YTS do not know > where to get your snippet and don't care about "onerror". Thus it just > error out with an empty response for you (not a good way to end, i > might add). > > If you try the following, it should work > > <html> > <head> > <title>ESI test</title> > </head> > <body> > <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p> > <esi:try><esi:attempt><esi:include > src="http://dev1.test/~ho/snippet.cgi"/></esi:attempt><esi:except>Sorry!</esi:except></esi:try> > </body> > </html> > > Anyway, let me know how it works for u. I am still getting my upstream > merged fork to compile. > If you check the email archieve, you will see I help out a guy to get > ESI working for him a couple months ago. Not too sure if he finally > uses ESI in production or not. Yahoo! is definitely using ESI in their > production. The code that ATS has is originally donated from yahoo! > But since then Y! did make a few fixes and the Y! people are also > aware of the fact that some parts of the spec are not working. And > they have their own documentation of what part of the spec to use > versus what part you should not use. Hope this info helps. I just hooked up the ESI unit tests. There's a bunch that fail and I'd be happy to merge patches to fix any of these. Hopefull this will make it clearer what works and what doesn't. > > Thanks. > > Kit > > On Fri, Aug 17, 2012 at 5:36 PM, James Peach <[email protected]> wrote: >> Add Andrew to CC ... >> >> On Aug 17, 2012, at 5:34 PM, Shu Kit Chan <[email protected]> wrote: >> >>> i am spending some times to fetch from upstream and trying things out. >>> Will report back later. >>> >>> Meanwhile... >>> >>> Can you show the squid.log of ats during your run? >>> Also, I think if you set >>> CONFIG proxy.config.diags.debug.tags STRING plugin_esi* >>> >>> There will be more meaningful debug messages in traffic.out for us as well. >>> >>> Thanks. >>> >>> Kit >>> >>> On Fri, Aug 17, 2012 at 5:10 PM, James Peach <[email protected]> wrote: >>>> On Aug 17, 2012, at 4:50 PM, Andrew Ho <[email protected]> wrote: >>>> >>>>> Hi James, >>>>> >>>>> Thanks for the offer for help! In this case, though, I think it's Kit who >>>>> needs to update his fork to merge in the upstream changes (specifically, >>>>> your commit 053cc43e which added the build bits for >>>>> plugin/experimental/esi). And that's just so his fork will build from a >>>>> fresh checkout—the problem that I'm seeing hasn't been diagnosed yet. >>>> >>>> There's a bit of both ... I've merged some of Kit's changes but not all. >>>> It's possible that the result is not working. I'm disappointed to hear >>>> that Kit's fork isn't working for you. I'll see whether I can dig into it >>>> a bit tonight ... >>>> >>>>> >>>>> As for the deprecated warnings, I'll be happy to dig into that and >>>>> propose a patch after I get basic ESI working. :) >>>>> >>>>> Humbly, >>>>> >>>>> Andrew >>>>> >>>>> -- >>>>> Andrew Ho >>>>> Senior Software Developer >>>>> [email protected] >>>>> >>>>> >>>>> >>>>> On Fri, Aug 17, 2012 at 4:18 PM, James Peach <[email protected]> wrote: >>>>> >>>>> On Aug 17, 2012, at 4:14 PM, Andrew Ho <[email protected]> wrote: >>>>> >>>>>> Hi Kit, >>>>>> >>>>>> Heh, I just stole the one that was checked in to the upstream repository >>>>>> earlier this week: >>>>>> >>>>>> wget -Oplugins/experimental/esi/Makefile.am >>>>>> https://raw.github.com/apache/trafficserver/master/plugins/experimental/esi/Makefile.am >>>>>> perl -pi.orig -e 's,experimental/lua,experimental/esi,' configure.ac >>>>>> autoreconf -i >>>>>> CFLAGS='-fPIC -Wno-deprecated' CXXFLAGS='-fPIC -Wno-deprecated' >>>>>> ./configure >>>>>> (cd plugins/experimental/esi && make) >>>>>> >>>>>> (The silly bit with Perl is because the default configure.ac in your >>>>>> fork had also not yet been updated.) Note that I had to add >>>>>> -Wno-deprecated because otherwise the -Wall resulted in some deprecation >>>>>> warnings. >>>>>> >>>>>> I'll try this out and also give your Makefile a try (but you might want >>>>>> to merge the upstream changes to the build system, it's nice that >>>>>> there's a Makefile.am for esi now). >>>>> >>>>> Hey, if you guys can pop up on IRC I should be able to turn the merges >>>>> around pretty quickly and we can verify what is working ... I'll be on >>>>> later tonight after 8pm Pacific >>>>> >>>>> >>>>>> >>>>>> Humbly, >>>>>> >>>>>> Andrew >>>>>> >>>>>> -- >>>>>> Andrew Ho >>>>>> Senior Software Developer >>>>>> [email protected] >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Aug 17, 2012 at 3:49 PM, Shu Kit Chan <[email protected]> >>>>>> wrote: >>>>>> Just added the Makefile I use. >>>>>> You can try it out. It is nothing fancy. A pretty simple one. >>>>>> >>>>>> On Fri, Aug 17, 2012 at 3:09 PM, Andrew Ho <[email protected]> wrote: >>>>>>> Hi Kit, >>>>>>> >>>>>>> Thanks for your quick response. I had come across your blog entries and >>>>>>> previous mailing list posts and they were quite useful for me so far. >>>>>>> >>>>>>> I noticed that there is neither a Makefile nor a Makefile.am in your >>>>>>> repository. How do you usually build the plugin from source? >>>>>>> >>>>>>> >>>>>>> Humbly, >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> -- >>>>>>> Andrew Ho >>>>>>> Senior Software Developer >>>>>>> [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Aug 17, 2012 at 1:31 PM, Shu Kit Chan <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> The current latest on git is still "experimental". >>>>>>>> I have made a bunch of fixes which James Peach is helping me to verify. >>>>>>>> The conversation can be found here - >>>>>>>> https://issues.apache.org/jira/browse/TS-1249 >>>>>>>> >>>>>>>> If you really want to try, you can try to compile the forked version I >>>>>>>> have. >>>>>>>> >>>>>>>> https://github.com/shukitchan/trafficserver/tree/master/plugins/experimental/esi >>>>>>>> and i hope it should work for you. >>>>>>>> >>>>>>>> It is at least working quite well for me. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Kit >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Aug 17, 2012 at 12:42 PM, Andrew Ho <[email protected]> wrote: >>>>>>>>> Hi Traffic Server users, >>>>>>>>> >>>>>>>>> I can't seem to get ESIs working on Traffic Server. When I request an >>>>>>>>> ESI >>>>>>>>> enabled page, I get a 200 OK status, but "Content-Length: 0" and an >>>>>>>>> empty >>>>>>>>> entity body. There is no evidence on the origin server side that the >>>>>>>>> ESI >>>>>>>>> sub-pages were ever loaded. Does this symptom sound familiar to >>>>>>>>> anybody? >>>>>>>>> >>>>>>>>> The gory details follow... >>>>>>>>> >>>>>>>>> I am using Traffic Server version 3.2.0 (Linux distribution is CentOS >>>>>>>>> 5.4, >>>>>>>>> and the test hosts are in Amazon EC2): >>>>>>>>> >>>>>>>>> % traffic_server --version >>>>>>>>> [TrafficServer] using root directory '/usr/local' >>>>>>>>> Apache Traffic Server - traffic_server - 3.2.0 - (build # 7719 on Aug >>>>>>>>> 7 >>>>>>>>> 2012 at 19:06:21) >>>>>>>>> >>>>>>>>> The ESI plugin I built was from commit 025cd279 of repository version: >>>>>>>>> https://git-wip-us.apache.org/repos/asf/trafficserver.git/ >>>>>>>>> >>>>>>>>> The following are the configuration changes I enacted to the defaults. >>>>>>>>> In >>>>>>>>> plugin.config, I added this line to enable the ESI plugin: >>>>>>>>> >>>>>>>>> esi.so >>>>>>>>> >>>>>>>>> In records.config, I added these lines to enable debug logging: >>>>>>>>> >>>>>>>>> CONFIG proxy.config.diags.debug.enabled INT 1 >>>>>>>>> CONFIG proxy.config.diags.debug.tags STRING plugin_esi_intercept* >>>>>>>>> >>>>>>>>> In remap.config, I added this line to add an origin server: >>>>>>>>> >>>>>>>>> map / http://dev1.test/~ho/proxied >>>>>>>>> >>>>>>>>> dev1.test is an internal development host which runs a vanilla Apache >>>>>>>>> HTTPD >>>>>>>>> configured to serve up content from ~/public_html directories. In >>>>>>>>> dev1.test:~ho/public_html/proxied, I have a couple CGI scripts >>>>>>>>> (snippet.cgi >>>>>>>>> emits a bare HTML fragment, esi.cgi emits an "X-Esi: 1" header and an >>>>>>>>> HTML >>>>>>>>> document that includes snippet.cgi with a fully-qualified URL): >>>>>>>>> >>>>>>>>> % cat ~ho/public_html/proxied/snippet.cgi >>>>>>>>> #!/usr/local/bin/perl >>>>>>>>> >>>>>>>>> my $body = <<'EndHTML'; >>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p> >>>>>>>>> EndHTML >>>>>>>>> >>>>>>>>> print "Content-Type: text/html\n", >>>>>>>>> "Content-Length: ", length($body), "\n", >>>>>>>>> "\n", >>>>>>>>> $body; >>>>>>>>> >>>>>>>>> % cat ~ho/public_html/proxied/esi.cgi >>>>>>>>> #!/usr/local/bin/perl >>>>>>>>> >>>>>>>>> my $body = <<'EndHTML'; >>>>>>>>> <html> >>>>>>>>> <head> >>>>>>>>> <title>ESI test</title> >>>>>>>>> </head> >>>>>>>>> <body> >>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p> >>>>>>>>> <esi:include src="http://dev1.test/~ho/snippet.cgi"; >>>>>>>>> onerror="continue"/> >>>>>>>>> </body> >>>>>>>>> </html> >>>>>>>>> EndHTML >>>>>>>>> >>>>>>>>> print "Content-Type: text/html\n", >>>>>>>>> "Content-Length: ", length($body), "\n", >>>>>>>>> "X-Esi: 1\n", >>>>>>>>> "\n", >>>>>>>>> $body; >>>>>>>>> >>>>>>>>> From the host running Traffic server, I can definitely load those >>>>>>>>> URLs: >>>>>>>>> >>>>>>>>> % curl -si http://dev1/~ho/proxied/snippet.cgi >>>>>>>>> HTTP/1.1 200 OK >>>>>>>>> Date: Fri, 17 Aug 2012 19:24:20 GMT >>>>>>>>> Server: Apache/2.2.17 >>>>>>>>> Content-Length: 71 >>>>>>>>> Content-Type: text/html >>>>>>>>> >>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p> >>>>>>>>> >>>>>>>>> % curl -si http://dev1/~ho/proxied/esi.cgi >>>>>>>>> HTTP/1.1 200 OK >>>>>>>>> Date: Fri, 17 Aug 2012 19:24:22 GMT >>>>>>>>> Server: Apache/2.2.17 >>>>>>>>> X-Esi: 1 >>>>>>>>> Content-Length: 219 >>>>>>>>> Content-Type: text/html >>>>>>>>> >>>>>>>>> <html> >>>>>>>>> <head> >>>>>>>>> <title>ESI test</title> >>>>>>>>> </head> >>>>>>>>> <body> >>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p> >>>>>>>>> <esi:include src="http://dev1.test/~ho/snippet.cgi"; >>>>>>>>> onerror="continue"/> >>>>>>>>> </body> >>>>>>>>> </html> >>>>>>>>> >>>>>>>>> And when I issue requests on the hosts running Traffic Server, reverse >>>>>>>>> proxying is generally working, as I can load snippet.cgi: >>>>>>>>> >>>>>>>>> % curl -si http://localhost:8080/snippet.cgi >>>>>>>>> HTTP/1.1 200 OK >>>>>>>>> Date: Fri, 17 Aug 2012 19:24:55 GMT >>>>>>>>> Server: ATS/3.2.0 >>>>>>>>> Content-Length: 71 >>>>>>>>> Content-Type: text/html >>>>>>>>> Age: 0 >>>>>>>>> Connection: keep-alive >>>>>>>>> >>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p> >>>>>>>>> >>>>>>>>> However, I get a "Content-Length: 0" and an empty entity body on the >>>>>>>>> ESI >>>>>>>>> enabled request to esi.cgi (despite a 200 OK status!): >>>>>>>>> >>>>>>>>> % curl -si http://localhost:8080/esi.cgi >>>>>>>>> HTTP/1.1 200 OK >>>>>>>>> Date: Fri, 17 Aug 2012 19:25:15 GMT >>>>>>>>> Server: ATS/3.2.0 >>>>>>>>> Content-Type: text/html >>>>>>>>> Content-Length: 0 >>>>>>>>> Connection: keep-alive >>>>>>>>> >>>>>>>>> >>>>>>>>> Here is the logging information captured in traffic.out for that >>>>>>>>> request: >>>>>>>>> >>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [setupServerIntercept] Setup server intercept successfully >>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [serverIntercept] Received net accept event >>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [init] ContData initialized! >>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [serverIntercept] Received read ready event >>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [handleRead] Parsed header >>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [handleRead] Got content length as 257 >>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [serverIntercept] Received read ready event >>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [handleRead] Appending 356 bytes to body >>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [handleRead] Appending 257 bytes to body >>>>>>>>> [Aug 17 15:30:44.030] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [serverIntercept] Received read complete/eos event 104 >>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [processRequest] Wrote reply of size 761 >>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [serverIntercept] Processed request successfully >>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [serverIntercept] Completed request processing. Shutting down... >>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>> (plugin_esi_intercept) >>>>>>>>> [~ContData] Destroying continuation data >>>>>>>>> >>>>>>>>> On the server side, I only see the single request for esi.cgi (and not >>>>>>>>> for >>>>>>>>> snippet.cgi): >>>>>>>>> >>>>>>>>> 6071 dev1 x.x.x.x - - [17/Aug/2012:15:30:13 -0400] "GET >>>>>>>>> /~ho/proxied/esi.cgi >>>>>>>>> HTTP/1.1" 200 219 "-" "-" 193 354 5083 >>>>>>>>> >>>>>>>>> We are evaluating Traffic Server specifically for ESI support, so this >>>>>>>>> bug >>>>>>>>> is quite a blocker for me. I've tried a few different things based on >>>>>>>>> searching the Traffic Server mailing list in the past (for example, I >>>>>>>>> made >>>>>>>>> sure deflate was disabled on the origin server), and now I'm stuck. >>>>>>>>> Does >>>>>>>>> anybody know what is going on, or can someone suggest a next idea for >>>>>>>>> me >>>>>>>>> to >>>>>>>>> debug the problem? Thanks! >>>>>>>>> >>>>>>>>> Humbly, >>>>>>>>> >>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Andrew Ho >>>>>>>>> Senior Software Developer >>>>>>>>> [email protected] >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>
