Yes. The plugin uses TSFetchUrl to fetch for each of the ESI Includes, which only works if the src of the ESI include is accounted for in remap.config
Thanks. Kit On Fri, Aug 17, 2012 at 10:11 PM, James Peach <[email protected]> wrote: > Bringing this thread back to the mailing list so the useful info below can be > archived :) > > On 17/08/2012, at 6:36 PM, Shu Kit Chan <[email protected]> wrote: > >> Andrew, >> >> Sorry. I accidentally press the "Reply" instead of "Reply All" last time >> >> I took a longer look at your original email and I tried to simulate in >> my own setup. I think there is at least two thing missing definitely. >> >> #1) In remap.config, you need to have >> >> map / http://dev1.test/~ho/proxied >> map http://dev1.test/~ho/proxied http://dev1.test/~ho/proxied >> >> It's a bit counter-intuitive but that's just how it works for making >> internal call with ATS (i.e. the use of TSFetchUrl). > > Yeh that seems a bit redundant to me too. > >> >> #2) Unfortunately, not everything in esi spec works fine in this >> plugin. So a bit of documentation is needed. Specifically, "onerror" >> is not working fine. So with the problem in #1, the YTS do not know >> where to get your snippet and don't care about "onerror". Thus it just >> error out with an empty response for you (not a good way to end, i >> might add). >> >> If you try the following, it should work >> >> <html> >> <head> >> <title>ESI test</title> >> </head> >> <body> >> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p> >> <esi:try><esi:attempt><esi:include >> src="http://dev1.test/~ho/snippet.cgi"/></esi:attempt><esi:except>Sorry!</esi:except></esi:try> >> </body> >> </html> >> >> Anyway, let me know how it works for u. I am still getting my upstream >> merged fork to compile. >> If you check the email archieve, you will see I help out a guy to get >> ESI working for him a couple months ago. Not too sure if he finally >> uses ESI in production or not. Yahoo! is definitely using ESI in their >> production. The code that ATS has is originally donated from yahoo! >> But since then Y! did make a few fixes and the Y! people are also >> aware of the fact that some parts of the spec are not working. And >> they have their own documentation of what part of the spec to use >> versus what part you should not use. Hope this info helps. > > I just hooked up the ESI unit tests. There's a bunch that fail and I'd be > happy to merge patches to fix any of these. Hopefull this will make it > clearer what works and what doesn't. > >> >> Thanks. >> >> Kit >> >> On Fri, Aug 17, 2012 at 5:36 PM, James Peach <[email protected]> wrote: >>> Add Andrew to CC ... >>> >>> On Aug 17, 2012, at 5:34 PM, Shu Kit Chan <[email protected]> wrote: >>> >>>> i am spending some times to fetch from upstream and trying things out. >>>> Will report back later. >>>> >>>> Meanwhile... >>>> >>>> Can you show the squid.log of ats during your run? >>>> Also, I think if you set >>>> CONFIG proxy.config.diags.debug.tags STRING plugin_esi* >>>> >>>> There will be more meaningful debug messages in traffic.out for us as well. >>>> >>>> Thanks. >>>> >>>> Kit >>>> >>>> On Fri, Aug 17, 2012 at 5:10 PM, James Peach <[email protected]> wrote: >>>>> On Aug 17, 2012, at 4:50 PM, Andrew Ho <[email protected]> wrote: >>>>> >>>>>> Hi James, >>>>>> >>>>>> Thanks for the offer for help! In this case, though, I think it's Kit >>>>>> who needs to update his fork to merge in the upstream changes >>>>>> (specifically, your commit 053cc43e which added the build bits for >>>>>> plugin/experimental/esi). And that's just so his fork will build from a >>>>>> fresh checkout—the problem that I'm seeing hasn't been diagnosed yet. >>>>> >>>>> There's a bit of both ... I've merged some of Kit's changes but not all. >>>>> It's possible that the result is not working. I'm disappointed to hear >>>>> that Kit's fork isn't working for you. I'll see whether I can dig into it >>>>> a bit tonight ... >>>>> >>>>>> >>>>>> As for the deprecated warnings, I'll be happy to dig into that and >>>>>> propose a patch after I get basic ESI working. :) >>>>>> >>>>>> Humbly, >>>>>> >>>>>> Andrew >>>>>> >>>>>> -- >>>>>> Andrew Ho >>>>>> Senior Software Developer >>>>>> [email protected] >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Aug 17, 2012 at 4:18 PM, James Peach <[email protected]> wrote: >>>>>> >>>>>> On Aug 17, 2012, at 4:14 PM, Andrew Ho <[email protected]> wrote: >>>>>> >>>>>>> Hi Kit, >>>>>>> >>>>>>> Heh, I just stole the one that was checked in to the upstream >>>>>>> repository earlier this week: >>>>>>> >>>>>>> wget -Oplugins/experimental/esi/Makefile.am >>>>>>> https://raw.github.com/apache/trafficserver/master/plugins/experimental/esi/Makefile.am >>>>>>> perl -pi.orig -e 's,experimental/lua,experimental/esi,' configure.ac >>>>>>> autoreconf -i >>>>>>> CFLAGS='-fPIC -Wno-deprecated' CXXFLAGS='-fPIC -Wno-deprecated' >>>>>>> ./configure >>>>>>> (cd plugins/experimental/esi && make) >>>>>>> >>>>>>> (The silly bit with Perl is because the default configure.ac in your >>>>>>> fork had also not yet been updated.) Note that I had to add >>>>>>> -Wno-deprecated because otherwise the -Wall resulted in some >>>>>>> deprecation warnings. >>>>>>> >>>>>>> I'll try this out and also give your Makefile a try (but you might want >>>>>>> to merge the upstream changes to the build system, it's nice that >>>>>>> there's a Makefile.am for esi now). >>>>>> >>>>>> Hey, if you guys can pop up on IRC I should be able to turn the merges >>>>>> around pretty quickly and we can verify what is working ... I'll be on >>>>>> later tonight after 8pm Pacific >>>>>> >>>>>> >>>>>>> >>>>>>> Humbly, >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> -- >>>>>>> Andrew Ho >>>>>>> Senior Software Developer >>>>>>> [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Aug 17, 2012 at 3:49 PM, Shu Kit Chan <[email protected]> >>>>>>> wrote: >>>>>>> Just added the Makefile I use. >>>>>>> You can try it out. It is nothing fancy. A pretty simple one. >>>>>>> >>>>>>> On Fri, Aug 17, 2012 at 3:09 PM, Andrew Ho <[email protected]> wrote: >>>>>>>> Hi Kit, >>>>>>>> >>>>>>>> Thanks for your quick response. I had come across your blog entries and >>>>>>>> previous mailing list posts and they were quite useful for me so far. >>>>>>>> >>>>>>>> I noticed that there is neither a Makefile nor a Makefile.am in your >>>>>>>> repository. How do you usually build the plugin from source? >>>>>>>> >>>>>>>> >>>>>>>> Humbly, >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> -- >>>>>>>> Andrew Ho >>>>>>>> Senior Software Developer >>>>>>>> [email protected] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Aug 17, 2012 at 1:31 PM, Shu Kit Chan <[email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> The current latest on git is still "experimental". >>>>>>>>> I have made a bunch of fixes which James Peach is helping me to >>>>>>>>> verify. >>>>>>>>> The conversation can be found here - >>>>>>>>> https://issues.apache.org/jira/browse/TS-1249 >>>>>>>>> >>>>>>>>> If you really want to try, you can try to compile the forked version I >>>>>>>>> have. >>>>>>>>> >>>>>>>>> https://github.com/shukitchan/trafficserver/tree/master/plugins/experimental/esi >>>>>>>>> and i hope it should work for you. >>>>>>>>> >>>>>>>>> It is at least working quite well for me. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Kit >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Aug 17, 2012 at 12:42 PM, Andrew Ho <[email protected]> wrote: >>>>>>>>>> Hi Traffic Server users, >>>>>>>>>> >>>>>>>>>> I can't seem to get ESIs working on Traffic Server. When I request an >>>>>>>>>> ESI >>>>>>>>>> enabled page, I get a 200 OK status, but "Content-Length: 0" and an >>>>>>>>>> empty >>>>>>>>>> entity body. There is no evidence on the origin server side that the >>>>>>>>>> ESI >>>>>>>>>> sub-pages were ever loaded. Does this symptom sound familiar to >>>>>>>>>> anybody? >>>>>>>>>> >>>>>>>>>> The gory details follow... >>>>>>>>>> >>>>>>>>>> I am using Traffic Server version 3.2.0 (Linux distribution is CentOS >>>>>>>>>> 5.4, >>>>>>>>>> and the test hosts are in Amazon EC2): >>>>>>>>>> >>>>>>>>>> % traffic_server --version >>>>>>>>>> [TrafficServer] using root directory '/usr/local' >>>>>>>>>> Apache Traffic Server - traffic_server - 3.2.0 - (build # 7719 on >>>>>>>>>> Aug 7 >>>>>>>>>> 2012 at 19:06:21) >>>>>>>>>> >>>>>>>>>> The ESI plugin I built was from commit 025cd279 of repository >>>>>>>>>> version: >>>>>>>>>> https://git-wip-us.apache.org/repos/asf/trafficserver.git/ >>>>>>>>>> >>>>>>>>>> The following are the configuration changes I enacted to the >>>>>>>>>> defaults. >>>>>>>>>> In >>>>>>>>>> plugin.config, I added this line to enable the ESI plugin: >>>>>>>>>> >>>>>>>>>> esi.so >>>>>>>>>> >>>>>>>>>> In records.config, I added these lines to enable debug logging: >>>>>>>>>> >>>>>>>>>> CONFIG proxy.config.diags.debug.enabled INT 1 >>>>>>>>>> CONFIG proxy.config.diags.debug.tags STRING plugin_esi_intercept* >>>>>>>>>> >>>>>>>>>> In remap.config, I added this line to add an origin server: >>>>>>>>>> >>>>>>>>>> map / http://dev1.test/~ho/proxied >>>>>>>>>> >>>>>>>>>> dev1.test is an internal development host which runs a vanilla Apache >>>>>>>>>> HTTPD >>>>>>>>>> configured to serve up content from ~/public_html directories. In >>>>>>>>>> dev1.test:~ho/public_html/proxied, I have a couple CGI scripts >>>>>>>>>> (snippet.cgi >>>>>>>>>> emits a bare HTML fragment, esi.cgi emits an "X-Esi: 1" header and an >>>>>>>>>> HTML >>>>>>>>>> document that includes snippet.cgi with a fully-qualified URL): >>>>>>>>>> >>>>>>>>>> % cat ~ho/public_html/proxied/snippet.cgi >>>>>>>>>> #!/usr/local/bin/perl >>>>>>>>>> >>>>>>>>>> my $body = <<'EndHTML'; >>>>>>>>>> <p>This is >>>>>>>>>> <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p> >>>>>>>>>> EndHTML >>>>>>>>>> >>>>>>>>>> print "Content-Type: text/html\n", >>>>>>>>>> "Content-Length: ", length($body), "\n", >>>>>>>>>> "\n", >>>>>>>>>> $body; >>>>>>>>>> >>>>>>>>>> % cat ~ho/public_html/proxied/esi.cgi >>>>>>>>>> #!/usr/local/bin/perl >>>>>>>>>> >>>>>>>>>> my $body = <<'EndHTML'; >>>>>>>>>> <html> >>>>>>>>>> <head> >>>>>>>>>> <title>ESI test</title> >>>>>>>>>> </head> >>>>>>>>>> <body> >>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p> >>>>>>>>>> <esi:include src="http://dev1.test/~ho/snippet.cgi"; >>>>>>>>>> onerror="continue"/> >>>>>>>>>> </body> >>>>>>>>>> </html> >>>>>>>>>> EndHTML >>>>>>>>>> >>>>>>>>>> print "Content-Type: text/html\n", >>>>>>>>>> "Content-Length: ", length($body), "\n", >>>>>>>>>> "X-Esi: 1\n", >>>>>>>>>> "\n", >>>>>>>>>> $body; >>>>>>>>>> >>>>>>>>>> From the host running Traffic server, I can definitely load those >>>>>>>>>> URLs: >>>>>>>>>> >>>>>>>>>> % curl -si http://dev1/~ho/proxied/snippet.cgi >>>>>>>>>> HTTP/1.1 200 OK >>>>>>>>>> Date: Fri, 17 Aug 2012 19:24:20 GMT >>>>>>>>>> Server: Apache/2.2.17 >>>>>>>>>> Content-Length: 71 >>>>>>>>>> Content-Type: text/html >>>>>>>>>> >>>>>>>>>> <p>This is >>>>>>>>>> <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p> >>>>>>>>>> >>>>>>>>>> % curl -si http://dev1/~ho/proxied/esi.cgi >>>>>>>>>> HTTP/1.1 200 OK >>>>>>>>>> Date: Fri, 17 Aug 2012 19:24:22 GMT >>>>>>>>>> Server: Apache/2.2.17 >>>>>>>>>> X-Esi: 1 >>>>>>>>>> Content-Length: 219 >>>>>>>>>> Content-Type: text/html >>>>>>>>>> >>>>>>>>>> <html> >>>>>>>>>> <head> >>>>>>>>>> <title>ESI test</title> >>>>>>>>>> </head> >>>>>>>>>> <body> >>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p> >>>>>>>>>> <esi:include src="http://dev1.test/~ho/snippet.cgi"; >>>>>>>>>> onerror="continue"/> >>>>>>>>>> </body> >>>>>>>>>> </html> >>>>>>>>>> >>>>>>>>>> And when I issue requests on the hosts running Traffic Server, >>>>>>>>>> reverse >>>>>>>>>> proxying is generally working, as I can load snippet.cgi: >>>>>>>>>> >>>>>>>>>> % curl -si http://localhost:8080/snippet.cgi >>>>>>>>>> HTTP/1.1 200 OK >>>>>>>>>> Date: Fri, 17 Aug 2012 19:24:55 GMT >>>>>>>>>> Server: ATS/3.2.0 >>>>>>>>>> Content-Length: 71 >>>>>>>>>> Content-Type: text/html >>>>>>>>>> Age: 0 >>>>>>>>>> Connection: keep-alive >>>>>>>>>> >>>>>>>>>> <p>This is >>>>>>>>>> <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p> >>>>>>>>>> >>>>>>>>>> However, I get a "Content-Length: 0" and an empty entity body on the >>>>>>>>>> ESI >>>>>>>>>> enabled request to esi.cgi (despite a 200 OK status!): >>>>>>>>>> >>>>>>>>>> % curl -si http://localhost:8080/esi.cgi >>>>>>>>>> HTTP/1.1 200 OK >>>>>>>>>> Date: Fri, 17 Aug 2012 19:25:15 GMT >>>>>>>>>> Server: ATS/3.2.0 >>>>>>>>>> Content-Type: text/html >>>>>>>>>> Content-Length: 0 >>>>>>>>>> Connection: keep-alive >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Here is the logging information captured in traffic.out for that >>>>>>>>>> request: >>>>>>>>>> >>>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [setupServerIntercept] Setup server intercept successfully >>>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [serverIntercept] Received net accept event >>>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [init] ContData initialized! >>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [serverIntercept] Received read ready event >>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [handleRead] Parsed header >>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [handleRead] Got content length as 257 >>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [serverIntercept] Received read ready event >>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [handleRead] Appending 356 bytes to body >>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [handleRead] Appending 257 bytes to body >>>>>>>>>> [Aug 17 15:30:44.030] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [serverIntercept] Received read complete/eos event 104 >>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [processRequest] Wrote reply of size 761 >>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [serverIntercept] Processed request successfully >>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [serverIntercept] Completed request processing. Shutting down... >>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG: >>>>>>>>>> (plugin_esi_intercept) >>>>>>>>>> [~ContData] Destroying continuation data >>>>>>>>>> >>>>>>>>>> On the server side, I only see the single request for esi.cgi (and >>>>>>>>>> not >>>>>>>>>> for >>>>>>>>>> snippet.cgi): >>>>>>>>>> >>>>>>>>>> 6071 dev1 x.x.x.x - - [17/Aug/2012:15:30:13 -0400] "GET >>>>>>>>>> /~ho/proxied/esi.cgi >>>>>>>>>> HTTP/1.1" 200 219 "-" "-" 193 354 5083 >>>>>>>>>> >>>>>>>>>> We are evaluating Traffic Server specifically for ESI support, so >>>>>>>>>> this >>>>>>>>>> bug >>>>>>>>>> is quite a blocker for me. I've tried a few different things based on >>>>>>>>>> searching the Traffic Server mailing list in the past (for example, I >>>>>>>>>> made >>>>>>>>>> sure deflate was disabled on the origin server), and now I'm stuck. >>>>>>>>>> Does >>>>>>>>>> anybody know what is going on, or can someone suggest a next idea >>>>>>>>>> for me >>>>>>>>>> to >>>>>>>>>> debug the problem? Thanks! >>>>>>>>>> >>>>>>>>>> Humbly, >>>>>>>>>> >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Andrew Ho >>>>>>>>>> Senior Software Developer >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >
