Re: coyote httpconnector design
thanks for responding to my post remy. I started looking at the connector coyote last week, but I haven't gone through all of it yet. Remy Maucherat wrote: There's the source code at this point ;-) j-t-c/coyote, j-t-c/http11 and of course j-t-c/util. coyote is an adapter for Tomcat (3.3.x and 4.x). http11 is an HTTP/1.1 processor (takes an input stream with the request bytes, and writes the response bytes to an output stream). The httpconnector has the two mixed together (so it's harder to maintain), with an unusual way of handling response buffering. The HTTP request parsing code is relatively similar, with the new one having fancier and more efficient buffering. The HTTP response in Coyote is based on the OutputBuffer from Tomcat 3.3, which makes the ouptut a lot more efficient, and allows to recycle the writers and output streams used. Generally, the implementation of the new HTTP processor is a lot more elegant and robust (plus it's quite fast, which doesn't hurt). My benchmarks definitely show the gains in the improved buffering and object recycling. I suspect most of the in-efficiencies are in the code generated by the jsp compiler. I have no idea why the directive is slower. In any case, I don't see how it would make a difference for the connector, so the explanation is probably in the Jasper code. One test which would be useful to get a better idea would be to assemble the pages manually, and comparing with the previous results. Perhaps when you have time, we can continue this discussion privately and get into nitty gritty details. Later today and next week I am going to run my next set of tests as I mentioned earlier. If jasper is the cause, then my next set of tests which doesn't use jstl and only java should be faster than both jslt w/include directive and jslt w/action include. Or atleast that is my guess at this point. It's designed to be fast, but I don't know why the performance gains get bigger when including. Remy Here is an example of the kind of code it generates when using include directive with jstl. _jspx_th_c_set_1.setVar(navtest); _jspx_th_c_set_1.setScope(request); try { int _jspx_eval_c_set_1 = _jspx_th_c_set_1.doStartTag(); if (_jspx_eval_c_set_1 != javax.servlet.jsp.tagext.Tag.SKIP_BODY) { try { if (_jspx_eval_c_set_1 != javax.servlet.jsp.tagext.Tag.EVAL_BODY_INCLUDE) { out = pageContext.pushBody(); _jspx_th_c_set_1.setBodyContent((javax.servlet.jsp.tagext.BodyContent) out); _jspx_th_c_set_1.doInitBody(); } do { // end // HTML // begin [file=/common/init.jsp;from=(13,41);to=(14,2)] out.write(\r\n ); // end // begin [file=/common/init.jsp;from=(14,2);to=(14,12)] /* c:choose */ org.apache.taglibs.standard.tag.common.core.ChooseTag _jspx_th_c_choose_0 = new org.apache.taglibs.standard.tag.common.core.ChooseTag(); _jspx_th_c_choose_0.setPageContext(pageContext); _jspx_th_c_choose_0.setParent(_jspx_th_c_set_1); try { int _jspx_eval_c_choose_0 = _jspx_th_c_choose_0.doStartTag(); if (_jspx_eval_c_choose_0 == javax.servlet.jsp.tagext.BodyTag.EVAL_BODY_BUFFERED) throw new JspTagException(Since tag handler class org.apache.taglibs.standard.tag.common.core.ChooseTag does not implement BodyTag, it can't return BodyTag.EVAL_BODY_TAG); if (_jspx_eval_c_choose_0 != javax.servlet.jsp.tagext.Tag.SKIP_BODY) { do { // end // HTML // begin [file=/common/init.jsp;from=(14,12);to=(15,4)] out.write(\r\n); // end // begin [file=/common/init.jsp;from=(15,4);to=(15,47)] /* c:when */ org.apache.taglibs.standard.tag.el.core.WhenTag _jspx_th_c_when_0 = new org.apache.taglibs.standard.tag.el.core.WhenTag(); _jspx_th_c_when_0.setPageContext(pageContext); _jspx_th_c_when_0.setParent(_jspx_th_c_choose_0); _jspx_th_c_when_0.setTest(${param.A == 'one'}); try { int _jspx_eval_c_when_0 = _jspx_th_c_when_0.doStartTag(); if (_jspx_eval_c_when_0 == javax.servlet.jsp.tagext.BodyTag.EVAL_BODY_BUFFERED) throw new JspTagException(Since tag handler class org.apache.taglibs.standard.tag.el.core.WhenTag does not implement BodyTag, it can't return BodyTag.EVAL_BODY_TAG);
Re: coyote httpconnector design
Since include directive is like C include. I originally though include directive would always perform better than action include. For one thread, include directive does perform better, but 2 threads action include is twice as fast. One interesting observation with coyote include directive for 1 thread is 95% of the responses were 30ms, but the standard deviation or variance is almost 3x the mean. Other noteable observations for 1 thread is include directive see dramatic fluctuations in cpu and memory usage. This suggests JSP tags create and destroy lots of objects associated with parsing and processing. The 1 thread test could be really unpredictable (if GC happens, it will just stop all requests during that time, which will greatly increase the average). That's the only explnation I can think of. include directive - 1 thread 1000 - httpconnector ave - 33 cpu usage - 40-70 coyote ave - 179 cpu usage - 50-90% action include - 1 thread 1000 - httpconnector ave - 81 cpu usage - 60-85% coyote ave - 48 cpu usage - 40-50% include directive - 2 threads 500 - httpconnector ave - 453 cpu usage - 85-100% coyote ave - 212 cpu usage - 80-90% action include - 2 threads 500 - httpconnector ave - 299 cpu usage - 80-100% coyote ave - 89 cpu usage - 50-80% Remy -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
Re: coyote httpconnector design
Remy Maucherat wrote: Since include directive is like C include. I originally though include directive would always perform better than action include. For one thread, include directive does perform better, but 2 threads action include is twice as fast. One interesting observation with coyote include directive for 1 thread is 95% of the responses were 30ms, but the standard deviation or variance is almost 3x the mean. Other noteable observations for 1 thread is include directive see dramatic fluctuations in cpu and memory usage. This suggests JSP tags create and destroy lots of objects associated with parsing and processing. The 1 thread test could be really unpredictable (if GC happens, it will just stop all requests during that time, which will greatly increase the average). That's the only explnation I can think of. I downloaded an eval of JProbe just now, so hopefully next week I can get you more data on what's happening. If indeed the load performance is a result of over active GC and excessive object instantiation, tuning the java parameters in conjunction with pooling tag classes should dramatically improve the performance. Right now if my jsp files use 50 when tags, it creates a new instance of WhenTag 50 times. Along with that, each tag that uses test to evaluate an expression also creates an instance of an expression parser. In the current JSTL expression language support classes, it doesn't create a pool of expression parsers. If I have time, I will write an expression manager that creates a pool of parsers and recycles them. I have some extreme performance requirements, so efficiency under high load is a must :) peter lin -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
Re: Re: coyote httpconnector design
Here is more additional information. It looks like my guess might be correct. When I repeated the same exact tests on a PIII 450mhz, win2K, 512mb RAM with the following heap settings -Xms50m -Xmx280m Httpconnector out performs the other system with a faster processor using coyote. One obvious difference is with 512mb of RAM, it never hits the disk cache. In fact, it never uses it at all. Even with 64 concurrent connections. My guess is the combination of disk cache and limited memory caused two separate things to occur. 1. disk cache was being used, therefore immediately performance started degrading. 2. with less memory the heap size was smaller, there by causing GC to start at a lower threshold. This is probably why CPU usage increased so rapidly. With the large heap setting, the CPU usage is more normal and doesn't fluctuate wildly for most of the tests. Both conditions together cause the sharp performance degredation. Once I run these tests again with JProbe, I will know for sure if that is the cause. Peter Lin __ Do You Yahoo!? Yahoo! Movies - coverage of the 74th Academy Awards® http://movies.yahoo.com/ -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
coyote httpconnector design
I searched the tomcat-user archive and couldn't find anything on the design difference between coyote and httpconnector in 4.0.1-4.0.3 tomcat releases. If anyone can point in the right direction, I'd really appreciate it. Here is a little background on why I am looking for the information. I've been doing some performance benchmarks comparing coyote and httpconnector on 4.0.2 and 4.0.3 with JSTL. My test pages use a lot of includes to dynamically build the header, footer and look of a page. when I used include directive %@ include file= % the performance for 4-16 concurrent connections causes dramatic increases in CPU utilization. When I use action include as in jsp:include page=/ the performance is better. Tomcat is running on a resource limited box, 600mhz w/256Mb ram. Using include directive, the compiled class file gets close to the 64K limit (around 61K). Using action include each compiled class file is under 20K, most around 8K. Aside from the obvious business logic should be in beans, I am trying to figure out ways to improve the performance and get a better understanding of why CPU utilization shoots through the roof. Using the latest coyote beta with 4.0.3 seems to perform 2-4 times better than httpconnector, depending on the page. I've also done benchmarks with very simple pages that print out http header/request parameters and coyote seems to be twice as fast. For complex pages that have a lot of includes that call includes, the performance gains are bigger. Obviously using JSTL is more process intense than putting java code in the jsp pages, but part of the goal of this experiment is to see how much of performance hit JSTL incurs. One of the goals of this test is to hide java code, so that designers and html coders don't see java code. It may be that some of the repetative logic should be made into custom tags, but before I do that, I want to get a deeper understanding of coyote architecture. thanks. peter lin -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
Re: coyote httpconnector design
On Thu, 21 Mar 2002, peter lin wrote: Date: Thu, 21 Mar 2002 12:24:24 -0500 From: peter lin [EMAIL PROTECTED] Reply-To: Tomcat Users List [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Subject: coyote httpconnector design I searched the tomcat-user archive and couldn't find anything on the design difference between coyote and httpconnector in 4.0.1-4.0.3 tomcat releases. If anyone can point in the right direction, I'd really appreciate it. I'll let Remy speak to the details (Coyote is his baby), but the primary motivation was to improve performance. A secondary motivation was to fix some HTTP/1.1 things that were hard to make work correctly in the original HttpConnector design. Here is a little background on why I am looking for the information. I've been doing some performance benchmarks comparing coyote and httpconnector on 4.0.2 and 4.0.3 with JSTL. My test pages use a lot of includes to dynamically build the header, footer and look of a page. when I used include directive %@ include file= % the performance for 4-16 concurrent connections causes dramatic increases in CPU utilization. When I use action include as in jsp:include page=/ the performance is better. Tomcat is running on a resource limited box, 600mhz w/256Mb ram. Hmm, this result is a little counter-intuitive. The %@ include % directive causes a single (larger) JSP page to be created -- like the #include directive in C code -- versus multiple independent pages that are linked via RequestDispatcher.include() calls. I'm wondering if the resource limited part of your description is kicking in. It would be useful to compare all four combinations: - Old connector, include directive - Old connector, include action - New connector, include dirctive - New connector, include action Using include directive, the compiled class file gets close to the 64K limit (around 61K). This is a fundamental limitation of the current JSP page compiler in Jasper), because all the code generated for your page ends up in a single _jspService() method. Using action include each compiled class file is under 20K, most around 8K. Aside from the obvious business logic should be in beans, I am trying to figure out ways to improve the performance and get a better understanding of why CPU utilization shoots through the roof. Tracking down whether that CPU usage is in the connector versus in the JSP page execution would be useful -- they are pretty much independent of each other. Using the latest coyote beta with 4.0.3 seems to perform 2-4 times better than httpconnector, depending on the page. I've also done benchmarks with very simple pages that print out http header/request parameters and coyote seems to be twice as fast. For complex pages that have a lot of includes that call includes, the performance gains are bigger. I wouldn't be surprised to see even larger improvements on some real world apps, depending on how they do their request and response I/O. Obviously using JSTL is more process intense than putting java code in the jsp pages, but part of the goal of this experiment is to see how much of performance hit JSTL incurs. One of the goals of this test is to hide java code, so that designers and html coders don't see java code. It may be that some of the repetative logic should be made into custom tags, but before I do that, I want to get a deeper understanding of coyote architecture. I would suggest that you make the experiments on which connector independent of which JSP implementation technique is used, as well as running them in combination. The performance of a given JSP page is very much driven by the quality of the code generated by the compiler (just like any situation where you're compiling code). What is not obvious from your reports to date is how much joint impact there is -- but it sounds like there is more interdependency here than I would have expected. For the long term, though, I would plan on optimizing performance based on using Coyote for Tomcat stand-alone use -- it's looking pretty darn good. thanks. peter lin Craig -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
Re: coyote httpconnector design
thanks craig for responding. Here are more details. Craig R. McClanahan wrote: Here is a little background on why I am looking for the information. I've been doing some performance benchmarks comparing coyote and httpconnector on 4.0.2 and 4.0.3 with JSTL. My test pages use a lot of includes to dynamically build the header, footer and look of a page. when I used include directive %@ include file= % the performance for 4-16 concurrent connections causes dramatic increases in CPU utilization. When I use action include as in jsp:include page=/ the performance is better. Tomcat is running on a resource limited box, 600mhz w/256Mb ram. Hmm, this result is a little counter-intuitive. The %@ include % directive causes a single (larger) JSP page to be created -- like the #include directive in C code -- versus multiple independent pages that are linked via RequestDispatcher.include() calls. I'm wondering if the resource limited part of your description is kicking in. It would be useful to compare all four combinations: - Old connector, include directive - Old connector, include action - New connector, include dirctive - New connector, include action Before I saw the results, I expected include directive to perform better. Actually I did several series of tests, including the ones you suggested. My guess is it's a combination of the code generated and the thread management causing the CPU spike. One noteable detail is for 32+ concurrent connections, coyote's CPU usage occasionally went down to zero. The CPU drop had a direct effect on the response time. Here are some numbers. All times are miliseconds. include directive - 1 thread 1000 iterations - httpconnector ave - 33 cpu usage - 40-70% action include - 1 thread 1000 - httpconnector ave - 81 cpu usage - 60-85% include directive - 2 threads 500 - httpconnector ave - 453 cpu usage - 85-100% action include - 2 threads 500 - httpconnector ave - 299 cpu usage - 80-100% include directive - 4 threads 250 - httpconnector ave - 27273 cpu usage - 100% coyote connector ave - 590 action include - 4 threads 250 - httpconnector ave - 738 cpu usage - 95-100% coyote connector ave - 211 cpu usage - 15% less than httpconnector include directive - 8 threads 125 - httpconnector - failed to complete cpu usage - 100% coyote ave - 1546 cpu usage - 80-100% action include - 8 threads 125 - httpconnector ave - 1867 cpu usage - 95-100% coyote ave - 370 cpu usage - 90-100% action include - 16 threads 63 - httpconnector ave - 1323 cpu usage - 95-100% coyote ave - 792 cpu usage - 95-100% people2.jsp - 32 threads 63 - coyote ave - 1215 cpu usage - 95-100% Simple jsp page that prints out http headers java and jstl to print req param 1 thread 1000 iterations - httpconnector ave - 10 cpu usage - 20-30% 2 thread 500 iterations - httpconnector ave - 11 cpu usage - 30-40% 4 thread 250 iterations - httpconnector ave - 16 cpu usage - 40-60% 32 threads 65 iterations - httpconnector ave - 252 coyote ave - 84 64 threads 35 iterations - coyote ave - 136 Using include directive, the compiled class file gets close to the 64K limit (around 61K). This is a fundamental limitation of the current JSP page compiler in Jasper), because all the code generated for your page ends up in a single _jspService() method. related to this, are there plans to improve JSP page compilation? Tracking down whether that CPU usage is in the connector versus in the JSP page execution would be useful -- they are pretty much independent of each other. My next set of benchmarks I will compare static pages to JSP pages of varying complexity. the results from the static pages should establish the theoritical limit, but other than use something like JProbe to see what is happening, I'm not clear on the best way to see the exact cause. I would suggest that you make the experiments on which connector independent of which JSP implementation technique is used, as well as running them in combination. The performance of a given JSP page is very much driven by the quality of the code generated by the compiler (just like any situation where you're compiling code). What is not obvious from your reports to date is how much joint impact there is -- but it sounds like there is more interdependency here than I would have expected. As I perform more tests and talk with Remy, I can track down what causes the CPU spike. Or atleast I hope :) peter lin -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
Re: coyote httpconnector design
I searched the tomcat-user archive and couldn't find anything on the design difference between coyote and httpconnector in 4.0.1-4.0.3 tomcat releases. If anyone can point in the right direction, I'd really appreciate it. There's the source code at this point ;-) j-t-c/coyote, j-t-c/http11 and of course j-t-c/util. coyote is an adapter for Tomcat (3.3.x and 4.x). http11 is an HTTP/1.1 processor (takes an input stream with the request bytes, and writes the response bytes to an output stream). The httpconnector has the two mixed together (so it's harder to maintain), with an unusual way of handling response buffering. The HTTP request parsing code is relatively similar, with the new one having fancier and more efficient buffering. The HTTP response in Coyote is based on the OutputBuffer from Tomcat 3.3, which makes the ouptut a lot more efficient, and allows to recycle the writers and output streams used. Generally, the implementation of the new HTTP processor is a lot more elegant and robust (plus it's quite fast, which doesn't hurt). Here is a little background on why I am looking for the information. I've been doing some performance benchmarks comparing coyote and httpconnector on 4.0.2 and 4.0.3 with JSTL. My test pages use a lot of includes to dynamically build the header, footer and look of a page. when I used include directive %@ include file= % the performance for 4-16 concurrent connections causes dramatic increases in CPU utilization. When I use action include as in jsp:include page=/ the performance is better. Tomcat is running on a resource limited box, 600mhz w/256Mb ram. I have no idea why the directive is slower. In any case, I don't see how it would make a difference for the connector, so the explanation is probably in the Jasper code. One test which would be useful to get a better idea would be to assemble the pages manually, and comparing with the previous results. Using the latest coyote beta with 4.0.3 seems to perform 2-4 times better than httpconnector, depending on the page. I've also done benchmarks with very simple pages that print out http header/request parameters and coyote seems to be twice as fast. For complex pages that have a lot of includes that call includes, the performance gains are bigger. It's designed to be fast, but I don't know why the performance gains get bigger when including. Remy -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
Re: coyote httpconnector design
I have a few curiosity questions if you don't mind. First, what's your test configuration? What platform is the machine below running, and what else is running on it? Also, are you running the load generator on the same machine or on a different machine? Second, have you checked to see if the system is swapping memory during your tests? And, what JVM are you using? Below, I noticed you said you were going to see how fast static content would work. It might be as or more interesting to measure a servlet doing the same work as your JSP pages. I know that it's supposed to run just like a servlet after the compile is done on the first request, but it would be interesting nonetheless to remove that overhead and look at the differences. Anyway, just curious for my own information. I'll be having to do some load testing of my own in the next couple of months so this would be timely information. Best Regards, Jason Koeninger JJ Computer Consulting On Thu, 21 Mar 2002 13:48:15 -0500, peter lin wrote: thanks craig for responding. Here are more details. Craig R. McClanahan wrote: Here is a little background on why I am looking for the information. I've been doing some performance benchmarks comparing coyote and httpconnector on 4.0.2 and 4.0.3 with JSTL. My test pages use a lot of includes to dynamically build the header, footer and look of a page. when I used include directive %@ include file= % the performance for 4-16 concurrent connections causes dramatic increases in CPU utilization. When I use action include as in jsp:include page=/ the performance is better. Tomcat is running on a resource limited box, 600mhz w/256Mb ram. Hmm, this result is a little counter-intuitive. The %@ include % directive causes a single (larger) JSP page to be created -- like the #include directive in C code -- versus multiple independent pages that are linked via RequestDispatcher.include() calls. I'm wondering if the resource limited part of your description is kicking in. It would be useful to compare all four combinations: - Old connector, include directive - Old connector, include action - New connector, include dirctive - New connector, include action Before I saw the results, I expected include directive to perform better. Actually I did several series of tests, including the ones you suggested. My guess is it's a combination of the code generated and the thread management causing the CPU spike. One noteable detail is for 32+ concurrent connections, coyote's CPU usage occasionally went down to zero. The CPU drop had a direct effect on the response time. Here are some numbers. All times are miliseconds. include directive - 1 thread 1000 iterations - httpconnector ave - 33 cpu usage - 40-70% action include - 1 thread 1000 - httpconnector ave - 81 cpu usage - 60-85% include directive - 2 threads 500 - httpconnector ave - 453 cpu usage - 85-100% action include - 2 threads 500 - httpconnector ave - 299 cpu usage - 80-100% include directive - 4 threads 250 - httpconnector ave - 27273 cpu usage - 100% coyote connector ave - 590 action include - 4 threads 250 - httpconnector ave - 738 cpu usage - 95-100% coyote connector ave - 211 cpu usage - 15% less than httpconnector include directive - 8 threads 125 - httpconnector - failed to complete cpu usage - 100% coyote ave - 1546 cpu usage - 80-100% action include - 8 threads 125 - httpconnector ave - 1867 cpu usage - 95-100% coyote ave - 370 cpu usage - 90-100% action include - 16 threads 63 - httpconnector ave - 1323 cpu usage - 95-100% coyote ave - 792 cpu usage - 95-100% people2.jsp - 32 threads 63 - coyote ave - 1215 cpu usage - 95-100% Simple jsp page that prints out http headers java and jstl to print req param 1 thread 1000 iterations - httpconnector ave - 10 cpu usage - 20-30% 2 thread 500 iterations - httpconnector ave - 11 cpu usage - 30-40% 4 thread 250 iterations - httpconnector ave - 16 cpu usage - 40-60% 32 threads 65 iterations - httpconnector ave - 252 coyote ave - 84 64 threads 35 iterations - coyote ave - 136 Using include directive, the compiled class file gets close to the 64K limit (around 61K). This is a fundamental limitation of the current JSP page compiler in Jasper), because all the code generated for your page ends up in a single _jspService() method. related to this, are there plans to improve JSP page compilation? Tracking down whether that CPU usage is in the connector versus in the JSP page execution would be useful -- they are pretty much independent of each other. My next set of benchmarks I will compare static pages to JSP pages of varying complexity. the results from the static pages should establish the
Re: coyote httpconnector design
Hi jason, Here are some answers to your questions. I should have mentioned this in the original or second message. --- Jason Koeninger [EMAIL PROTECTED] wrote: I have a few curiosity questions if you don't mind. First, what's your test configuration? What platform is the machine below running, and what else is running on it? PIII 600mhz, 256meg ram, win2K, jdk1.4, memory settings in catalina.bat is default. The configuration for coyote is the stock configuration suggested on the coyote readme. The httpconnector is set higher than the normal. I'm at home now, but it should be 100min, 200max processors. With lower min/max processors the performance is not as good. The other values are default. Also, are you running the load generator on the same machine or on a different machine? Second, have you checked to see if the system is swapping memory during your tests? And, what JVM are you using? Below, I noticed you said you were going to see how fast static content would work. It might be as or more interesting to measure a servlet doing the same work as your JSP pages. I know that it's supposed to run just like a servlet after the compile is done on the first request, but it would be interesting nonetheless to remove that overhead and look at the differences. I'm using Jakarta JMeter to do the benchmark from a separate system. the other applications running on the system running tomcat are text editor and windows taskmanager. The disk caching is more heavy with httpconnector. Not very scientific, but basically I listen to the hard drive. With httpconnector, the things spins with 4+ concurrent connections. With coyote, 16+ concurrent connections I start to hear a lot of disk caching. I've been know to kill hard drives that way :) Anyway, just curious for my own information. I'll be having to do some load testing of my own in the next couple of months so this would be timely information. The other tests I am including in my next set include static html and a third set of pages that use java with include directive. When I look at the code generated, using include directive with multiple c:choose statements and nested includes within c:when, the code is pretty messy in terms of how many times tagfactory is called and nested conditional statements. Here is an example: page 1 --- %@ include file=page2.jsp % page 2 --- c:choose c:when test=some condition%@ include file=page3.jsp %/c:when repeat previous line about 10 times.. c:otherwise%@ include file=page14.jsp % /c:choose page 3 --- c:choose c:when test=some condition%@ include file=page15.jsp %/c:when repeat previous line about 10 times.. c:otherwise%@ include file=page26.jsp % /c:choose page 15 --- c:choose c:when test=some condition%@ include file=page27.jsp %/c:when repeat previous line about 10 times.. c:otherwise%@ include file=page38.jsp % /c:choose The actual pages sometimes add one more layer of c:choose to print out some text, based on request parameters. Now of course, the obvious question is why in the world are you nesting files that way? Unfortunately, I've worked on systems that use includes to build presentation logic that are pretty complex. Say I built a web based recipe database and I want to make it easy to programmatically add new custom templates, like a cobrand or myrecipe site. Alot of HTML code is dynamically generated. Using JSTL could be a way to make it so a person with minimal HTML skills can create their own cobrand easily. Best Regards, Jason Koeninger JJ Computer Consulting peter lin __ Do You Yahoo!? Yahoo! Movies - coverage of the 74th Academy Awards® http://movies.yahoo.com/ -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]