Re: Tomcat - a search engine liability?!?!
On Wed, 11 Dec 2002, Bill Barker wrote: Date: Wed, 11 Dec 2002 23:33:43 -0800 From: Bill Barker [EMAIL PROTECTED] Reply-To: Tomcat Users List [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Re: Tomcat - a search engine liability?!?! It's probably not much help, but Tomcat 5.x will almost certainly have support to do an internal-redirect (like Apache/httpd), instead of the 302 response. It is more-or-less required by the current draft of JSR-154 (that Tomcat 5 will implement). To be of even less help ;-), there is an option in Tomcat 3.3.2-dev (aka nightly) to suppress the 302, and serve the welcome-page directly. And when you take advantage of either of these things, be prepared and code your pages to avoid the most common gotcha -- forwarding instead of redirecting will change how relative URLs inside your page are resolved (such as in an img src=... path). The best solution I've found for this is to use a base element inside head, but that can be pretty tedious unless you have a custom tag that figures out the write base URL for you (like Struts does). Craig neal [EMAIL PROTECTED] wrote in message [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... Thanks but the problem isn't with jsps, and actually, mapping to another extension was more of a secondary problem. The primary problem becomes that I need Tomcat to not redirect (http 302) to a different URL, I want it to stay at http://www.xyz.com and simply show the contents of the default page rather than redirect ... a behavior seen in every other http server. The servlets issue was mentioned (a) to let everyone know that they don't index and (b) to say that using a servlet mapped to / that would simply forward to correct default file won't work because servlets don't index. So, I'm guessing this filter thing is something to look at, mod_rewrite too. I'm also going to take a look at apache - my big hesitation there is that the SSL certificate is already bound to Tomcat ... I think I'd have to buy another to work with Apache. Sigh. Thanks for the thoughts. Neal -Original Message- From: Joe Tomcat [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 3:50 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! On Wed, 2002-12-11 at 19:19, neal wrote: Thanks for the suggestions, I'll definitely have to look into them. For someone unfamiliar with these things those mod_rewrite or filters do you think it would be easier to just throw in the towel on using tomcat for http serving and move Apache in front of it, or is it easier to write one of these solutions? Don't throw in that towel yet! Here's one easy solution: Configure Tomcat so that .htm pages are handled as jsps. This is easy to do: Modify the web.xml (for the server, not the application itself) so that the jsp servlet handles .htm files. Also .htm should be taken out of the mime types of web.xml in that same file. Obviously, be very careful that all your static html files are .html if you do this. One advantage of this is that it means that your server isn't advertising what kind of back-end technology it is using. Less information is often better. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
ASP, and JSP were the only dynamic extensions I consistantly saw that were being indexed. Google does NOT index any servlet, framework class, or cgi file. I haven't reviewed your facts for accuracy, so take this with a grain of salt. But *IF* the world according to Google is as you say it is, and I needed to use some funky extension, I would consider using mod_rewrite to rewrite request URLs, and a filter to rewrite URLs in the response data stream. This is similar to an issue recently raised in a thread static url routing. In your case, the browser might see http://host/mypath/mdlx/page.html and you would want tomcat to see /mypath/page.mdlx. Actually, I would always rewrite request URLs, but only rewrite the response data stream for a search engine like Google. Waste of cycles otherwise, and I'd want to eliminate the rewrite when search engines are more RESTful. Tomcat standalone automatically redirects (http 302) to [welcome file] I assume that you mean '/' -- 302 -- '/index.jsp', as in your example of http://www.xyz.com [goes to] http://www.xyz.com/index.html;? IIRC, you can eliminate the round trip to the browser by rewriting the URL, e.g., RewriteRule ^(.*)/$ $1/index.jsp -- or whatever you want to use. --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
This is certainly no long term and all encompassing solution, but if you had certain Servlets you wanted indexed by a search engine, you can map almost any URL to a servlet in the web.xml. Check servlet spec for details, but I seem to remember the ability to even map a file.html to a servlet class of com.company.mycode.SomeServlet. -Original Message- From: Noel J. Bergman [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 6:25 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! ASP, and JSP were the only dynamic extensions I consistantly saw that were being indexed. Google does NOT index any servlet, framework class, or cgi file. I haven't reviewed your facts for accuracy, so take this with a grain of salt. But *IF* the world according to Google is as you say it is, and I needed to use some funky extension, I would consider using mod_rewrite to rewrite request URLs, and a filter to rewrite URLs in the response data stream. This is similar to an issue recently raised in a thread static url routing. In your case, the browser might see http://host/mypath/mdlx/page.html and you would want tomcat to see /mypath/page.mdlx. Actually, I would always rewrite request URLs, but only rewrite the response data stream for a search engine like Google. Waste of cycles otherwise, and I'd want to eliminate the rewrite when search engines are more RESTful. Tomcat standalone automatically redirects (http 302) to [welcome file] I assume that you mean '/' -- 302 -- '/index.jsp', as in your example of http://www.xyz.com [goes to] http://www.xyz.com/index.html;? IIRC, you can eliminate the round trip to the browser by rewriting the URL, e.g., RewriteRule ^(.*)/$ $1/index.jsp -- or whatever you want to use. --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
Colin, The difference is that servlet mapping doesn't allow regex replacement, whereas mod_rewrite allows: RewriteRule ^(.*)/mdlx/(.*).html$ $1/$2.mdlx That is completely off-the-cuff, so it might be syntactically wrong. If anyone here isn't familar with mod_rewrite, I highly recommend the URL Rewriting Guide: http://httpd.apache.org/docs-2.0/misc/rewriteguide.html. To paraphrase a famous slogan: mod_rewrite -- Don't Run Your Web Server Without It. --- Noel -Original Message- From: Madere, Colin [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 19:47 To: 'Tomcat Users List' Subject: RE: Tomcat - a search engine liability?!?! This is certainly no long term and all encompassing solution, but if you had certain Servlets you wanted indexed by a search engine, you can map almost any URL to a servlet in the web.xml. Check servlet spec for details, but I seem to remember the ability to even map a file.html to a servlet class of com.company.mycode.SomeServlet. -Original Message- From: Noel J. Bergman [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 6:25 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! ASP, and JSP were the only dynamic extensions I consistantly saw that were being indexed. Google does NOT index any servlet, framework class, or cgi file. I haven't reviewed your facts for accuracy, so take this with a grain of salt. But *IF* the world according to Google is as you say it is, and I needed to use some funky extension, I would consider using mod_rewrite to rewrite request URLs, and a filter to rewrite URLs in the response data stream. This is similar to an issue recently raised in a thread static url routing. In your case, the browser might see http://host/mypath/mdlx/page.html and you would want tomcat to see /mypath/page.mdlx. Actually, I would always rewrite request URLs, but only rewrite the response data stream for a search engine like Google. Waste of cycles otherwise, and I'd want to eliminate the rewrite when search engines are more RESTful. Tomcat standalone automatically redirects (http 302) to [welcome file] I assume that you mean '/' -- 302 -- '/index.jsp', as in your example of http://www.xyz.com [goes to] http://www.xyz.com/index.html;? IIRC, you can eliminate the round trip to the browser by rewriting the URL, e.g., RewriteRule ^(.*)/$ $1/index.jsp -- or whatever you want to use. --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
On Wed, 11 Dec 2002, Noel J. Bergman wrote: Date: Wed, 11 Dec 2002 20:15:32 -0500 From: Noel J. Bergman [EMAIL PROTECTED] Reply-To: Tomcat Users List [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Subject: RE: Tomcat - a search engine liability?!?! Colin, The difference is that servlet mapping doesn't allow regex replacement, whereas mod_rewrite allows: RewriteRule ^(.*)/mdlx/(.*).html$ $1/$2.mdlx That is completely off-the-cuff, so it might be syntactically wrong. If anyone here isn't familar with mod_rewrite, I highly recommend the URL Rewriting Guide: http://httpd.apache.org/docs-2.0/misc/rewriteguide.html. To paraphrase a famous slogan: mod_rewrite -- Don't Run Your Web Server Without It. Or implement the equivalent remapping in a Filter if you'd prefer a pure-Java solution. It's pretty easy to do -- and you can even use regexp replacement if you want :-). --- Noel Craig -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
Thanks for the suggestions, I'll definitely have to look into them. For someone unfamiliar with these things those mod_rewrite or filters do you think it would be easier to just throw in the towel on using tomcat for http serving and move Apache in front of it, or is it easier to write one of these solutions? Also, jsut to clarify - these mappings filters and rewrites would all occur prior to initial output and result in a completely normal, well-formed server header for a requested document? I mean, http status code would be 200, the extention and content-type could appear to be html, etc? Would there be any clue to a search engine that any of these things aren't what they appear to be? Thanks. Neal -Original Message- From: Craig R. McClanahan [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 7:10 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! On Wed, 11 Dec 2002, Noel J. Bergman wrote: Date: Wed, 11 Dec 2002 20:15:32 -0500 From: Noel J. Bergman [EMAIL PROTECTED] Reply-To: Tomcat Users List [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Subject: RE: Tomcat - a search engine liability?!?! Colin, The difference is that servlet mapping doesn't allow regex replacement, whereas mod_rewrite allows: RewriteRule ^(.*)/mdlx/(.*).html$ $1/$2.mdlx That is completely off-the-cuff, so it might be syntactically wrong. If anyone here isn't familar with mod_rewrite, I highly recommend the URL Rewriting Guide: http://httpd.apache.org/docs-2.0/misc/rewriteguide.html. To paraphrase a famous slogan: mod_rewrite -- Don't Run Your Web Server Without It. Or implement the equivalent remapping in a Filter if you'd prefer a pure-Java solution. It's pretty easy to do -- and you can even use regexp replacement if you want :-). --- Noel Craig -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
The difference is that servlet mapping doesn't allow regex replacement, whereas mod_rewrite allows: RewriteRule ^(.*)/mdlx/(.*).html$ $1/$2.mdlx Or implement the equivalent remapping in a Filter if you'd prefer a pure-Java solution. It's pretty easy to do -- and you can even use regexp replacement if you want :-). Sheesh ... I leave off or use a Filter in one message ... LOL :-) Yes, someone could write a URLRewriteFilter. Might be worth doing, actually. But in the case of the person who started this thread, I had the impression that timeliness was a factor for him. --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
On Wed, 2002-12-11 at 19:19, neal wrote: Thanks for the suggestions, I'll definitely have to look into them. For someone unfamiliar with these things those mod_rewrite or filters do you think it would be easier to just throw in the towel on using tomcat for http serving and move Apache in front of it, or is it easier to write one of these solutions? Don't throw in that towel yet! Here's one easy solution: Configure Tomcat so that .htm pages are handled as jsps. This is easy to do: Modify the web.xml (for the server, not the application itself) so that the jsp servlet handles .htm files. Also .htm should be taken out of the mime types of web.xml in that same file. Obviously, be very careful that all your static html files are .html if you do this. One advantage of this is that it means that your server isn't advertising what kind of back-end technology it is using. Less information is often better. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
Neal, For someone unfamiliar with these things [do] you think it would be easier to [move] Apache in front of it, or is it easier to write [a Filter?] I would be comfortable with either, but I've already apache as the front end and experience with mod_rewrite, so I'd go for the 30 second solution before writing a new Filter. What are you more confident in your ability to do? Develop a new Filter in Java, or install apache as a front end? The only person who can answer that is you. [would] these mappings filters and rewrites [occur] prior to initial output and result in a completely normal, well-formed server header for a requested document? I mean, http status code would be Would there be any clue to a search engine that any of these things aren't what they appear to be? The user-agent wouldn't have a clue. The only tricky thing would be that if you don't have a Filter to rewrite outbound URLs, you want to write your tags as a href=/mypath/mdlx/page.html so that they are returned that way to the user-agent. Then you just map them to /mypath/page.mdlx on the way into tomcat. But as far as the user-agent is concerned, it gave you page.html, and it has no idea that you gave it back the content from page.mdlx. --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
Thanks but the problem isn't with jsps, and actually, mapping to another extension was more of a secondary problem. The primary problem becomes that I need Tomcat to not redirect (http 302) to a different URL, I want it to stay at http://www.xyz.com and simply show the contents of the default page rather than redirect ... a behavior seen in every other http server. The servlets issue was mentioned (a) to let everyone know that they don't index and (b) to say that using a servlet mapped to / that would simply forward to correct default file won't work because servlets don't index. So, I'm guessing this filter thing is something to look at, mod_rewrite too. I'm also going to take a look at apache - my big hesitation there is that the SSL certificate is already bound to Tomcat ... I think I'd have to buy another to work with Apache. Sigh. Thanks for the thoughts. Neal -Original Message- From: Joe Tomcat [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 3:50 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! On Wed, 2002-12-11 at 19:19, neal wrote: Thanks for the suggestions, I'll definitely have to look into them. For someone unfamiliar with these things those mod_rewrite or filters do you think it would be easier to just throw in the towel on using tomcat for http serving and move Apache in front of it, or is it easier to write one of these solutions? Don't throw in that towel yet! Here's one easy solution: Configure Tomcat so that .htm pages are handled as jsps. This is easy to do: Modify the web.xml (for the server, not the application itself) so that the jsp servlet handles .htm files. Also .htm should be taken out of the mime types of web.xml in that same file. Obviously, be very careful that all your static html files are .html if you do this. One advantage of this is that it means that your server isn't advertising what kind of back-end technology it is using. Less information is often better. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
Joe, Don't throw in that towel yet! Here's one easy solution: Configure Tomcat so that .htm pages are handled as jsps. He isn't serving html pages. He is serving mdlx pages, whatever those are, and he wants Google to index them. Supposedly Google doesn't, or this would all be moot. I do wonder, now that you raise a related issue, if adding a text or html mime-type for the mdlx extension would cause Google to index it. --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
the SSL certificate is already bound to Tomcat ... I think I'd have to buy another to work with Apache. Sigh. Don't do it until you prototype your solution and make sure that it works for you. The primary problem [is] to not redirect (http 302) to a different URL, I want it to stay at http://www.xyz.com and simply show the contents of the default page To make sure I wasn't giving you a bum steer, I did a quick test. Without any RewriteRule, if I went to a test http://host/, I would get: GET / HTTP/1.1 302 647 - GET /index.html HTTP/1.1 200 2841 - Adding: RewriteRule ^/$ /index.html [L] changed it so that I got: GET / HTTP/1.1 200 2841 - No more 302 redirect. :-) --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
Oh wow ... very cool. Thanks Joel. I'm going to dig into this and learn exactly what a rewrite rule is first thing tomorrow! :) Neal -Original Message- From: Noel J. Bergman [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 8:38 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! the SSL certificate is already bound to Tomcat ... I think I'd have to buy another to work with Apache. Sigh. Don't do it until you prototype your solution and make sure that it works for you. The primary problem [is] to not redirect (http 302) to a different URL, I want it to stay at http://www.xyz.com and simply show the contents of the default page To make sure I wasn't giving you a bum steer, I did a quick test. Without any RewriteRule, if I went to a test http://host/, I would get: GET / HTTP/1.1 302 647 - GET /index.html HTTP/1.1 200 2841 - Adding: RewriteRule ^/$ /index.html [L] changed it so that I got: GET / HTTP/1.1 200 2841 - No more 302 redirect. :-) --- Noel -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Tomcat - a search engine liability?!?!
I don't think that you are completly right with your conclusion. We don't use any extensions. A typical URL looks like this: www.someserver.tld/_default/_a/qp9jew/_default/SomeAction Our observations with google are: - Pages with just that URL are indexed - Pages with that URL + at least one query parameter are not indexed -Original Message- From: neal [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 11:48 PM To: Tomcat Users List Subject: Tomcat - a search engine liability?!?! First a conclusion: 1. Google does NOT index any servlet, framework class, or cgi file. If it doesn't end in jsp forget about it. My own framework ending in .mdlx and servlets without extensions appear to be S.O.L as well. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: Tomcat - a search engine liability?!?!
It's probably not much help, but Tomcat 5.x will almost certainly have support to do an internal-redirect (like Apache/httpd), instead of the 302 response. It is more-or-less required by the current draft of JSR-154 (that Tomcat 5 will implement). To be of even less help ;-), there is an option in Tomcat 3.3.2-dev (aka nightly) to suppress the 302, and serve the welcome-page directly. neal [EMAIL PROTECTED] wrote in message [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... Thanks but the problem isn't with jsps, and actually, mapping to another extension was more of a secondary problem. The primary problem becomes that I need Tomcat to not redirect (http 302) to a different URL, I want it to stay at http://www.xyz.com and simply show the contents of the default page rather than redirect ... a behavior seen in every other http server. The servlets issue was mentioned (a) to let everyone know that they don't index and (b) to say that using a servlet mapped to / that would simply forward to correct default file won't work because servlets don't index. So, I'm guessing this filter thing is something to look at, mod_rewrite too. I'm also going to take a look at apache - my big hesitation there is that the SSL certificate is already bound to Tomcat ... I think I'd have to buy another to work with Apache. Sigh. Thanks for the thoughts. Neal -Original Message- From: Joe Tomcat [mailto:[EMAIL PROTECTED]] Sent: Wednesday, December 11, 2002 3:50 PM To: Tomcat Users List Subject: RE: Tomcat - a search engine liability?!?! On Wed, 2002-12-11 at 19:19, neal wrote: Thanks for the suggestions, I'll definitely have to look into them. For someone unfamiliar with these things those mod_rewrite or filters do you think it would be easier to just throw in the towel on using tomcat for http serving and move Apache in front of it, or is it easier to write one of these solutions? Don't throw in that towel yet! Here's one easy solution: Configure Tomcat so that .htm pages are handled as jsps. This is easy to do: Modify the web.xml (for the server, not the application itself) so that the jsp servlet handles .htm files. Also .htm should be taken out of the mime types of web.xml in that same file. Obviously, be very careful that all your static html files are .html if you do this. One advantage of this is that it means that your server isn't advertising what kind of back-end technology it is using. Less information is often better. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
