Andy, You could use CF to handle the .txt extension and then write a dyanmic script that "figures out" what the output should look like - including absolute paths etc. A single robots.txt file could gin up various outputs based on data in a database or parsing through the CGI vars or whatever.
The downside is that the CF engine is now going to take a look at all your txt files - so if your site serves a lot of static txt files it's additional overhead to the CF engine. But if you don't server more than the occasional txt file and/or The 404 idea is not going to work because a 404 is a status code error that is thrown (to the browser I think) - which is going to cause indexers to turn away assuming there is not robots file. -Mark Mark A. Kruger, CFG, MCSE (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -----Original Message----- From: Andy Matthews [mailto:[email protected]] Sent: Wednesday, May 27, 2009 8:41 AM To: cf-talk Subject: Dynamic robots.txt We have an application which delivers about 2000 websites. This is really nice for lots of things, but not so nice for items such as robots.txt where we might want unique values. Reviewing the spec for robots.txt it requires that sitemap references be absolute paths (while everything else can be root relative). As a hard coded file we obviously can't do this. A coworker had the idea to "delete" our robots.txt file and use our 404 handler to toss that request to ColdFusion, and write in the absolute path to our sitemap each time the file is requested. 1) Can this be done reliably? 2) Is anyone out there doing this already and could answer these questions? Andy Matthews Senior Web Developer www.dealerskins.com P Please consider the environment before printing this e-mail. Total customer satisfaction is my number 1 priority! If you are not completely satisfied with the service I have provided, please let me know right away so I can correct the problem, or notify my manager Aaron West at [email protected]. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:322825 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
