On Mon, 2005-02-14 at 16:10 -0500, David A. Desrosiers wrote: > > I imagine you're aware that it sounds like while this is what Should > > happen, it isn't what Is happening. > > Agreed, so that needs to be addressed.
Happy to hear you see it that way. :) > > Yes, yes, of course. The point appears to be that it Isn't > > happening. > > So let's pinpoint it and fix it! ;) Agreed, and yet I barely have time (actually, I -don't- have time) to pay adequate attention to the FLOSS projects I've already more or less committed my time to. > > Interesting. I don't suppose "pounding" means "approximately once a > > day"? > > I count 513 hits so far.. that would be a bit more than one > hit per-day. Its not a major issue, but spidering plkr.org with the > Plucker spider once a day (every link), is probably not a great idea. > You'll proably want to be checking whether or not the content has > changed first, before re-spidering it. That's about what I expected then, except I was expecting to only do an RSS feed... Should adding -c to my plucker-build help? (How do you check if something has changed, without first downloading it? Is there some sort of timestamping and/or message digesting going on that I'm not familiar with?) > >> Once the error in their script has been corrected, I will > >> remove the block. > > > Please define "error". :) > > Spidering the site repeatedly (every link, several hundred) > without checking whether the content has changed or not. If every user > of Plucker did that, the site would be so inundated with hits from > spiders, that no actual valid (human) users would be able to visit the > site to take advanrage of it. OK, no problem. I've reduced the frequency to once every two weeks, instead of once per day. The site doesn't seem to change much anyway. > I've removed the block for now, please adhere to robots.txt > and increase the delay between your spidering hits, so you don't end > up blocking users who want to use the site. Thanks. BTW, does plucker-build respect robots.txt?
signature.asc
Description: This is a digitally signed message part
