Jeremy Bensley wrote:
There are posts every three or four days to the nutch-agent regarding bots
submitting empty forms to websites. I don't think I've seen any regular devs
reply in-list to these issues, and am just wondering if these cases are
being analyzed.
1. Is there a known (resolved or current) bug regarding Nutch submitting
forms? I could find no bug listings in JIRA for this. If it is known and
resolved, what versions of the bot exhibit this behavior?
Yes, there was a discussion on the list about this - I'm afraid this
behavior is present in both 0.7.x and 0.8. I'm going to remove the
offending code (or to put it as an option, turned off by default).
I think the biggest issue is following links for a form POST. This
definitely seems wrong to me, and thus should never be done.
There's a separate issue re whether it's OK to follow form links that
do a GET, since that's what the guy complained to us about recently.
He agreed that his form should be doing a POST, since it triggers a
massive build process, but he also said that no other crawl besides
Nutch was following these links.
I could see making that a configurable option, where it was false by
default. But we'd probably need to modify this setting to be
domain-specific, ie some sites we crawl require us to follow these
types of links to get at content, but in general we'd want to not
follow them.
2. Are the Nutch Devs replying to the emails sent to this list? I could
understand if they are replying off-list, but to an outside observer such as
myself it appears as though webmasters are not getting many replies to their
inqueries.
I can speak for myself only .. I'm not tracking that list. What about others?
I did respond to John Masone at MacFixer.net, to get the URL to the
form where Nutch was triggering a submit. So just FYI for testing the
fix, it's:
http://www.macfixer.net/contact
I don't mean to be alarmist, but I think it is in the community's best
interests to make sure that these kinds of complaints get resolved such that
nutch is a good 'citizen' and isn't blacklisted from searching sites.
Of course you are right, there is no ill will here on our part, just
a long queue of issues to address ... but it seems we have to
prioritize this one.
This brings up an issue I've been thinking about. It might make sense
to require everybody set the user-agent string, versus it having
default values that point to Nutch.
The first time you run Nutch, it would display an error re the
user-agent string not being set, but if the instructions for how to
do this were explicit, this wouldn't be much of a hardship for
anybody trying it out.
I could write up some quick text for the Wiki re what a good user
agent string should contain, and what should be on the web page that
it refers to, since we also went through that same process not too
long ago.
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
