Thanks for the quick feedback, Stefan! :) Just to clarify the one
example you provided:
Also you plugin need to be a little bit smarter than just look for
different values, since
foo.com/cms.do?page=1 and foo.com/cms.do?page=2 are two different
pages.
The filter would flag this only if _every_ link returned in the page
that referred to foo.com contained "page=2". All you need is a single
link to different page and the filter will let this page pass.
The only case I can really see this happening is if there was no
navigation back at all (to the homepage, etc). In this case, it's
likely a recursive trap anyway.
--Matt
On Mar 13, 2006, at 8:43 PM, Stefan Groschupf wrote:
Hi Matt,
sounds very interesting. Having a extension point until updating
would be great since it would also allow to implement plugins
dealing with identically pages but different meta data, since we
have not the ability of meta data.
So instead of design the api that we give a old and a new
mapWritable we can process a crawlDatum from CrawlDb and a new one
form the fetch process.
I see following problems:
Actually the url is used as key so you can not find a old and a new
crawlDatum based on the url in case there are dynamically parameters.
Also you plugin need to be a little bit smarter than just look for
different values, since
foo.com/cms.do?page=1 and foo.com/cms.do?page=2 are two different
pages.
What I can imaging is using normalized urls to find identically
pages. Store the parameters as meta data in the crawlDatum, have a
plugin that is able to process the crawlDatum from crawlDb and the
one from the segment until database update reducing.
Just my 2 cents.
Greetings,
Stefan
Am 14.03.2006 um 02:33 schrieb Matt Kangas:
Hi nutch-dev,
I know that we have RegexUrlNormalizer already for removing
session-ids from URLs, but lately I've been wondering if there
isn't a more general way to solve this, without relying on pre-
built patterns.
I think I have an answer that will work. I haven't seen this
approach published anywhere, so any failings are entirely my
fault. ;) What I'm wondering is:
- Does this seem like a good (effective, efficient) algorithm for
catching session-id URLs?
- If so, where is the best place to implement it within Nutch?
Basic idea: session ids within URLs only cause problems for
crawlers when they change. This typically occurs when a server-
side session expires and a new id is issued. So, rather than
looking for URL argument patterns (as RegexUrlNormalizer does),
look for a value-transition pattern.
Algorithm:
1) Iterate over each page in a fetched segment
2) For each successful fetch, extract:
- The fetched URL. Call this (u0)
- All links on the page that refer to the same site/domain. Call
this set (u1..N)
3) Parse u0 into parameters (p0) as follows:
- named parameters: add (key,value) to Map
- positional (path) params: add (position,value) to Map
So for the url "http://foo.bar/spam/eggs?x=true&y=2";, pseudocode
would look like:
p0 = new HashMap();
p0.put(new Integer(1), "spam");
p0.put(new Integer(2), "eggs");
p0.put("x", "true");
p0.put("y", "2");
4) Parse u1..N into (p1..N) using the same method
5) Compare p0 with p1..N. Look for the following pattern:
- keys that are present for all p0..N, and
- values that are identical for all p1..N, and
- the value in p0 is _different_
If you see this condition, flag the page as "contains session id
that just changed" and deal with it accordingly. (Delete from
crawldb, etc)
So... for anyone who's still reading ;), does this seem like it
would work for catching session-ids? What corner-cases would trip
it up? Can you think of cases when it would fall flat? And if it
still seems worthwhile, where's the best place within Nutch to put
it? (Perhaps a new ExtensionPoint that is used by "nutch updatedb"?)
--Matt
--
Matt Kangas / [EMAIL PROTECTED]
---------------------------------------------
blog: http://www.find23.org
company: http://www.media-style.com
--
Matt Kangas / [EMAIL PROTECTED]
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
