If you wrote a simple URL fetcher function for Cascading, you would
have a very powerful web crawler that would dwarf Nutch in flexibility.
That said, Nutch is optimized for storage, has supporting tools,
ranking algorithms, and has been up against some nasty html and other
document types. building a really robust crawler is non-trivial.
If i was just starting out and needed to implement a proprietary
process, I would use Nutch for fetching raw content, and refreshing
it. then use Cascading for parsing, indexing, etc.
cheers,
chris
On Sep 8, 2008, at 12:42 AM, tarjei wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Alex (and others).
You should take a look at Nutch. It's a search-engine built on
Lucene,
though it can be setup on top of Hadoop. Take a look:
This didn't help me much. Although the description I gave of the basic
flow of the app seems to be close to what Nutch is doing (and I've
been
looking at the Nutch code), the questions are more general and not
related to indexing as such, but about code organization. If someone
has
more input to those, feel free to add it.
On Mon, Sep 8, 2008 at 2:54 AM, Tarjei Huse <[EMAIL PROTECTED]> wrote:
Hi, I'm planning to use Hadoop in for a set of typical crawler/
indexer
tasks. The basic flow is
input: array of urls
actions: |
1. get pages
|
2. extract new urls from pages -> start new job
extract text -> index / filter (as new jobs)
What I'm considering is how I should build this application to fit
into the
map/reduce context. I'm thinking that step 1 and 2 should be
separate
map/reduce tasks that then pipe things on to the next step.
This is where I am a bit at loss to see how it is smart to
organize the
code in logical units and also how to spawn new tasks when an old
one is
over.
Is the usual way to control the flow of a set of tasks to have an
external
application running that listens to jobs ending via the
endNotificationUri
and then spawns new tasks or should the job itself contain code to
create
new jobs? Would it be a good idea to use Cascading here?
I'm also considering how I should do job scheduling (I got a lot of
reoccurring tasks). Has anyone found a good framework for job
control of
reoccurring tasks or should I plan to build my own using quartz ?
Any tips/best practices with regard to the issues described above
are most
welcome. Feel free to ask further questions if you find my
descriptions of
the issues lacking.
Kind regards,
Tarjei
Kind regards,
Tarjei
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFIxNdWYVRKCnSvzfIRAnJ0AJ9EcXzdyZgouN8q6wtad63SUHP/twCfZ88o
9km8MTJcTQxnc7bijR1Oxs0=
=79fZ
-----END PGP SIGNATURE-----
--
Chris K Wensel
[EMAIL PROTECTED]
http://chris.wensel.net/
http://www.cascading.org/
