On 06/22/2010 05:37 AM, Emmanuel de Castro Santana wrote:
Hi Dennis
I was rather thinking in a simpler approach, something like a Hadoop class
I
could extend and let Hadoop take care of making it a single instance
throughout all nodes for me.
AFAIK there isn't any type of singleton cache shared among all Hadoop nodes
that could be updated in real time. The Hadoop cache places a given file or
files on all machines but it isn't real time.
If there is no such a mechanism, I will try the second approach you
described.
Running Nutch Crawl class on Eclipse, a static reference to a Map-based
cache on my plugin solves my problem, but I guess that would not work in a
distributed environment.
Correct. On a single machine using something like a MapRunner (old API),
or yes a static ref keeping everything in memory you could solve the
problem. No it wouldn't work on anything other than local mode because in
distributed mode (besides being on separate machines), each child task has
its own JVM and is executed as a separate process on the OS.
Any distributed singleton would need to be either external (memcached) or
batch based (update to HDFS and pull down at beginning of job). In both
instances you are relying on some external data store to the jobs to act as
a shared memory area for processing.
I am studying some more of the mechanics for plugin loading in Nutch, much
of it is still a mistery to me.
It is based on Eclipse 2.0 plugin architecture if that helps. The short of
it is there are extension points with interfaces. Each extension point has
plugins, classes that implement the interface. The extension points and
plugins are loaded dynamically at the beginning of each job from the
classpath.
Does it try to maintain state of the objects throughout the nodes, or each
process is supposed to run independently ?
Plugin classloaders are independent and have a dependency mechanism setup
in the plugin.xml file inside the plugin's folder. Plugins do have access
to the Nutch classes and libs. They don't necessarily have access to other
plugin classes and their libs.
Dennis
Thanks for the help.
2010/6/20 Dennis Kubes<[email protected]>
Off the top of my head there are a couple of different ways to do this:
1. Create a list of the images parsed and add them to some file(s) on
the HDFS. Then use that file to ignore going forward. You would
probably need to do some type of file merge at the end and your
complexity would grow with the number of images parsed. The file
type could also be something like SQLLite or BDB for fast access.
The key here is either you have multiple files you are searching
or you are merging intermediate and existing files at some point.
2. Use Memcached or a similar system to hold your real time
information and access that from your Hadoop tasks. This should
give you the singelton type of structure you are looking for. It
is still possible that some images are parsed more than once due
to distributed race conditions but it would significantly reduce
that number. You could also use HBase for something like this
although I think key value (memcached) versus record store (HBase)
is better in this instance.
The distributed cache isn't really what you are looking for unless the
batch ignore method described above is what you want. In that case the
case
gives you an optimization of have the file on every local machine between
jobs.
Dennis
On 06/20/2010 09:35 AM, Emmanuel de Castro Santana wrote:
Hi everybody !
I am developing a image parsing plugin for Nutch 1.1 to use it for
intranet
pages.
This plugin traverses the document getting all images and parsing them
synchronously.
The parsing process is not expensive if we do not parse the same images
again and again whenever they are found throughout the site.
About 70% of the parsed pages contain only the same images, things like
logos, footer images, etc.
To avoid the unnecessary load, we are now trying to build a sort of
cache
in
which we put all image addresses that have already been processed.
The problem is that we are running in a distributed environment.
With that in mind, is there a kind of Hadoop level Distributed Cache our
plugin can access so all cores know what has already been parsed by the
other cores ? A singleton object all cores could see would suffice.
I have read about Hadoop's DistributedCache, but it doesn't seem to be
what
I need.
http://hadoop.apache.org/common/docs/r0.18.3/api/org/apache/hadoop/filecache/DistributedCache.html
http://hadoop.apache.org/common/docs/r0.18.3/mapred_tutorial.html#DistributedCache
Thanks in advance,
Emmanuel de Castro Santana
