[Nutch-dev] [patch] web-based fetcher monitor

[Stefan Groschupf]

Hi Nutch user community,

a short question, do you wish to monitor you nutch remotely or webbased like this?
http://www.stat.ee.ethz.ch/mrtg/

Now you can! :-)
I'm happy to announce a nutch SNMP plugin and a FetcherStatus information provider plugin.
These 2 plugins was the first real test for the plugin system, so with the patch comes some tiny bug fixes as well. ;-)

http://www.media-style.com/gfx/nutch/nutch-snmp-patch.zip�


To use these 2 plugin:
1.) just checkout the latest source from cvs
2.) patch the source with the snmp-patch.txt file.
3.) build nutch with "ant tar" and decompress the tar.gz to your NUTCH_TEST_HOME folder
3.) create a "plugins" folder in NUTCH_TEST_HOME
4.) store the binary distribution of "nutch.snmp.server" and "nutch.snmp.fetcher" in your NUTCH_TEST_HOME/plugins folder
5.) start the fetcher.
6.) install your favorite SNMP Gui
That's it.

In case your SNMP Gui need a MIB file you need to create one, see the nutch.snmp.fetcher/README for details.

To provide own SNMP monitor-able values for example to monitor the indexer:

1.) copy and rename "nutch.snmp.fetcher" from the plugin_sources distribution
2.) rename and edit nutch.snmp.fetcher/src/java/net/nutch/snmp/FetchOidHandler.java
3.) edit nutch.snmp.fetcher/plugin.xml
4.) contribute it to the nutch community
That's it.

I attach below the README files of both plugin and the change.log that document my changes in the nutch core as well.


Now the advertising part of this message:
<advertising>
In case you like these plugins and wish to use it for commercial issues, keep in mind there are people that never will have the chance to sit before a computer.
*Please* support
"medecins sans frontieres"
http://www.msf.org/
by making a donation.
http://www.msf.org/donations/index.cfm
In case you wish to use this plugins for non commercial projects and wasn't hungry today, think about a donations as well.
In case you serious thinking you haven't any money to make a donation please support the nutch project by improving these plugins or nutch core code by fixing possible bugs.

Thank you very much for taking this serious!
</advertising>

Hope you find these nutch add-ons useful.
Best Regards,
Stefan Groschupf

P.S. ... and remember guy's, information should be free. ;-)

=====================================
nutch-snmp-server/README:
=====================================
/* Copyright (c) 2003 The Nutch Organization. All rights reserved. */
/* Use subject to the conditions in http://www.nutch.org/LICENSE.txt. */

Stefan Groschupf, [EMAIL PROTECTED]

The nutch-snmp-server plugin provides the possibility to monitor a nutch instance via SNMP. In the plugin.xml you can setup the port on witch the SNMP server starts. Actually the port "8090" is setuped but this is not the standard SNMP port.

To monitor nutch's SNMP values you can use all SNMP Gui's that support SNMP V1, you will found a set of open source projects on www.sourceforge.net.

This plugin only implement the SNMP protocol handling and starts an SNMP agent server.
The nutch-snmp-server plugin extends a "net.nutch.startup" extension point that was added to the nutch core.
At this time this extension point will only invoked until fetcher start but by adding "ComponentStartUp.startUp();" it can be start everywhere else until start up.

To provide monitor-able information it is necessary to implement a "net.nutch.snmp.server.oidhandler" extension point by another plugin.
You can install as much "oidhandler" implementation as you wish to monitor search query traffic, content extraction or indexing processes or other values.
The ISnmpOidHandler interface is an interface with just one method, so it takes you more time to code an API to access you monitor-able values then to plug in your information source to the SNMP server.

=====================================
nutch-snmp-fetcher/README
=====================================

/* Copyright (c) 2003 The Nutch Organization. All rights reserved. */
/* Use subject to the conditions in http://www.nutch.org/LICENSE.txt. */

Stefan Groschupf, [EMAIL PROTECTED]

The nutch-snmp-fetcher plugin provides an extension to the nutch-snmp-server plugin by providing a set of fetcher status information as SNMP values.
You have to setup a sense-full OID prefix in the plugin.xml file.
Actually "5." is used as prefix, but this should changed by something like "1.2.3.4.".
The plugin extends this OID prefix then with a set of sub OID's ("1.2.3.4.x") (see below).
It make sense to create a SNMP "Management Information Base" file since you SNMP Gui may be require such an MIB file.

Following informations the nutch-snmp-fetcher plugin provides:

prefix1 RequestScheduler running time
prefix2 Requests (rate)
prefix3 Requests fetchList req/sec
prefix4 Requests robots.txt req/sec
prefix5 Retries (rate)
prefix6 Retries fetchList %
prefix7 Retries robots.txt %
prefix8 Redirects (rate)
prefix9 Redirects fetchList
prefix10 Redirects robots.txt
prefix11 success (rate)
prefix12 success fetchList
prefix13 success robots.txt

=====================================
nutch/change.log
=====================================

/* Copyright (c) 2003 The Nutch Organization. All rights reserved. */
/* Use subject to the conditions in http://www.nutch.org/LICENSE.txt. */

Change log documentation of nutch.org

(Please add your entries in the top of this document)


joa23 23.05.04 (nice date *23* + *5* + 4 = 32)

+ IStartUp: another extension point was added.

+ ComponentStartup: an singleton object that just startup the extensions registered at the IStartUp xpoint.
ComponentStartup.startUp() can be called by nutch components until start. The idea is to provide an xpoint that can be used to provide watchdog extensions or other services that need to boot up at the same time as an the nutch tool starts.


+ CoreExtensionPoints: "net.nutch.component.startup" xpoint will be installed plugin-system bootup.

+ PluginManifestParser: fixed a bug that was crashing the plugin system, when no folder was located in the plugin-folder.

+ PluginRepository: add a addExtensionPoint method to allow installing extension point via API until runtime.

+ PluginTestUil: a tiny object in the test package that encapsulate installing of extension points via API for testing issues.

+ nutch-snmp-server: was created
nutch-snmp-server is a snmp agent server that can easily extended to provide monitor functionality to nutch components like fetcher or indexer.
It provides a extension point "net.nutch.snmp.oidhandler". All SNMP requests are identified by an Object ID so called OID like "1.2.5.6.7"
The nutch-snmp-server plugin just encapsulate the SNMP protocol and forward the requests to extensions that are installed at the "net.nutch.snmp.oidhandler" xpoint. The OIDHandler extension implementation handle the request and response a SNMP value.
So everything can be monitored by just providing a new OIDHandler extension implementation from fetcher success to the performance of an summarizing plugin.
Just be creative. ;)

+ RequestScheduler: add one line to the main method to use the IStartUp extension point.

+ FetcherStatus: and its data was not available from outside, because until object generation all data was reseted.
I had made a Singleton out of the FetcherStatus Object so it is now accessible from outside as well.

+ RequestScheduler: gets the FetcherStatus now via the FetcherStatus.getInstance() method.

+ Plugin.xml: added to the root folder of nutch to define nutchs core extension points in a manifest file as well.

+ PluginManifestParser: fixed a bug. The dependency entries of a plugin.xml was never parsed but completely handled in the rest of the system. Strange...

+ TestPluginSystem: add a test that checks if the classloader contains all required libraries of its depended Plugins.

+ PluginManifestParser: correct a typo "extport" to "export" in the plugin xml.

+ UnifiedClassRepository: added a cache for already loaded classes to fix the problem, that classes of plugin loaded with different classloaders throws a class cast exception. Here may be more work can be done, to improve the mechanism.

+ FetcherStatus: adds a set of public getter methods to access information to implement a fetcher snmp information source.

+ nutch-snmp-fetcher: A new plugin that provide fetcher information as SNMP Values.