The thing does not look very rosy ;-<
I have been trying to use Plugin.java to keep ftp
connections persistently open, one connection per fetcher thread.
When running fetcher using multiple threads (2, 4, 8, etc.), there are
always threads (not all), in which such a connection (socket)
disconnects/drops sporadically.
I suggest Stefan to add in one more junit test in
./src/test/net/nutch/plugin/SimpleTestPlugin.java, that will
call a plugin extension in two or more threads, while keeping
some simple sockets open cross calls.
The current SimpleTestPlugin.java merely tests configuration of the
plugin system and initial startup of dummy extensions in single thread.
A true test had better, to some extent, simulate nutch's typical
running mode: multiple threading. Such a test will also help people
tremendously if they are about to write flexible plugin extensions,
given the fact that source codes are the only "docs" they can use.
The nutch-snmp-patch.zip has code showing use of Plugin.java,
but it's for single thread only. That patch aslo has quite a few lines
for the plugin core. Don't know if they are crucial. One other thing
not clear to me is when/how to use method shutDown() properly.
John
On Tue, Jul 20, 2004 at 12:00:54AM -0700, [EMAIL PROTECTED] wrote:
> Stefan,
>
> It seems working now. If nothing unusual happens, I am going
> to commit the change in two days.
>
> Thanks,
>
> John
>
> On Mon, Jul 19, 2004 at 12:07:51AM +0200, Stefan Groschupf wrote:
> > Hi John,
> > sorry, I can not reproduce your problem here on my machine. (macosx,
> > jdk 1.4.2)
> > Can you please try this patch and tell me if this fix the problem?
>
> > Index: Extension.java
> > ===================================================================
> > RCS file: /cvsroot/nutch/nutch/src/java/net/nutch/plugin/Extension.java,v
> > retrieving revision 1.1
> > diff -u -r1.1 Extension.java
> > --- Extension.java 20 May 2004 18:19:04 -0000 1.1
> > +++ Extension.java 18 Jul 2004 21:58:19 -0000
> > @@ -117,21 +117,26 @@
> > * @return Object An instance of the extension implementation
> > */
> > public Object getExtensionInstance() throws PluginRuntimeException {
> > - try {
> > - PluginClassLoader loader = fDescriptor.getClassLoader();
> > - Class extensionClazz = loader.loadClass(getClazz());
> > - // lazy loading of Plugin in case there is no instance of the plugin
> > - // already.
> > - PluginRepository.getInstance().getPluginInstance(getDiscriptor());
> > - Object object = extensionClazz.newInstance();
> > - return object;
> > - } catch (ClassNotFoundException e) {
> > - throw new PluginRuntimeException(e);
> > - } catch (InstantiationException e) {
> > - throw new PluginRuntimeException(e);
> > - } catch (IllegalAccessException e) {
> > - throw new PluginRuntimeException(e);
> > - }
> > + synchronized (getId()) {
> > + try {
> > +
> > + PluginClassLoader loader = fDescriptor.getClassLoader();
> > + Class extensionClazz = loader.loadClass(getClazz());
> > + // lazy loading of Plugin in case there is no instance of the
> > + // plugin
> > + // already.
> > + PluginRepository.getInstance().getPluginInstance(
> > + getDiscriptor());
> > + Object object = extensionClazz.newInstance();
> > + return object;
> > + } catch (ClassNotFoundException e) {
> > + throw new PluginRuntimeException(e);
> > + } catch (InstantiationException e) {
> > + throw new PluginRuntimeException(e);
> > + } catch (IllegalAccessException e) {
> > + throw new PluginRuntimeException(e);
> > + }
> > + }
> > }
> > /**
> > * return the plugin descriptor.
> > Index: PluginRepository.java
> > ===================================================================
> > RCS file: /cvsroot/nutch/nutch/src/java/net/nutch/plugin/PluginRepository.java,v
> > retrieving revision 1.3
> > diff -u -r1.3 PluginRepository.java
> > --- PluginRepository.java 6 Jul 2004 17:28:30 -0000 1.3
> > +++ PluginRepository.java 18 Jul 2004 21:58:19 -0000
> > @@ -177,6 +177,7 @@
> > if (fActivatedPlugins.containsKey(pDescriptor.getPluginId()))
> > return (Plugin) fActivatedPlugins.get(pDescriptor.getPluginId());
> > try {
> > + synchronized (pDescriptor) {
> > PluginClassLoader loader = pDescriptor.getClassLoader();
> > Class pluginClass = loader.loadClass(pDescriptor.getPluginClass());
> > Constructor constructor = pluginClass
> > @@ -186,6 +187,7 @@
> > plugin.startUp();
> > fActivatedPlugins.put(pDescriptor.getPluginId(), plugin);
> > return plugin;
> > + }
> > } catch (ClassNotFoundException e) {
> > throw new PluginRuntimeException(e);
> > } catch (InstantiationException e) {
>
> >
> >
> > Can you provide any junit test that let reproduce the problem on your
> > and other machines?
> >
> > Thanks!
> > Stefan
> >
> > ---------------------------------------------------------------
> > enterprise information technology consulting
> > open technology: http://www.media-style.com
> > open discussion: http://www.text-mining.org
> > open thoughts: http://www.find23.net
>
> __________________________________________
> http://www.neasys.com - A Good Place to Be
> Come to visit us today!
>
__________________________________________
http://www.neasys.com - A Good Place to Be
Come to visit us today!
-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
