Thanks, Chris! (And thank you, Andrzej for interpreting my rantings!)
That plan sounds fantastic and I would be happy to help out.
Scott
On Jun 5, 2006, at 1:01 PM, Chris Mattmann wrote:
Hi Andrzej,
The main problem, as Scott observed, is that the static flag
affects all
instances of the task executing inside the same JVM. If there are
several Fetcher tasks (or any other tasks that check for SEVERE
flag!),
belonging to different jobs, all of them will quit. This is certainly
not the intended behavior.
Got it.
In fact, I believe that this would make a fantastic anti-
pattern. If this
kind of behavior is *really* wanted (and I argue that it should
not be
below),
it should be done through an explicit mechanism, not as a side-
effect.
I have a proposal for a simple solution: set a flag in the current
Configuration instance, and check for this flag. The Configuration
instance provides a task-specific context persisting throughout the
lifetime of a task - but limited only to that task. Voila - problem
solved. We get rid of the dubious use of LogFormatter (I hope
Chris that
even you would agree that this pattern is slightly .. unusual ;) )
What, "unusual"? Huh? :-)
and
we gain flexible mechanism limited in scope to the current task,
which
ensures isolation from other tasks in the same JVM. How about that?
+1
I like your proposed solution. I haven't used multiple fetchers really
inside the same process too, much however, I do have an application
that
calls fetches in more of a sequential way in the same JVM. So, I
guess I
just never ran across the behavior. The thing I like about the
proposed
solution is its separation and isolation of a task context, which I
think
that Nutch (now relying on Hadoop as the underlying architectural
computing
platform) needed to address.
So, to summarize, the proposed resolution is:
* add flag field in Configuration instance to signify whether or not a
SEVERE error has been logged within a task's context
* check this field within the fetcher to determine whether or not
to stop
the fetcher, just for that fetching task identified by its
Configuration
(and no others)
Is this representative of what you're proposing Andrzej? If so, I'd
like to
take the lead on contributing a small patch that handles this, and
then it
would be great if people like Scott could test this out in their
existing
environments where this error was manifesting itself.
Thanks!
Cheers,
Chris
(BTW: would you like me to re-open the JIRA issue, or do you want
to do it?)
______________________________________________
Chris A. Mattmann
[EMAIL PROTECTED]
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not
reflect
those of either NASA, JPL, or the California Institute of Technology.
