Hi Andrzej, > > The main problem, as Scott observed, is that the static flag affects all > instances of the task executing inside the same JVM. If there are > several Fetcher tasks (or any other tasks that check for SEVERE flag!), > belonging to different jobs, all of them will quit. This is certainly > not the intended behavior. >
Got it. >> >>> In fact, I believe that this would make a fantastic anti-pattern. If this >>> kind of behavior is *really* wanted (and I argue that it should not be >>> below), >>> it should be done through an explicit mechanism, not as a side-effect. >>> >> >> > > I have a proposal for a simple solution: set a flag in the current > Configuration instance, and check for this flag. The Configuration > instance provides a task-specific context persisting throughout the > lifetime of a task - but limited only to that task. Voila - problem > solved. We get rid of the dubious use of LogFormatter (I hope Chris that > even you would agree that this pattern is slightly .. unusual ;) ) What, "unusual"? Huh? :-) > and > we gain flexible mechanism limited in scope to the current task, which > ensures isolation from other tasks in the same JVM. How about that? +1 I like your proposed solution. I haven't used multiple fetchers really inside the same process too, much however, I do have an application that calls fetches in more of a sequential way in the same JVM. So, I guess I just never ran across the behavior. The thing I like about the proposed solution is its separation and isolation of a task context, which I think that Nutch (now relying on Hadoop as the underlying architectural computing platform) needed to address. So, to summarize, the proposed resolution is: * add flag field in Configuration instance to signify whether or not a SEVERE error has been logged within a task's context * check this field within the fetcher to determine whether or not to stop the fetcher, just for that fetching task identified by its Configuration (and no others) Is this representative of what you're proposing Andrzej? If so, I'd like to take the lead on contributing a small patch that handles this, and then it would be great if people like Scott could test this out in their existing environments where this error was manifesting itself. Thanks! Cheers, Chris (BTW: would you like me to re-open the JIRA issue, or do you want to do it?) ______________________________________________ Chris A. Mattmann [EMAIL PROTECTED] Staff Member Modeling and Data Management Systems Section (387) Data Management Systems and Technologies Group _________________________________________________ Jet Propulsion Laboratory Pasadena, CA Office: 171-266B Mailstop: 171-246 _______________________________________________________ Disclaimer: The opinions presented within are my own and do not reflect those of either NASA, JPL, or the California Institute of Technology.
