Thanks, Chris! (And thank you, Andrzej for interpreting my rantings!) That plan sounds fantastic and I would be happy to help out.
Scott On Jun 5, 2006, at 1:01 PM, Chris Mattmann wrote: > Hi Andrzej, > >> >> The main problem, as Scott observed, is that the static flag >> affects all >> instances of the task executing inside the same JVM. If there are >> several Fetcher tasks (or any other tasks that check for SEVERE >> flag!), >> belonging to different jobs, all of them will quit. This is certainly >> not the intended behavior. >> > > Got it. > >>> >>>> In fact, I believe that this would make a fantastic anti- >>>> pattern. If this >>>> kind of behavior is *really* wanted (and I argue that it should >>>> not be >>>> below), >>>> it should be done through an explicit mechanism, not as a side- >>>> effect. >>>> >>> >>> >> >> I have a proposal for a simple solution: set a flag in the current >> Configuration instance, and check for this flag. The Configuration >> instance provides a task-specific context persisting throughout the >> lifetime of a task - but limited only to that task. Voila - problem >> solved. We get rid of the dubious use of LogFormatter (I hope >> Chris that >> even you would agree that this pattern is slightly .. unusual ;) ) > > What, "unusual"? Huh? :-) > >> and >> we gain flexible mechanism limited in scope to the current task, >> which >> ensures isolation from other tasks in the same JVM. How about that? > > +1 > > I like your proposed solution. I haven't used multiple fetchers really > inside the same process too, much however, I do have an application > that > calls fetches in more of a sequential way in the same JVM. So, I > guess I > just never ran across the behavior. The thing I like about the > proposed > solution is its separation and isolation of a task context, which I > think > that Nutch (now relying on Hadoop as the underlying architectural > computing > platform) needed to address. > > So, to summarize, the proposed resolution is: > > * add flag field in Configuration instance to signify whether or not a > SEVERE error has been logged within a task's context > > * check this field within the fetcher to determine whether or not > to stop > the fetcher, just for that fetching task identified by its > Configuration > (and no others) > > Is this representative of what you're proposing Andrzej? If so, I'd > like to > take the lead on contributing a small patch that handles this, and > then it > would be great if people like Scott could test this out in their > existing > environments where this error was manifesting itself. > > Thanks! > > Cheers, > Chris > > (BTW: would you like me to re-open the JIRA issue, or do you want > to do it?) > > ______________________________________________ > Chris A. Mattmann > [EMAIL PROTECTED] > Staff Member > Modeling and Data Management Systems Section (387) > Data Management Systems and Technologies Group > > _________________________________________________ > Jet Propulsion Laboratory Pasadena, CA > Office: 171-266B Mailstop: 171-246 > _______________________________________________________ > > Disclaimer: The opinions presented within are my own and do not > reflect > those of either NASA, JPL, or the California Institute of Technology. > > _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
