On Sep 1, 2011, at 2:47 PM, Sean Owen wrote: > That's completely right. The use case is more for restarting a failed job > rather than configuring the pipeline. You "really" want to do something > different like piece together your own job.
yeah, this is the downside to our big monolithic drivers. Oozie or others might be useful here. > > This could be as complex as we want -- it could be its own project, defining > a slightly-higher-level definition language for MR. In fact there are > already one or two like that. I was just thinking a registerJob to complement prepareJob might be useful and simple and hook into the AbstractJob/ CLI params > > I like the idea... somehow I think you'll find it hard to implement across > all the jobs since they're not even all in the same "format" at this point! +1. Standardizing this stuff is important. > > > On Thu, Sep 1, 2011 at 7:43 PM, Grant Ingersoll <[email protected]> wrote: > >> Other than opening the code and looking, is there a way we register our >> phases such that one could, via the command line, know what they are? For >> instance, I think, for now, I can skip, in my application, the first two >> phases of the RecommenderJob, but it seems a bit awkward to say --startPhase >> 2 given that at some point in a new release a new phase could be added in >> and I would then have to go check the code. Not the end of the world, but >> it seems error prone and not readily maintainable. I suppose as a bonus, >> it would be nice if one could also know where each phase expects things to >> be and in what format. Would it make sense to have the equivalent of >> prepareJob that does registerJob up front and can then be dumped out so that >> one could see the phases and their inputs, etc? >> >> -Grant >> >> -------------------------------------------- >> Grant Ingersoll >> http://www.lucidimagination.com >> Lucene Eurocon 2011: http://www.lucene-eurocon.com >> >> -------------------------------------------- Grant Ingersoll http://www.lucidimagination.com Lucene Eurocon 2011: http://www.lucene-eurocon.com
