I don't know that much about how to manage jobs in Hadoop using the API. Maybe Tom can provide a good answer to this. I completely understand the elegance part :)
-- Andrei Savu On Thu, Sep 22, 2011 at 3:17 PM, David Alves <[email protected]> wrote: > First there is the question of accuracy, as I said I am collecting metrics > that I'd like to be as accurate as possible. > Second there is the matter of elegance. I always like to avoid polls whenever > possible. > > That being said, I don't wan't to embark in some odyssey just to avoid poll, > so if it really is too much trouble I am ok with letting it go. > Anyhow even with poll is there something already implemented that enables it > in generic cases? > > thanks > -david > > On Sep 22, 2011, at 5:09 PM, Andrei Savu wrote: > >> Why is so important to avoid having a poll? The cost is low and almost >> any job is running at least for a few minutes. >> >> -- Andrei >> >> On Thu, Sep 22, 2011 at 3:07 PM, David Alves <[email protected]> wrote: >>> Hi Andrei >>> >>> I know… >>> The thing is that code used the Hadoop JobClient class's runJob() >>> method that actually polls for progress. >>> I am not using hadoop (in hindsight using the word "job" might have >>> been a mistake) and I was wondering if there is already a way to do that >>> for generic cases (e.g., scripts or java programs). >>> In particular as I'm collecting accurate metrics I'd like a non poll >>> based technique. >>> Even if there is none I can always try and code it, so all ideas are >>> welcome. >>> >>> thanks >>> david >>> >>> >>> On Sep 22, 2011, at 4:52 PM, Andrei Savu wrote: >>> >>>> This is exactly what the example code is doing (and the hadoop >>>> integration test). The job running code is blocking while the job is >>>> executing. >>>> >>>> -- Andrei Savu / andreisavu.ro >>>> >>>> On Thu, Sep 22, 2011 at 2:03 PM, David Alves <[email protected]> wrote: >>>>> Hi All >>>>> >>>>> I need to launch a cluster run a job and terminate the cluster as >>>>> the job is finished (as soon as possible). >>>>> Is there any "nice" way to do this, or do you have any suggestions? >>>>> On the top of my head I can imagine some quick and dirty solutions >>>>> (like creating a file whenever the task is completed and polling for its >>>>> existence from the whirr handler) but I'd like to do it without polling >>>>> if possible. Any ideas? >>>>> >>>>> thanks >>>>> -david >>>>> >>> >>> > >
