Hi,
* Eduardo Afonso Ferreira <[email protected]> [2012-08-24 12:40:57]:
> I'm also interested in knowing what tools are out there to facilitate
> managing oozie coordinators and jobs.
> Are there tools that allow administrators to manage them via UI?
> Useful features are to suspend/resume (disable/enable), rerun, kill, etc.
> Other features could be to submit new coordinators/jobs.
> It would be nice to have it manipulate coordinator start/stop dates easily.
> Any way to specify stop date as never? Or start date as right now?
Personally I ended up writing a small deploy script in Python that is
configuration driven for our various Oozie environments (production,
testing, local, etc.). It considers a coordinator + a workflow as an
"Oozie application". Each environment has a set of applications that
need to run in it, and specifies the HDFS location, the jobtracker
location, etc.
The script will automate the following:
- clean up the deployed Oozie applications in HDFS
- deploy Oozie applications to HDFS, along with generated
configuration (that's specific to our use case).
- figure out the state of the applications configured in that
environment, if they are running or not, etc.
- kill an application
- submit an application with a start time of now and an end time of
your choice.
The script allows to automatically do these on one or all of the
applications configured in an environment. Pretty much all it does is
call the hadoop fs command line and the oozie command line to perform
all these.
Unfortunately I can't share that script because it's very tied to things
that are specific to our deployments and enviroment, but that's the
general idea.
Some future features that I'm planning to add in there are:
- making it independant of our environment so I can share the damn
thing, providing hooks so we can still have the functionality we
need but in a more "pluggable" way;
- ability to gracefully stop an application by changing the end date
to now and letting it finish;
- rerun a failed workflow of an application.
Still no UI on there though, it's purely command-line driven.
HTH,
/Maxime
>
> Thank you.
> Eduardo.
>
>
> ________________________________
> From: Miguel Lucero <[email protected]>
> To: "[email protected]" <[email protected]>
> Sent: Thursday, August 23, 2012 4:52 PM
> Subject: Oozie Scaling and Management...
>
> Hi oozie-users,
>
> I wanted to ask if anyone could point me in the direction of any resources
> that might clarify scaling oozie to a large number of applications. I'm
> interested in the deployment and management aspects of larger oozie
> platforms. I haven't been able to find anything that goes beyond surface
> level like "use automation for deployment" etc. I'm trying to understand
> exactly how others are accomplishing that. Automation frameworks? Job
> Templating? The environment I manage is growing quickly, and has very
> distinct characteristics for each of our applications making automation a fun
> challenge so I am curious about how other users are tackling this. How have
> others handled hundreds, or thousands, of oozie application/workflow
> deployments in their environment? I apologize if there is a resource I'm
> missing online for information like this, but if there isn't one, can anyone
> share their insights?
>
> Thanks in advance and I apologize if this isn't the forum I should be using
> for questions like these...
>
> ml
>
> ________________________________
> This message is private and confidential. If you have received it in error,
> please notify the sender and remove it from your system.
--
{ name: 'Maxime Petazzoni', title: 'Sr. Platform Engineer',
company: { name: 'Turn, Inc', url: 'http://www.turn.com' } }
