Paco NATHAN wrote:
What seems to be emerging here is a pattern for another special node
associated with a Hadoop cluster.
The need is to have a machine which can:
* handle setup and shutdown of a Hadoop cluster on remote server resources
* manage loading and retrieving data via a storage grid
* interact and synchronize events via a message broker
* capture telemetry (logging, exceptions, job/task stats) from the
remote cluster
On our team, one of the engineers named it a CloudController, as
distinct from JobTracker and NameNode.
In the discussion here, the CloudController pattern derives from
services provided by AWS. However, it could just as easily be mapped
to other elastic services for servers / storage buckets / message
queues -- based on other vendors, company data centers, etc.
It really depends on how the other infrastructures allocate their
machines. FWIW, the Ec2 APIs while simple, are fairly limited : you dont
get to spec out your topology, or hint at the data you want wo work with.
This pattern has come up in several other discussions I've had with
other companies making large use of Hadoop. We're generally trying to
address these issues:
* long-running batch jobs and how to manage complex workflows for them
* managing trade-offs between using a cloud provider (AWS,
Flexiscale, AppNexus, etc.) and using company data centers
* managing trade-offs between cluster size vs. batch window time
vs. total cost
Our team chose to implement this functionality using Python scripts --
replacing the shell scripts. That makes it easier to handle the many
potential exceptions of leasing remote elastic resources. FWIW, our
team is also moving these CloudController scripts to run under
RightScale, to manage AWS resources more effectively -- especially the
logging after node failures.
What do you think of having an optional CloudController added to the
definition of a Hadoop cluster?
Its very much a management problem.
If you look at https://issues.apache.org/jira/browse/HADOOP-3628
you can see the hadoop services slowly getting the ability to be
started/stopped more easily; then we need to work on the configuration
to remove the need to push out XML files to every node to change behaviour.
As a result, we can bring up clusters with different configurations on
existing, VMWare allocated or remote (EC2) farms. I say that, with the
caveat that I haven't been playing with EC2 recently on account of
having more local machines to hand, including a new laptop with enough
cores to act like its own mini-cluster.
slideware:
http://people.apache.org/~stevel/slides/deploying_on_ec2.pdf
http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf
I'm putting in for an apachecon eu talk on the topic, "Dynamic Hadoop
Clusters".
--
Steve Loughran http://www.1060.org/blogxter/publish/5
Author: Ant in Action http://antbook.org/
