+1 Very good idea and perfectly fits with Hadoop ecosystem.
On Sat, Aug 20, 2011 at 2:31 AM, Eric Yang <[email protected]> wrote: > Greetings All, > > We would like to propose HMS Project for inclusion in ASF Incubator as a > new podling. HMS is monitoring, administration and lifecycle management > project for Apache Hadoop clusters. The complete proposal can be found at: > > http://wiki.apache.org/incubator/HMSProposal > > The initial contents of this proposal are also pasted below for convenience. > > Thanks and Regards, > Eric > > = HMS Proposal = > > == Abstract == > > HMS is monitoring, administration and lifecycle management project for > Apache Hadoop clusters. > > == Proposal == > > HMS will simplify the process of deployment, configuration, management > and monitoring of the collection of Hadoop services and applications > that compose a Hadoop cluster. The collection of services (Hadoop > Stack) will include at least HDFS, !MapReduce, HBase, Hive, HCatalog, > Pig and Zookeeper. HMS will be easily configurable to add additional > services and applications to the stack. Our plan is to support the > Hadoop stack as a unit of deployment and configuration where only > certain pre-tested versions of software components are supported to be > part of Hadoop stack. Administrators can always enable/disable the > individual software components from the Hadoop stack per their > deployment needs. > > The main use cases that HMS is trying to address are the following: > * Hadoop stack deployment and upgrades > * Hadoop services configuration & management > * Administration of Hadoop services > * Includes starting and stopping services > * Hadoop system maintenance tasks, such as fsck, format, re-balance, > and compaction > * User access & quota management on Hadoop clusters > * Easily check and be alerted to failures in Hadoop servers > * Automated discovery of new machines that become available > * Expanding and contracting Hadoop clusters > * Automatic resynchronization to ‘desired’ state (of Hadoop stack) to > handle faulty nodes > * Handle node burn-ins (stress test nodes using Hadoop before > deploying them for production use) > * Simple monitoring and management UI > * Dynamic configuration - Hadoop configuration deduced from machine > attributes (e.g., RAM, CPU, Disk) > * Operational HBase-based (inspired by OpenTSDB) monitoring for Hadoop > clusters > * Make it possible for administrators to deploy other Hadoop related > services and client applications > > HMS is targeted to administrators responsible for managing Hadoop > clusters. HMS leverages existing data center management and monitoring > infrastructure - Nagios, LDAP, Kerberos, etc. All HMS functionality > and data will be accessible via RESTFUL APIs and command line tools to > facilitate its integration with existing data center management > suites. > > For the bare metal provisioning, the cluster admins continue to use > their existing infrastructure. Provisioning a machine from scratch is > not in the scope of the current roadmap. > > == Background == > > Hadoop’s ecosystem includes many projects (HDFS, !MapReduce, Pig, > HBase, etc.). In many cases, users and operators typically want to > deploy a combination of some projects as a stack. It takes a > significant amount of time to get a properly configured Hadoop cluster > up and running. HMS has been designed to solve that problem. HMS > automates the whole process of deploying a stack. > > HMS is being developed by developers employed with Yahoo!, Hortonworks > and IBM. Such a tool would have a large number of users and increase > the adoption of Apache Hadoop’s ecosystem. We are therefore proposing > to make HMS Apache open source. > > == Rationale == > > Hadoop clusters are complicated and difficult to deploy and manage. > The HMS project aims to improve the usability of Apache Hadoop. Doing > so will demoncratize Apache Hadoop, growing its community and > increasing the places Hadoop can be used and the problems it can > solve. By developing HMS in Apache we hope to gather a diverse > community of contributors, helping to make sure that HMS is deployable > in as many different situations as possible. members of the Hadoop > development community will be able to influence HMS’s roadmap, and > contribute to it. We believe having HMS as part of the Apache Hadoop > ecosystem will be a great benefit to all of Hadoop's users. > > == Current Status == > > Prototype available, developed by the list of initial committers. > > === Meritocracy === > > Our intent with this incubator proposal is to start building a diverse > developer community around HMS following the Apache meritocracy model. > We have wanted to make the project open source and encourage > contributors from multiple organizations from the start. We plan to > provide plenty of support to new developers and to quickly recruit > those who make solid contributions to committer status. > > === Community === > > We are happy to report that multiple organizations are already > represented by initial team. We hope to extend the user and developer > base further in the future and build a solid open source community > around HMS. > > === Core Developers === > > HMS is currently being developed by four engineers from Hortonworks - > Eric Yang, Owen O’Malley, Vitthal (a.k.a Suhas) Gogate and Devaraj > Das. In addition, a Yahoo! employee, Jagane Sundar, and an IBM > employee, Kan Zhang, are also involved. Eric, Jagane and Kan are the > original developers. All the engineers have deep expertise in Hadoop > and are quite familiar with the Hadoop Ecosystem. > > === Alignment === > > The ASF is a natural host for HMS given that it is already the home of > Hadoop, Pig, HBase, Cassandra, and other emerging cloud software > projects. HMS has been designed to solve the deployment, management > and configuration problems of the Hadoop ecosystem family of products. > HMS fills the gap that Hadoop ecosystem has been lacking in the areas > of configuration, deployment and manageability. > > == Known Risks == > > === Orphaned products & Reliance on Salaried Developers === > > The core developers plan to work full time on the project. There is > very little risk of HMS getting orphaned. HMS is in use by companies > we work for so the companies have an interest in its continued > vitality. > > === Inexperience with Open Source === > > All of the core developers are active users and followers of open > source. Eric Yang is a committer on Apache Chukwa. Owen O’Malley is > the lead of the Apache Hadoop project. Devaraj Das is an Apache > Hadoop committer and Apache Hadoop PMC member. Vitthal (Suhas) Gogate > has contributed extensively to the Hadoop Vaidya project (part of > Apache Hadoop). Jagane Sundar has been contributing, in terms of > ideas, to the Hadoop project. Kan Zhang is a Hadoop Committer. > > === Homogeneous Developers === > > The current core developers are from Hortonworks, IBM, and, Yahoo!. > However, we hope to establish a developer community that includes > contributors from several corporations. > > === Reliance on Salaried Developers === > > Currently, the developers are paid to do work on HMS. However, once > the project has a community built around it, we expect to get > committers and developers from outside the current core developers. > > === Relationships with Other Apache Products === > > HMS is going to be used by the users of Hadoop and the Hadoop > ecosystem in general. > > === A Excessive Fascination with the Apache Brand === > > While we respect the reputation of the Apache brand and have no doubts > that it will attract contributors and users, our interest is primarily > to give HMS a solid home as an open source project following an > established development model. We have also given reasons in the > Rationale and Alignment sections. > > == Documentation == > > There is documentation in Hortonworks’s internal repositories. > > == Initial Source == > > The source is currently in Hortonworks’s internal repositories. > > == Source and Intellectual Property Submission Plan == > > The complete HMS code is under Apache Software License 2. > > == External Dependencies == > > The dependencies all have Apache compatible licenses. These include > BSD, MIT licensed dependencies. > > == Cryptography == > > None > > == Required Resources == > > == Mailing lists == > > * hms-dev AT incubator DOT apache DOT org > * hms-commits AT incubator DOT apache DOT org > * hms-user AT hms incubator apache DOT org > * hms-private AT incubator DOT apache DOT org > > == Subversion Directory == > > https://svn.apache.org/repos/asf/incubator/hms > > == Issue Tracking == > > JIRA HMS > > == Initial Committers == > > * Devaraj Das (ddas AT apache DOT org) > * Vitthal Suhas Gogate (gogate AT apache DOT org) > * Owen O'Malley (omalley AT apache DOT org) > * Jagane Sunder (jagane AT sundar DOT org) > * Eric Yang (eyang AT apache DOT org) > * Kan Zhang (kzhang AT apache DOT org) > > == Affiliations == > > * Devaraj Das (Hortonworks) > * Vitthal Suhas Gogate (Hortonworks) > * Owen O'Malley (Hortonworks) > * Jagane Sunder (Yahoo) > * Eric Yang (Hortonworks) > * Kan Zhang (IBM) > > = Sponsors = > > == Champion == > > * Owen O'Malley > > === Nominated Mentors === > > * Owen O'Malley > * Arun C Murthy > > === Sponsoring Entity === > > Incubator PMC > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Thanks - Mohammad Nour ---- "Life is like riding a bicycle. To keep your balance you must keep moving" - Albert Einstein --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
