Re: [PROPOSAL] Bigtop for the Apache Incubator
Tom, Looking through the proposal with a bit more attention I've noticed that initial source code linked in here https://github.com/cloudera/bigtop refers to iTest framework as the foundation of BigTop testing. However, iTest repository at github.com/cloudera/iTest isn't available? Will it be available along with the rest of the code or it isn't opened to the public? What's the deal there? I believe it'd be beneficial for the community to be able to see the source code of the testing engine. Please correct me if I'm wrong though ;) -- Thanks, Konstantin (Cos) Boudnik 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing. On Tue, Jun 14, 2011 at 11:43, Konstantin Boudnik c...@apache.org wrote: On 14/06/11 05:26, Tom White wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. Here's a link to the proposal on the wiki http://wiki.apache.org/incubator/BigtopProposal I've also included the initial contents below. Cheers, Tom I've added my name to the committer list, I won't be working on this in much/any of work time, and am fairly overcommitted, so don't expect that much. I can contribute some of my experience in VM setup/teardown for testing RPM installations, and how to do functional testing of dynamically created Hadoop clusters. I am going to add my name to the list of the committers too. Considering my other commitments I might not be able to work much on this project, but I guess the fact that I have wrote like 50% of the underlying system framework might count for something. Cos - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Tom,
I'd like to join as a committer and mentor as well.
Alan.
On Jun 13, 2011, at 9:26 PM, Tom White wrote:
Hi,
I would like to propose Bigtop to be an Apache Incubator project.
Bigtop is a project for the development of packaging and tests of the
Hadoop ecosystem. The goal is to do testing at various levels
(packaging, platform, runtime, upgrade, etc...) developed by a
community with a focus on the system as a whole, rather than
individual projects.
Here's a link to the proposal on the wiki
http://wiki.apache.org/incubator/BigtopProposal
I've also included the initial contents below.
Cheers,
Tom
= Bigtop - Apache Hadoop Ecosystem Packaging and Test =
== Abstract ==
Bigtop - a project for the development of packaging and tests of the
Hadoop ecosystem.
== Proposal ==
The primary goal of Bigtop is to build a community around the
packaging and interoperability testing of Hadoop-related projects.
This includes testing at various levels (packaging, platform, runtime,
upgrade, etc...) developed by a community with a focus on the system
as a whole, rather than individual projects.
Build, packaging and integration test code that depends upon official
releases of the Apache Hadoop-related projects (HDFS, MapReduce,
HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by
this project. As bugs and other issues are found we expect these to be
fixed upstream.
== Background ==
The initial packaging and test code for Bigtop was developed by
Cloudera to package projects from the Apache Hadoop ecosystem and
provide a consistent, inter-operable framework.
== Rationale ==
Hadoop defines itself as:
{{{
The Apache Hadoop project develops open-source software for reliable,
scalable, distributed computing. Hadoop includes these subprojects:
* Hadoop Common: The common utilities that support the other Hadoop
subprojects.
* HDFS: A distributed file system that provides high throughput access
to application data.
* MapReduce: A software framework for distributed processing of large
data sets on compute clusters.
}}}
There are also several other Hadoop-related projects at Apache. Some
TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There
are also several new projects in the Incubator such as HCatalog and
Sqoop.
There is limited interoperability testing done the projects
themselves. The intent of this project is to build a community where
the projects are brought together, packaged, and tested for
interoperability.
Projects such as Apache Whirr (incubating), which deploy and use a
collection of Hadoop-related projects, would benefit from the
interoperability testing done by Bigtop, rather than picking and
testing project combinations themselves.
== Initial Goals ==
Much of the code for Bigtop has been released by Cloudera under the
Apache 2.0 license for over two years.
Some current goals include:
* create a set of packages for the Hadoop ecosystem, over a wide
range of platforms
* interoperability test these projects
* document project sets that are known to work well together
Bigtop’s release artifact would consist of a single tarball of
packaging and test code that, when built, would produce source and
binary Linux packages for the upstream projects.
= Current Status =
== Meritocracy ==
Bigtop was originally developed and released as an open source
packaging infrastructure, CDH, by Cloudera.
== Community ==
The community is primarily the original developers at Cloudera,
however a number of contributions to the packaging specifications have
been accepted from outside contributors. Growing a diverse community
is the main reason to bring Bigtop to the Apache Incubator.
== Core Developers ==
The core developers for Bigtop project are:
* Andrew Bayer has extensive expertise with build tools, specifically
Jenkins continuous integration and Maven.
* Peter Linnell has contributed to the RPM packaging.
* Bruno Mahé has overseen much of the development of the RPM and
Debian packaging system.
* Roman Shaposhnik designed and implemented the system testing
framework.
Many of the committers to the Bigtop project have contributed towards
Hadoop or related Apache projects (Alejandro Abdelnur, Eli Collins,
Patrick Hunt, John Sichi, Michael Stack, Tom White) and are familiar
with Apache principals and philosophy for community driven software
development.
== Alignment ==
We expect projects in Bigtop to be drawn from Hadoop and related
projects at Apache. Bigtop will complement these projects (Hadoop,
Pig, Hive, HBase, etc...) by providing an environment for contributors
interested in building more complex data processing pipelines to work
together integrating more than a single project into a well tested
whole.
= Known Risks =
== Orphaned Products ==
The contributors are leading vendors of Hadoop-based technologies and
have a long standing in the Hadoop community. There is minimal risk of
this work becoming non-strategic and the contributors are confident
that a larger
Re: [PROPOSAL] Bigtop for the Apache Incubator
Oh, I see. Looks like a dangling links indeed... Thanks for pointing out. On Thu, Jun 16, 2011 at 05:32PM, Roman Shaposhnik wrote: Hi Cos, On Thu, Jun 16, 2011 at 5:21 PM, Konstantin Boudnik c...@apache.org wrote: However, iTest repository at github.com/cloudera/iTest isn't available? Will it be available along with the rest of the code or it isn't opened to the public? A error in the document? What's the deal there? I believe it'd be beneficial for the community to be able to see the source code of the testing engine. Please correct me if I'm wrong though ;) iTest is now part of bigtop and resides under test/src/itest-common: https://github.com/cloudera/bigtop/tree/master/test/src/itest-common So in that sense it is there, but not as a separate project. The original documentation for iTest is available under this link: http://cloudera.github.com/bigtop/iTest/ And it looks like there are could be a couple of dangling links there still pointing to where iTest used to be on GitHub as a separate repository. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
On Mon, Jun 13, 2011 at 9:26 PM, Tom White tomwh...@apache.org wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. I'd like to join as a committer and mentor. -- Owen
Re: [PROPOSAL] Bigtop for the Apache Incubator
Owen, Thanks for offering to help out. Cheers, Tom On Thu, Jun 16, 2011 at 7:13 AM, Owen O'Malley omal...@apache.org wrote: On Mon, Jun 13, 2011 at 9:26 PM, Tom White tomwh...@apache.org wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. I'd like to join as a committer and mentor. -- Owen - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Tom, Looking through the proposal with a bit more attention I've noticed that initial source code linked in here https://github.com/cloudera/bigtop refers to iTest framework as the foundation of BigTop testing. However, iTest repository at github.com/cloudera/iTest isn't available? Will it be available along with the rest of the code or it isn't opened to the public? A error in the document? What's the deal there? I believe it'd be beneficial for the community to be able to see the source code of the testing engine. Please correct me if I'm wrong though ;) -- ═ Thanks, Konstantin (Cos) Boudnik 2CAC 8312 4870 D885 8616 ═6115 220F 6980 1F27 E622 Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing. On Tue, Jun 14, 2011 at 11:43, Konstantin Boudnik c...@apache.org wrote: On 14/06/11 05:26, Tom White wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. Here's a link to the proposal on the wiki http://wiki.apache.org/incubator/BigtopProposal I've also included the initial contents below. Cheers, Tom I've added my name to the committer list, I won't be working on this in much/any of work time, and am fairly overcommitted, so don't expect that much. I can contribute some of my experience in VM setup/teardown for testing RPM installations, and how to do functional testing of dynamically created Hadoop clusters. I am going to add my name to the list of the committers too. Considering my other commitments I might not be able to work much on this project, but I guess the fact that I have wrote like 50% of the underlying system framework might count for something. Cos - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Hi Cos, On Thu, Jun 16, 2011 at 5:21 PM, Konstantin Boudnik c...@apache.org wrote: However, iTest repository at github.com/cloudera/iTest isn't available? Will it be available along with the rest of the code or it isn't opened to the public? A error in the document? What's the deal there? I believe it'd be beneficial for the community to be able to see the source code of the testing engine. Please correct me if I'm wrong though ;) iTest is now part of bigtop and resides under test/src/itest-common: https://github.com/cloudera/bigtop/tree/master/test/src/itest-common So in that sense it is there, but not as a separate project. The original documentation for iTest is available under this link: http://cloudera.github.com/bigtop/iTest/ And it looks like there are could be a couple of dangling links there still pointing to where iTest used to be on GitHub as a separate repository. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Oh, I see. Looks like a dangling links indeed... Thanks for pointing out. Cos On Thu, Jun 16, 2011 at 05:32PM, Roman Shaposhnik wrote: Hi Cos, On Thu, Jun 16, 2011 at 5:21 PM, Konstantin Boudnik c...@apache.org wrote: However, iTest repository at github.com/cloudera/iTest isn't available? Will it be available along with the rest of the code or it isn't opened to the public? A error in the document? What's the deal there? I believe it'd be beneficial for the community to be able to see the source code of the testing engine. Please correct me if I'm wrong though ;) iTest is now part of bigtop and resides under test/src/itest-common: https://github.com/cloudera/bigtop/tree/master/test/src/itest-common So in that sense it is there, but not as a separate project. The original documentation for iTest is available under this link: http://cloudera.github.com/bigtop/iTest/ And it looks like there are could be a couple of dangling links there still pointing to where iTest used to be on GitHub as a separate repository. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Alan Gates has volunteered to be a committer and mentor, so I added
him to the proposal. (For some reason his email to this list didn't
come through.)
Thanks Alan!
Tom
On Mon, Jun 13, 2011 at 9:26 PM, Tom White tomwh...@apache.org wrote:
Hi,
I would like to propose Bigtop to be an Apache Incubator project.
Bigtop is a project for the development of packaging and tests of the
Hadoop ecosystem. The goal is to do testing at various levels
(packaging, platform, runtime, upgrade, etc...) developed by a
community with a focus on the system as a whole, rather than
individual projects.
Here's a link to the proposal on the wiki
http://wiki.apache.org/incubator/BigtopProposal
I've also included the initial contents below.
Cheers,
Tom
= Bigtop - Apache Hadoop Ecosystem Packaging and Test =
== Abstract ==
Bigtop - a project for the development of packaging and tests of the
Hadoop ecosystem.
== Proposal ==
The primary goal of Bigtop is to build a community around the
packaging and interoperability testing of Hadoop-related projects.
This includes testing at various levels (packaging, platform, runtime,
upgrade, etc...) developed by a community with a focus on the system
as a whole, rather than individual projects.
Build, packaging and integration test code that depends upon official
releases of the Apache Hadoop-related projects (HDFS, MapReduce,
HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by
this project. As bugs and other issues are found we expect these to be
fixed upstream.
== Background ==
The initial packaging and test code for Bigtop was developed by
Cloudera to package projects from the Apache Hadoop ecosystem and
provide a consistent, inter-operable framework.
== Rationale ==
Hadoop defines itself as:
{{{
The Apache Hadoop project develops open-source software for reliable,
scalable, distributed computing. Hadoop includes these subprojects:
* Hadoop Common: The common utilities that support the other Hadoop
subprojects.
* HDFS: A distributed file system that provides high throughput access
to application data.
* MapReduce: A software framework for distributed processing of large
data sets on compute clusters.
}}}
There are also several other Hadoop-related projects at Apache. Some
TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There
are also several new projects in the Incubator such as HCatalog and
Sqoop.
There is limited interoperability testing done the projects
themselves. The intent of this project is to build a community where
the projects are brought together, packaged, and tested for
interoperability.
Projects such as Apache Whirr (incubating), which deploy and use a
collection of Hadoop-related projects, would benefit from the
interoperability testing done by Bigtop, rather than picking and
testing project combinations themselves.
== Initial Goals ==
Much of the code for Bigtop has been released by Cloudera under the
Apache 2.0 license for over two years.
Some current goals include:
* create a set of packages for the Hadoop ecosystem, over a wide
range of platforms
* interoperability test these projects
* document project sets that are known to work well together
Bigtop’s release artifact would consist of a single tarball of
packaging and test code that, when built, would produce source and
binary Linux packages for the upstream projects.
= Current Status =
== Meritocracy ==
Bigtop was originally developed and released as an open source
packaging infrastructure, CDH, by Cloudera.
== Community ==
The community is primarily the original developers at Cloudera,
however a number of contributions to the packaging specifications have
been accepted from outside contributors. Growing a diverse community
is the main reason to bring Bigtop to the Apache Incubator.
== Core Developers ==
The core developers for Bigtop project are:
* Andrew Bayer has extensive expertise with build tools, specifically
Jenkins continuous integration and Maven.
* Peter Linnell has contributed to the RPM packaging.
* Bruno Mahé has overseen much of the development of the RPM and
Debian packaging system.
* Roman Shaposhnik designed and implemented the system testing framework.
Many of the committers to the Bigtop project have contributed towards
Hadoop or related Apache projects (Alejandro Abdelnur, Eli Collins,
Patrick Hunt, John Sichi, Michael Stack, Tom White) and are familiar
with Apache principals and philosophy for community driven software
development.
== Alignment ==
We expect projects in Bigtop to be drawn from Hadoop and related
projects at Apache. Bigtop will complement these projects (Hadoop,
Pig, Hive, HBase, etc...) by providing an environment for contributors
interested in building more complex data processing pipelines to work
together integrating more than a single project into a well tested
whole.
= Known Risks =
== Orphaned Products ==
Re: [PROPOSAL] Bigtop for the Apache Incubator
I'm happy to see this project joining the Foundation. As an Apache
Whirr committer I feel a lot of pain when it comes to testing
compatibility for multiple services across multiple operating systems
(and multiple cloud providers).
I would love to join the Bigtop team as one of the initial committers
an do some of testing work we are already doing in Whirr.
PS: having ready to use .rpm and .deb packages sounds really great!
Cheers,
-- Andrei Savu / andreisavu.ro
On Tue, Jun 14, 2011 at 7:26 AM, Tom White tomwh...@apache.org wrote:
Hi,
I would like to propose Bigtop to be an Apache Incubator project.
Bigtop is a project for the development of packaging and tests of the
Hadoop ecosystem. The goal is to do testing at various levels
(packaging, platform, runtime, upgrade, etc...) developed by a
community with a focus on the system as a whole, rather than
individual projects.
Here's a link to the proposal on the wiki
http://wiki.apache.org/incubator/BigtopProposal
I've also included the initial contents below.
Cheers,
Tom
= Bigtop - Apache Hadoop Ecosystem Packaging and Test =
== Abstract ==
Bigtop - a project for the development of packaging and tests of the
Hadoop ecosystem.
== Proposal ==
The primary goal of Bigtop is to build a community around the
packaging and interoperability testing of Hadoop-related projects.
This includes testing at various levels (packaging, platform, runtime,
upgrade, etc...) developed by a community with a focus on the system
as a whole, rather than individual projects.
Build, packaging and integration test code that depends upon official
releases of the Apache Hadoop-related projects (HDFS, MapReduce,
HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by
this project. As bugs and other issues are found we expect these to be
fixed upstream.
== Background ==
The initial packaging and test code for Bigtop was developed by
Cloudera to package projects from the Apache Hadoop ecosystem and
provide a consistent, inter-operable framework.
== Rationale ==
Hadoop defines itself as:
{{{
The Apache Hadoop project develops open-source software for reliable,
scalable, distributed computing. Hadoop includes these subprojects:
* Hadoop Common: The common utilities that support the other Hadoop
subprojects.
* HDFS: A distributed file system that provides high throughput access
to application data.
* MapReduce: A software framework for distributed processing of large
data sets on compute clusters.
}}}
There are also several other Hadoop-related projects at Apache. Some
TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There
are also several new projects in the Incubator such as HCatalog and
Sqoop.
There is limited interoperability testing done the projects
themselves. The intent of this project is to build a community where
the projects are brought together, packaged, and tested for
interoperability.
Projects such as Apache Whirr (incubating), which deploy and use a
collection of Hadoop-related projects, would benefit from the
interoperability testing done by Bigtop, rather than picking and
testing project combinations themselves.
== Initial Goals ==
Much of the code for Bigtop has been released by Cloudera under the
Apache 2.0 license for over two years.
Some current goals include:
* create a set of packages for the Hadoop ecosystem, over a wide
range of platforms
* interoperability test these projects
* document project sets that are known to work well together
Bigtop’s release artifact would consist of a single tarball of
packaging and test code that, when built, would produce source and
binary Linux packages for the upstream projects.
= Current Status =
== Meritocracy ==
Bigtop was originally developed and released as an open source
packaging infrastructure, CDH, by Cloudera.
== Community ==
The community is primarily the original developers at Cloudera,
however a number of contributions to the packaging specifications have
been accepted from outside contributors. Growing a diverse community
is the main reason to bring Bigtop to the Apache Incubator.
== Core Developers ==
The core developers for Bigtop project are:
* Andrew Bayer has extensive expertise with build tools, specifically
Jenkins continuous integration and Maven.
* Peter Linnell has contributed to the RPM packaging.
* Bruno Mahé has overseen much of the development of the RPM and
Debian packaging system.
* Roman Shaposhnik designed and implemented the system testing framework.
Many of the committers to the Bigtop project have contributed towards
Hadoop or related Apache projects (Alejandro Abdelnur, Eli Collins,
Patrick Hunt, John Sichi, Michael Stack, Tom White) and are familiar
with Apache principals and philosophy for community driven software
development.
== Alignment ==
We expect projects in Bigtop to be drawn from Hadoop and related
projects at Apache. Bigtop
Re: [PROPOSAL] Bigtop for the Apache Incubator
Edward, Andrei - thanks for volunteering to help out as initial
committers. Please add your names to the proposal on the wiki.
Cheers,
Tom
On Wed, Jun 15, 2011 at 6:15 AM, Andrei Savu as...@apache.org wrote:
I'm happy to see this project joining the Foundation. As an Apache
Whirr committer I feel a lot of pain when it comes to testing
compatibility for multiple services across multiple operating systems
(and multiple cloud providers).
I would love to join the Bigtop team as one of the initial committers
an do some of testing work we are already doing in Whirr.
PS: having ready to use .rpm and .deb packages sounds really great!
Cheers,
-- Andrei Savu / andreisavu.ro
On Tue, Jun 14, 2011 at 7:26 AM, Tom White tomwh...@apache.org wrote:
Hi,
I would like to propose Bigtop to be an Apache Incubator project.
Bigtop is a project for the development of packaging and tests of the
Hadoop ecosystem. The goal is to do testing at various levels
(packaging, platform, runtime, upgrade, etc...) developed by a
community with a focus on the system as a whole, rather than
individual projects.
Here's a link to the proposal on the wiki
http://wiki.apache.org/incubator/BigtopProposal
I've also included the initial contents below.
Cheers,
Tom
= Bigtop - Apache Hadoop Ecosystem Packaging and Test =
== Abstract ==
Bigtop - a project for the development of packaging and tests of the
Hadoop ecosystem.
== Proposal ==
The primary goal of Bigtop is to build a community around the
packaging and interoperability testing of Hadoop-related projects.
This includes testing at various levels (packaging, platform, runtime,
upgrade, etc...) developed by a community with a focus on the system
as a whole, rather than individual projects.
Build, packaging and integration test code that depends upon official
releases of the Apache Hadoop-related projects (HDFS, MapReduce,
HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by
this project. As bugs and other issues are found we expect these to be
fixed upstream.
== Background ==
The initial packaging and test code for Bigtop was developed by
Cloudera to package projects from the Apache Hadoop ecosystem and
provide a consistent, inter-operable framework.
== Rationale ==
Hadoop defines itself as:
{{{
The Apache Hadoop project develops open-source software for reliable,
scalable, distributed computing. Hadoop includes these subprojects:
* Hadoop Common: The common utilities that support the other Hadoop
subprojects.
* HDFS: A distributed file system that provides high throughput access
to application data.
* MapReduce: A software framework for distributed processing of large
data sets on compute clusters.
}}}
There are also several other Hadoop-related projects at Apache. Some
TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There
are also several new projects in the Incubator such as HCatalog and
Sqoop.
There is limited interoperability testing done the projects
themselves. The intent of this project is to build a community where
the projects are brought together, packaged, and tested for
interoperability.
Projects such as Apache Whirr (incubating), which deploy and use a
collection of Hadoop-related projects, would benefit from the
interoperability testing done by Bigtop, rather than picking and
testing project combinations themselves.
== Initial Goals ==
Much of the code for Bigtop has been released by Cloudera under the
Apache 2.0 license for over two years.
Some current goals include:
* create a set of packages for the Hadoop ecosystem, over a wide
range of platforms
* interoperability test these projects
* document project sets that are known to work well together
Bigtop’s release artifact would consist of a single tarball of
packaging and test code that, when built, would produce source and
binary Linux packages for the upstream projects.
= Current Status =
== Meritocracy ==
Bigtop was originally developed and released as an open source
packaging infrastructure, CDH, by Cloudera.
== Community ==
The community is primarily the original developers at Cloudera,
however a number of contributions to the packaging specifications have
been accepted from outside contributors. Growing a diverse community
is the main reason to bring Bigtop to the Apache Incubator.
== Core Developers ==
The core developers for Bigtop project are:
* Andrew Bayer has extensive expertise with build tools, specifically
Jenkins continuous integration and Maven.
* Peter Linnell has contributed to the RPM packaging.
* Bruno Mahé has overseen much of the development of the RPM and
Debian packaging system.
* Roman Shaposhnik designed and implemented the system testing framework.
Many of the committers to the Bigtop project have contributed towards
Hadoop or related Apache projects (Alejandro Abdelnur, Eli Collins,
Patrick Hunt, John Sichi, Michael Stack, Tom
Re: [PROPOSAL] Bigtop for the Apache Incubator
Hi, I would like my name to be added to the committer list. I am assembling and integration-testing Hadoop stacks at Yahoo. I look forward to participate in developing a community-accepted framework for packaging Hadoop components, and assembly and validation of Hadoop stacks. Thanks -- Andre Arcilla - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
On Wed, Jun 15, 2011 at 11:43 AM, Andre Arcilla arci...@yahoo-inc.com wrote: Hi, I would like my name to be added to the committer list. I am assembling and integration-testing Hadoop stacks at Yahoo. I look forward to participate in developing a community-accepted framework for packaging Hadoop components, and assembly and validation of Hadoop stacks. A strong +1. Being able to tackle some of the use cases that a company the size of Yahoo! would have to offer is definitely going to be a major factor for Bigtop success. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
+1. Looking forward to the vote.
On Jun 13, 2011, at 9:26 PM, Tom White wrote:
Hi,
I would like to propose Bigtop to be an Apache Incubator project.
Bigtop is a project for the development of packaging and tests of the
Hadoop ecosystem. The goal is to do testing at various levels
(packaging, platform, runtime, upgrade, etc...) developed by a
community with a focus on the system as a whole, rather than
individual projects.
Here's a link to the proposal on the wiki
http://wiki.apache.org/incubator/BigtopProposal
I've also included the initial contents below.
Cheers,
Tom
= Bigtop - Apache Hadoop Ecosystem Packaging and Test =
== Abstract ==
Bigtop - a project for the development of packaging and tests of the
Hadoop ecosystem.
== Proposal ==
The primary goal of Bigtop is to build a community around the
packaging and interoperability testing of Hadoop-related projects.
This includes testing at various levels (packaging, platform, runtime,
upgrade, etc...) developed by a community with a focus on the system
as a whole, rather than individual projects.
Build, packaging and integration test code that depends upon official
releases of the Apache Hadoop-related projects (HDFS, MapReduce,
HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by
this project. As bugs and other issues are found we expect these to be
fixed upstream.
== Background ==
The initial packaging and test code for Bigtop was developed by
Cloudera to package projects from the Apache Hadoop ecosystem and
provide a consistent, inter-operable framework.
== Rationale ==
Hadoop defines itself as:
{{{
The Apache Hadoop project develops open-source software for reliable,
scalable, distributed computing. Hadoop includes these subprojects:
* Hadoop Common: The common utilities that support the other Hadoop
subprojects.
* HDFS: A distributed file system that provides high throughput access
to application data.
* MapReduce: A software framework for distributed processing of large
data sets on compute clusters.
}}}
There are also several other Hadoop-related projects at Apache. Some
TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There
are also several new projects in the Incubator such as HCatalog and
Sqoop.
There is limited interoperability testing done the projects
themselves. The intent of this project is to build a community where
the projects are brought together, packaged, and tested for
interoperability.
Projects such as Apache Whirr (incubating), which deploy and use a
collection of Hadoop-related projects, would benefit from the
interoperability testing done by Bigtop, rather than picking and
testing project combinations themselves.
== Initial Goals ==
Much of the code for Bigtop has been released by Cloudera under the
Apache 2.0 license for over two years.
Some current goals include:
* create a set of packages for the Hadoop ecosystem, over a wide
range of platforms
* interoperability test these projects
* document project sets that are known to work well together
Bigtop’s release artifact would consist of a single tarball of
packaging and test code that, when built, would produce source and
binary Linux packages for the upstream projects.
= Current Status =
== Meritocracy ==
Bigtop was originally developed and released as an open source
packaging infrastructure, CDH, by Cloudera.
== Community ==
The community is primarily the original developers at Cloudera,
however a number of contributions to the packaging specifications have
been accepted from outside contributors. Growing a diverse community
is the main reason to bring Bigtop to the Apache Incubator.
== Core Developers ==
The core developers for Bigtop project are:
* Andrew Bayer has extensive expertise with build tools, specifically
Jenkins continuous integration and Maven.
* Peter Linnell has contributed to the RPM packaging.
* Bruno Mahé has overseen much of the development of the RPM and
Debian packaging system.
* Roman Shaposhnik designed and implemented the system testing framework.
Many of the committers to the Bigtop project have contributed towards
Hadoop or related Apache projects (Alejandro Abdelnur, Eli Collins,
Patrick Hunt, John Sichi, Michael Stack, Tom White) and are familiar
with Apache principals and philosophy for community driven software
development.
== Alignment ==
We expect projects in Bigtop to be drawn from Hadoop and related
projects at Apache. Bigtop will complement these projects (Hadoop,
Pig, Hive, HBase, etc...) by providing an environment for contributors
interested in building more complex data processing pipelines to work
together integrating more than a single project into a well tested
whole.
= Known Risks =
== Orphaned Products ==
The contributors are leading vendors of Hadoop-based technologies and
have a long standing in the Hadoop community. There is
Re: [PROPOSAL] Bigtop for the Apache Incubator
On 14/06/11 05:26, Tom White wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. Here's a link to the proposal on the wiki http://wiki.apache.org/incubator/BigtopProposal I've also included the initial contents below. Cheers, Tom I've added my name to the committer list, I won't be working on this in much/any of work time, and am fairly overcommitted, so don't expect that much. I can contribute some of my experience in VM setup/teardown for testing RPM installations, and how to do functional testing of dynamically created Hadoop clusters. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
I've added more on the limitations of the current process (not synchronised releases, not enough automated testing on multiple-host clusters), and on a risk of the project: the upstream projects need to care about and work on more synchronized releases. Cloudera managed to address that problem by patching their forks of the apache projects, then push back the patches by way of JIRA patches. Bigtop should not get into the game of having its own branches of any of the ASF projects, instead it should focus on getting its changes into trunk of the various projects SVN trees, and on having near-synchronized releases of the various projects That requires -engagement from the committer teams to get patches in. Having committers from the relevant projects in the BigTop project would address this. -support from the PMCs to do releases on a regular schedule. That's hard, but something like a six-monthly schedule for the big projects with the up and coming projects releasing in a proper factor of that timescale (3, 2, 1 month), could help. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Steve, Thanks for signing up. Bigtop should not maintain branches or patchsets on top of upstream projects. Any bugs or compatibility problems that are found will need to be resolved in the upstream projects and then released in those projects. This makes things harder for Bigtop, but, like you, I hope having committers from the upstream projects involved in Bigtop will give projects further impetus to make releases in a timely manner. Cheers, Tom On Tue, Jun 14, 2011 at 4:51 AM, Steve Loughran ste...@apache.org wrote: I've added more on the limitations of the current process (not synchronised releases, not enough automated testing on multiple-host clusters), and on a risk of the project: the upstream projects need to care about and work on more synchronized releases. Cloudera managed to address that problem by patching their forks of the apache projects, then push back the patches by way of JIRA patches. Bigtop should not get into the game of having its own branches of any of the ASF projects, instead it should focus on getting its changes into trunk of the various projects SVN trees, and on having near-synchronized releases of the various projects That requires -engagement from the committer teams to get patches in. Having committers from the relevant projects in the BigTop project would address this. -support from the PMCs to do releases on a regular schedule. That's hard, but something like a six-monthly schedule for the big projects with the up and coming projects releasing in a proper factor of that timescale (3, 2, 1 month), could help. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
On 14/06/11 05:26, Tom White wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. Here's a link to the proposal on the wiki http://wiki.apache.org/incubator/BigtopProposal I've also included the initial contents below. Cheers, Tom I've added my name to the committer list, I won't be working on this in much/any of work time, and am fairly overcommitted, so don't expect that much. I can contribute some of my experience in VM setup/teardown for testing RPM installations, and how to do functional testing of dynamically created Hadoop clusters. I am going to add my name to the list of the committers too. Considering my other commitments I might not be able to work much on this project, but I guess the fact that I have wrote like 50% of the underlying system framework might count for something. Cos - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
On Tue, Jun 14, 2011 at 11:43 AM, Konstantin Boudnik c...@apache.org wrote: On 14/06/11 05:26, Tom White wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. Here's a link to the proposal on the wiki http://wiki.apache.org/incubator/BigtopProposal I've also included the initial contents below. Cheers, Tom I've added my name to the committer list, I won't be working on this in much/any of work time, and am fairly overcommitted, so don't expect that much. I can contribute some of my experience in VM setup/teardown for testing RPM installations, and how to do functional testing of dynamically created Hadoop clusters. I am going to add my name to the list of the committers too. Considering my other commitments I might not be able to work much on this project, but I guess the fact that I have wrote like 50% of the underlying system framework might count for something. Welcome aboard Cos! Glad to have you on. Cos has made a ton of contributions to the test frameworks in Bigtop. Looking forward to your contributions! Thanks, Eli - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Bigtop for the Apache Incubator
Cool project. I'd also like to join as a committer so that I can contribute Hama package and continuous improvements to the community. May I join in? On Wed, Jun 15, 2011 at 3:48 AM, Eli Collins e...@cloudera.com wrote: On Tue, Jun 14, 2011 at 11:43 AM, Konstantin Boudnik c...@apache.org wrote: On 14/06/11 05:26, Tom White wrote: Hi, I would like to propose Bigtop to be an Apache Incubator project. Bigtop is a project for the development of packaging and tests of the Hadoop ecosystem. The goal is to do testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects. Here's a link to the proposal on the wiki http://wiki.apache.org/incubator/BigtopProposal I've also included the initial contents below. Cheers, Tom I've added my name to the committer list, I won't be working on this in much/any of work time, and am fairly overcommitted, so don't expect that much. I can contribute some of my experience in VM setup/teardown for testing RPM installations, and how to do functional testing of dynamically created Hadoop clusters. I am going to add my name to the list of the committers too. Considering my other commitments I might not be able to work much on this project, but I guess the fact that I have wrote like 50% of the underlying system framework might count for something. Welcome aboard Cos! Glad to have you on. Cos has made a ton of contributions to the test frameworks in Bigtop. Looking forward to your contributions! Thanks, Eli - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Best Regards, Edward J. Yoon @eddieyoon - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
