Love it! The question is: how can we integrate it into the project CI? Can we perhaps do some weekly or monthly runs to keep an eye on regressions and all?
Shall we link it from our Jenkins somehow? Thoughts? Thanks for doing this, guys! Juju rules! Cos On Wed, Apr 12, 2017 at 02:48AM, Evans Ye wrote: > I'd say great +1 for this. > This extends Bigtop CI from crossing distros to crossing clouds. > I guess more descriptions will make it easier for users to understand if we > put then on the home page, we can discuss the details latter. > Mind to fire up JIRAs? > > 2017-04-07 0:56 GMT+08:00 Kevin Monroe <[email protected]>: > > > Hi Bigtop Devs, > > > > TL;DR: we're testing/benchmarking Bigtop releases across clouds and would > > like to share our findings with the larger community. Read on for the > > details! > > > > When Bigtop/Juju was first introduced [0], I mentioned a cross-cloud > > testing tool that we were developing. During the 1.2 vote last week, Cos > > mentioned the idea of a “weather report” as part of the CI process -- that > > phrase prompted me to let you all know what we’ve been doing on that front. > > > > Cloud Weather Report (CWR) is a mechanism to exercise workloads across > > substrates using Juju charms. It’s important to note that while this is > > tightly coupled with Juju, our charms implement Bigtop best practices, and > > they deploy the standard Bigtop packages unchanged. They set appropriate > > hieradata and call puppet apply just like everyone else. Because of this, > > insights from CWR are valuable to all Bigtop users. > > > > We’ve set up an instance of CWR for Apache Bigtop at > > http://bigtop.charm.qa/. > > Here are a few highlights: > > > > - > > > > Covers multiple Bigtop releases. For example, Hadoop and Spark workload > > reports for each release can be seen below: > > > > > > - > > > > Bigtop-1.1 > > - > > > > http://bigtop.charm.qa/cwr_bundle_hadoop_processing/28/ > > report.html > > - > > > > http://bigtop.charm.qa/cwr_bundle_spark_processing/34/report.html > > > > > > > > - > > > > bigtop-1.2 > > - > > > > > > http://bigtop.charm.qa/cwr_bundle__bigdata_dev_hadoop_ > > processing/18/report.html > > - > > > > > > http://bigtop.charm.qa/cwr_bundle__bigdata_dev_spark_ > > processing/15/report.html > > > > > > - > > > > Exercises Bigtop smoke tests (even those not written yet) > > - > > > > If Bigtop smoke tests are available, we’ll run them (e.g.: Pig, > > Resource Manager, Zookeeper, etc) > > - > > > > If a component does not yet have a Bigtop smoke test, we’ll still > > deploy it and verify things seem ok (e.g.: ps -ef | grep <foo>) > > - > > > > Implicit Bigtop integration tests > > - > > > > Workloads implicitly test the integration of Bigtop components. For > > example, the hadoop-spark workload deploys Hadoop with Spark in > > “yarn-client” mode, and then runs a spark job to verify Spark > > and YARN are > > working together. > > - > > > > Chaos tests > > - > > > > After running the defined tests, CWR will invoke chaotic tests. > > These include things like “shut down a network interface”, “remove a > > datanode”, and “add a new spark master”. After the deployment > > settles, the > > smoke tests are re-run to ensure the workload handled the chaos. > > - > > > > Benchmarks per cloud/release > > - > > > > CWR can run a benchmark as part of its test. For Hadoop- and > > Spark-related workloads, we run Terasort (1GiB) and PageRank > > benchmarks. > > See the above Hadoop/Spark links for graphs of these benchmarks per > > cloud. > > - > > > > These results are helpful when investigating clouds on a > > price/performance basis. > > - > > > > This is also useful to compare perf across releases. For example, > > notice the same SparkPageRank benchmark takes ~50% longer to run > > in Bigtop > > 1.1 versus 1.2 across clouds: > > - > > > > http://bigtop.charm.qa/cwr_bundle_spark_processing/35/report.html > > - > > > > > > http://bigtop.charm.qa/cwr_bundle__bigdata_dev_spark_ > > processing/16/report.html > > > > > > Clouds being what they are, we don’t always get results for every cloud on > > every run -- mostly because of network timeouts. However, the trend is > > what’s more important to me. We use these results to gate Bigtop charm > > releases to http://jujucharms.com, and I think we’re at a point where this > > info would be useful to others. > > > > I would like to get these cross-cloud functional, integration, and > > performance test results to Bigtop users. The reports are static html that > > we can host in s3, so I'd like to suggest the following: > > > > > > - > > > > Link to (or insert) the reports under the CI section of the home page ( > > http://bigtop.apache.org/) > > - > > > > Add a “Weather report” heading/content under “Nightly builds” on the > > release page (http://bigtop.apache.org/download.html#releases) > > > > > > Thoughts? Feedback on the reports or any missing cross-cloud data-points > > are greatly appreciated -- as well as other ideas on getting this info in > > front of people that would find it useful. > > > > [0] > > http://mail-archives.apache.org/mod_mbox/bigtop-dev/201605.mbox/%3CCAJ-3c% > > 2bhgnscunsp1psxfw%[email protected]%3E > > > > Thanks! > > -Kevin Monroe > >
signature.asc
Description: Digital signature
