Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true.
Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > Not completely random places :) > People may use Cloudera / HWX distributions which include Kafka, but want > to verify that these bits match a specific upstream release. > > I think having the tests separately will be useful for this. In this case, > finding the tests are not a big issue - we'll add a download link :) > > On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > > > Hey Gwen, > > > > Currently the test and code are downloaded at the same time. Supposedly > > the tests in the same repository should cover match the code. > > Are you saying people downloaded a release from some random place and > want > > to verify it? If that is the case, does that mean people still need to > > find the correct place to download the right test artifact? > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > > > On 6/4/15, 4:29 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > > > > >Hi, > > > > > >Reviving the discussion a bit :) > > > > > >I think it will be nice if each Kafka version that we release will also > > >have a separate "tests" artifact that users can download, untar and > easily > > >run against a Kafka cluster of the same version. > > > > > >The idea is that if someone downloads packages that claim to contain > > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), > > >users > > >can easily download the tests and verify that it indeed passes the tests > > >for this version and therefore behaves the way this version is expected > to > > >behave. > > > > > >Does it make sense? > > > > > >Gwen > > > > > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <ge...@confluent.io> > > >wrote: > > > > > >> Hi Ashish, > > >> > > >> Looks like Ewen already hit the main points, but a few additions: > > >> > > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape > > >> ducktape itself will be pip installable in the near future, and Kafka > > >> system tests will be able to depend on a particular version of > ducktape. > > >> > > >> 2. The reporting is nothing fancy. We're definitely open to feedback, > > >>but > > >> it consists of: > > >> - top level summary of the test run (simple PASS/FAIL for each test) > > >> - top level info and debug logs > > >> - per-test info and debug logs > > >> - per-test "service" logs gathered from each service used in the test. > > >>For > > >> example, if your test pulls up a Kafka cluster with 5 brokers, the end > > >> result will have the Kafka logs from each of those 5 machines. > > >> > > >> Cheers, > > >> Geoff > > >> > > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava > > >><e...@confluent.io> > > >> wrote: > > >> > > >> > Ashish, > > >> > > > >> > 1. That was the plan. We put some effort into cleanly separating the > > >> > framework so it would be reusable across many projects. > > >> > 2. I think you're seeing a test in progress where the final report > > >>hasn't > > >> > been created yet. If you visit one of the older ones you'll see it > > >>has a > > >> > landing page with links: > > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/ > > >> Apparently > > >> > we need to adjust when we update the 'latest' symlink. The logs that > > >>are > > >> > collected for tests are configurable, and service implementations > > >>include > > >> > sane defaults (so, e.g., you will always get the normal log file for > > >> Kafka, > > >> > but only get the data files if the test asks for them). > > >> > 3. No code coverage support. Haven't looked into it, so I couldn't > > >> comment > > >> > on how hard it would be to add. > > >> > > > >> > -Ewen > > >> > > > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <asi...@cloudera.com> > > >> wrote: > > >> > > > >> > > Geoffrey, > > >> > > > > >> > > This looks great! > > >> > > > > >> > > A few questions. > > >> > > 1. Will ducktape be maintained separately as a github repo? > > >> > > 2. How easy is viewing the test results and logs. The link in KIP, > > >> > > http://testing.confluent.io/confluent_platform/latest/, lists a > > >>bunch > > >> of > > >> > > files and dirs. Could you add to KIP how the result and logs for > the > > >> > tests > > >> > > will be organized. > > >> > > 3. Does it support code coverage? If not, how easy/ difficult > would > > >>it > > >> > be? > > >> > > > > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson > > >><ge...@confluent.io > > >> > > > >> > > wrote: > > >> > > > > >> > > > Great, I'll work on putting together a more detailed map of this > > >> > > > replacement process. > > >> > > > > > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira < > > >> gshap...@cloudera.com> > > >> > > > wrote: > > >> > > > > > >> > > > > Love this idea :) > > >> > > > > > > >> > > > > I took a look at Ducktape API and it looks like a good fit - > > >>clean > > >> > API, > > >> > > > > extensible, easy to use and powerful enough for our use-case. > > >> > > > > > > >> > > > > Something I'd like to see as part of the KIP is a map of what > > >> > > system-test > > >> > > > > currently tests, which ones we want to replace and a JIRA for > > >> > replacing > > >> > > > > (possibly one for each group of tests). > > >> > > > > Basically, I know we all want to use the new system for new > test > > >> > cases > > >> > > > > (upgrades, etc), but I really want to make sure we don't get > > >>stuck > > >> > with > > >> > > > > both systems forever. > > >> > > > > > > >> > > > > Gwen > > >> > > > > > > >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson < > > >> > ge...@confluent.io > > >> > > > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > Hi, > > >> > > > > > > > >> > > > > > Just kicking off the discussion thread on KIP-25 > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im > > >>provements > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Geoff > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > -- > > >> > > > > >> > > Regards, > > >> > > Ashish > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Thanks, > > >> > Ewen > > >> > > > >> > > > > >
