Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the correct place to download the right test artifact?
Thanks, Jiangjie (Becket) Qin On 6/4/15, 4:29 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote: >Hi, > >Reviving the discussion a bit :) > >I think it will be nice if each Kafka version that we release will also >have a separate "tests" artifact that users can download, untar and easily >run against a Kafka cluster of the same version. > >The idea is that if someone downloads packages that claim to contain >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), >users >can easily download the tests and verify that it indeed passes the tests >for this version and therefore behaves the way this version is expected to >behave. > >Does it make sense? > >Gwen > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <ge...@confluent.io> >wrote: > >> Hi Ashish, >> >> Looks like Ewen already hit the main points, but a few additions: >> >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape >> ducktape itself will be pip installable in the near future, and Kafka >> system tests will be able to depend on a particular version of ducktape. >> >> 2. The reporting is nothing fancy. We're definitely open to feedback, >>but >> it consists of: >> - top level summary of the test run (simple PASS/FAIL for each test) >> - top level info and debug logs >> - per-test info and debug logs >> - per-test "service" logs gathered from each service used in the test. >>For >> example, if your test pulls up a Kafka cluster with 5 brokers, the end >> result will have the Kafka logs from each of those 5 machines. >> >> Cheers, >> Geoff >> >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava >><e...@confluent.io> >> wrote: >> >> > Ashish, >> > >> > 1. That was the plan. We put some effort into cleanly separating the >> > framework so it would be reusable across many projects. >> > 2. I think you're seeing a test in progress where the final report >>hasn't >> > been created yet. If you visit one of the older ones you'll see it >>has a >> > landing page with links: >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/ >> Apparently >> > we need to adjust when we update the 'latest' symlink. The logs that >>are >> > collected for tests are configurable, and service implementations >>include >> > sane defaults (so, e.g., you will always get the normal log file for >> Kafka, >> > but only get the data files if the test asks for them). >> > 3. No code coverage support. Haven't looked into it, so I couldn't >> comment >> > on how hard it would be to add. >> > >> > -Ewen >> > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <asi...@cloudera.com> >> wrote: >> > >> > > Geoffrey, >> > > >> > > This looks great! >> > > >> > > A few questions. >> > > 1. Will ducktape be maintained separately as a github repo? >> > > 2. How easy is viewing the test results and logs. The link in KIP, >> > > http://testing.confluent.io/confluent_platform/latest/, lists a >>bunch >> of >> > > files and dirs. Could you add to KIP how the result and logs for the >> > tests >> > > will be organized. >> > > 3. Does it support code coverage? If not, how easy/ difficult would >>it >> > be? >> > > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson >><ge...@confluent.io >> > >> > > wrote: >> > > >> > > > Great, I'll work on putting together a more detailed map of this >> > > > replacement process. >> > > > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira < >> gshap...@cloudera.com> >> > > > wrote: >> > > > >> > > > > Love this idea :) >> > > > > >> > > > > I took a look at Ducktape API and it looks like a good fit - >>clean >> > API, >> > > > > extensible, easy to use and powerful enough for our use-case. >> > > > > >> > > > > Something I'd like to see as part of the KIP is a map of what >> > > system-test >> > > > > currently tests, which ones we want to replace and a JIRA for >> > replacing >> > > > > (possibly one for each group of tests). >> > > > > Basically, I know we all want to use the new system for new test >> > cases >> > > > > (upgrades, etc), but I really want to make sure we don't get >>stuck >> > with >> > > > > both systems forever. >> > > > > >> > > > > Gwen >> > > > > >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson < >> > ge...@confluent.io >> > > > >> > > > > wrote: >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Just kicking off the discussion thread on KIP-25 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im >>provements >> > > > > > >> > > > > > Thanks, >> > > > > > Geoff >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > >> > > Regards, >> > > Ashish >> > > >> > >> > >> > >> > -- >> > Thanks, >> > Ewen >> > >>
