+chromium-dev
I don't think having a third mode makes sense. We should have terse
(default) and verbose (--verbose). One is for running locally and one is for
the bots. Having the flexibility of the --log option is fine, but we should
pick a default for terse mode that most people are happy with.
My intuition is that most people running the tests locally want:
1. To see unexpected results (passes and failures) as they happen
2. To know roughly which tests have run. The directories + dots gives this
feedback. Just directories is probably sufficient though.
I'd be OK with not printing out unexpected results at the end in terse mode
and instead printing them out as we go, or making that a --log option.
I think the default should be what you're currently calling third_mode, but
if you pass in --log "" then it could log as we do today.
As you said though, the current implementation of third_mode doesn't make
sense when we're sharding by directory. So, for this change, it's fine to
have (2) above addressed only for --experimental-fully-parallel, but (1)
should be done for both fully-parallel and directory-sharded.
Ojan
On Mon, Dec 14, 2009 at 4:17 PM, Dirk Pranke <dpra...@chromium.org> wrote:
> Okay, for further clarification ...
>
> There are currently (as checked in), two modes of output, the one-line
> version (default, aka "terse" mode) and --verbose. The patch adds a
> "third mode", which effectively disables sharding and enables
> --fully-parallel.
>
> Note that it is possible to run sharding or fully-parallel without "third
> mode".
>
> Which of the following are you suggesting (multiple are okay):
>
> 1) Do nothing (drop the patch)
>
> 2) Submit the patch as is - this adds a "third" mode enabled by a new
> command line flag which prints directories and dots but does not print
> unexpected results by default.
> "third" mode disables sharding. In either "terse" mode or "third"
> adding "--log failures" will log unexpected failures in real time but
> not passes.
>
> 3) Add another flag for unexpected passes in either mode, e.g. "--log
> passes"
>
> 4) Change "--log failures" to "--log unexpected" so that you get both
> or neither.
>
> 5) Always log unexpected failures in terse mode (no flag needed)
>
> 6) Always log unexpected passes in terse mode (no flag needed)
>
> 7) Always log unexpected failures in "third" mode
>
> 8) Always log unexpected passes in "third" mode
>
> 9) Modify the patch so that --experimental-fully-parallel enables
> "third mode". This means that you get "terse + sharding" or "third
> mode + fully-parallel"
>
> 10) Eliminate "terse" mode, so that we only have two modes - "third"
> and "verbose"
>
> I am fine with (1), (2), (3), (4), (7), and (8). I would prefer (3) or
> (4), and am happy with (2) with or without (7) and/or (8) even though
> I won't (often) use it. I could grudgingly accept (5) and (6),
> although I don't think it's as clean as making logging of unexpected
> results a separate feature (since we log the unexpected results at the
> end, I don't think they always need to be in the middle). I don't like
> (9), and I like (10) even less, since both of these options actually
> undo much of my previous work and produce test scripts that I want to
> use less than the versions that exist now.
>
> -- Dirk
>
> On Mon, Dec 14, 2009 at 3:37 PM, Ojan Vafai <o...@chromium.org> wrote:
> > On Mon, Dec 14, 2009 at 3:19 PM, Dirk Pranke <dpra...@chromium.org>
> wrote:
> >>
> >> (1) We can leave the output as is (discarding this patch)
> >> + It is the tersest
> >> + It works fine regardless of what order tests are executed in, and
> >> whether or not you are sharding or running in full parallel mode
> >> + It has the minimum amount of code
> >> - It loses the incremental "." output that webkit has
> >> - It loses the directory by directory output that webkit has
> >> - It loses the printing of unexpected test failures in real time
> >> that webkit has
> >>
> >> (1a) I can add the realtime printout of unexpected failures
> >> (1b) I can add the realtime printout of unexpected passes
> >
> > At least printing out unexpected failures in real-time seems to me to be
> a
> > non-negotiable requirement. I think we may as well printout unexpected
> > passes. It's useful information when running the tests.
> >
> >>
> >> (2) We can take the patch as is (with the minor cleanup Ojan
> >> suggested, of course)
> >> + It adds printout of unexpected failures
> >> + It adds the "." output
> >> + It adds the directory by directory output
> >> + This is as webkit-compatible as we can get
> >> - It loses sharding (I believe you can't do sharding and print
> >> directory-by-directory completion or "."s in any meaningful way; I
> >> don't know what the output would have to look like since the
> >> interleaving would be all over the place)
> >> - It introduces additional complexity (a manageable amount)
> >>
> >> (2a) I can add the realtime printout of unexpected passes as well
> >
> > I think this is fine. I just think we should tie this to
> > --experimental-fully-parallel instead of the webkit-log commandline
> option
> > (which I think should not exist).
> >
> >>
> >> (3) We can keep sharding, and print directory-by-directory completion
> >> + it keeps sharding, which may be the most efficient and reliable
> >> way to run the tests
> >> + it keeps directory completion
> >> - we can't print the "." output since there's no sensible ordering
> >> of the parallel streams (as explained above)
> >
> > Agreed.
> >
> >>
> >> - it is not that compatible with the webkit output, since the
> >> directories will be completing out of order.
> >
> > This seems fine.
> >
> >>
> >> - printing out realtime failures or passes gets a little weird since
> >> that'll be interleaved with directory completion as well
> >
> > This doesn't seem like a big deal to me and is what we do today in
> verbose
> > mode.
> >
> >>
> >> - due to the way the output is currently structured, there is no
> >> easy way to keep track of the shards or the progress through each
> >> shard or directory on the output thread. I think adding this will add
> >> a fair amount of complexity.
> >>
> >> (3a) we can keep the "# remaining" meter as well
> >
> > I agree. I don't think keeping track of the progress per-shard is worth
> the
> > development effort. Just printing out when we start/finish each shard and
> > the #remaining is sufficient.
> >
> >>
> >> Obviously, we can, given an arbitrary amount of time and space,
> >> implement all of the above, at increased maintenance cost and at a
> >> questionable return on investment. Pragmatically, I don't think we
> >> want to support that many different options, and at some point we're
> >> bikeshedding on individual preference.
> >
> > Agreed.
> >>
> >> Ojan, I'm also not sure that I understood what you were trying to say
> >> with "I'd like to see the distinction be between sharded vs. fully
> >> parallelized though rather than webkit-style vs. chromium-style. And
> >> for the sharded version to print out each shard as we start and finish
> >> it". Did I capture what you were looking for in the options above, or
> >> are you looking for even more output.
> >
> > All I'm really saying is that option #2 makes sense for
> > --experimental-fully-parallel and option #3 makes sense for the sharded
> > case. I think it makes sense to do both of those and not have a specific
> > webkit style output that noone actually uses.
> >>
> >> Bear in mind that we should tune the output to the people that
> >> actually run the tests and look at the output, rather than the people
> >> that maintain the tests. I don't think anybody but us cares about how
> >> things are sharded or when they start and complete, and so I'm a bit
> >> loathe for that to be anything other than debug/--verbose information.
> >
> > That's actually not my experience fixing bugs. When I fix an editing bug,
> I
> > want to run the whole test suite, but I pay extra attention to the
> editing
> > directory as it's running. It's very useful to me to see which parts of
> the
> > test suite have run.
> >
> >>
> >> So, given all that, I'd vote for and of the (1) or (2) variants, but
> >> not do (3). Can we do one of those for now and, if you guys want
> >> additional stuff, have one of you do the work to see if you can get
> >> the output the way you want it? Ojan, maybe you can see a way to
> >> implement sharded output and directory completion that's easier than
> >> what I'm seeing ...
> >
> > I'd say go ahead with finishing #2. I can do #3 later.
> > Ojan
>
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
