OK. Dirk and I talked offline and we came to this conclusion:
We default to "--log detailed-progress,unexpected". For now,
detailed-progress does nothing if --experimental-fully-parallel isn't also
specified. If it is also specified, then it logs directories + dots a la
upstream webkit. In a followup patch, I'll make it print out each shard as
we go for the default case of sharding by directory.
Ojan
On Mon, Dec 14, 2009 at 4:46 PM, Ojan Vafai <o...@chromium.org> wrote:
> +chromium-dev
>
> I don't think having a third mode makes sense. We should have terse
> (default) and verbose (--verbose). One is for running locally and one is for
> the bots. Having the flexibility of the --log option is fine, but we should
> pick a default for terse mode that most people are happy with.
>
> My intuition is that most people running the tests locally want:
> 1. To see unexpected results (passes and failures) as they happen
> 2. To know roughly which tests have run. The directories + dots gives this
> feedback. Just directories is probably sufficient though.
>
> I'd be OK with not printing out unexpected results at the end in terse mode
> and instead printing them out as we go, or making that a --log option.
>
> I think the default should be what you're currently calling third_mode, but
> if you pass in --log "" then it could log as we do today.
>
> As you said though, the current implementation of third_mode doesn't make
> sense when we're sharding by directory. So, for this change, it's fine to
> have (2) above addressed only for --experimental-fully-parallel, but (1)
> should be done for both fully-parallel and directory-sharded.
>
> Ojan
>
> On Mon, Dec 14, 2009 at 4:17 PM, Dirk Pranke <dpra...@chromium.org> wrote:
>
>> Okay, for further clarification ...
>>
>> There are currently (as checked in), two modes of output, the one-line
>> version (default, aka "terse" mode) and --verbose. The patch adds a
>> "third mode", which effectively disables sharding and enables
>> --fully-parallel.
>>
>> Note that it is possible to run sharding or fully-parallel without "third
>> mode".
>>
>> Which of the following are you suggesting (multiple are okay):
>>
>> 1) Do nothing (drop the patch)
>>
>> 2) Submit the patch as is - this adds a "third" mode enabled by a new
>> command line flag which prints directories and dots but does not print
>> unexpected results by default.
>> "third" mode disables sharding. In either "terse" mode or "third"
>> adding "--log failures" will log unexpected failures in real time but
>> not passes.
>>
>> 3) Add another flag for unexpected passes in either mode, e.g. "--log
>> passes"
>>
>> 4) Change "--log failures" to "--log unexpected" so that you get both
>> or neither.
>>
>> 5) Always log unexpected failures in terse mode (no flag needed)
>>
>> 6) Always log unexpected passes in terse mode (no flag needed)
>>
>> 7) Always log unexpected failures in "third" mode
>>
>> 8) Always log unexpected passes in "third" mode
>>
>> 9) Modify the patch so that --experimental-fully-parallel enables
>> "third mode". This means that you get "terse + sharding" or "third
>> mode + fully-parallel"
>>
>> 10) Eliminate "terse" mode, so that we only have two modes - "third"
>> and "verbose"
>>
>> I am fine with (1), (2), (3), (4), (7), and (8). I would prefer (3) or
>> (4), and am happy with (2) with or without (7) and/or (8) even though
>> I won't (often) use it. I could grudgingly accept (5) and (6),
>> although I don't think it's as clean as making logging of unexpected
>> results a separate feature (since we log the unexpected results at the
>> end, I don't think they always need to be in the middle). I don't like
>> (9), and I like (10) even less, since both of these options actually
>> undo much of my previous work and produce test scripts that I want to
>> use less than the versions that exist now.
>>
>> -- Dirk
>>
>> On Mon, Dec 14, 2009 at 3:37 PM, Ojan Vafai <o...@chromium.org> wrote:
>> > On Mon, Dec 14, 2009 at 3:19 PM, Dirk Pranke <dpra...@chromium.org>
>> wrote:
>> >>
>> >> (1) We can leave the output as is (discarding this patch)
>> >> + It is the tersest
>> >> + It works fine regardless of what order tests are executed in, and
>> >> whether or not you are sharding or running in full parallel mode
>> >> + It has the minimum amount of code
>> >> - It loses the incremental "." output that webkit has
>> >> - It loses the directory by directory output that webkit has
>> >> - It loses the printing of unexpected test failures in real time
>> >> that webkit has
>> >>
>> >> (1a) I can add the realtime printout of unexpected failures
>> >> (1b) I can add the realtime printout of unexpected passes
>> >
>> > At least printing out unexpected failures in real-time seems to me to be
>> a
>> > non-negotiable requirement. I think we may as well printout unexpected
>> > passes. It's useful information when running the tests.
>> >
>> >>
>> >> (2) We can take the patch as is (with the minor cleanup Ojan
>> >> suggested, of course)
>> >> + It adds printout of unexpected failures
>> >> + It adds the "." output
>> >> + It adds the directory by directory output
>> >> + This is as webkit-compatible as we can get
>> >> - It loses sharding (I believe you can't do sharding and print
>> >> directory-by-directory completion or "."s in any meaningful way; I
>> >> don't know what the output would have to look like since the
>> >> interleaving would be all over the place)
>> >> - It introduces additional complexity (a manageable amount)
>> >>
>> >> (2a) I can add the realtime printout of unexpected passes as well
>> >
>> > I think this is fine. I just think we should tie this to
>> > --experimental-fully-parallel instead of the webkit-log commandline
>> option
>> > (which I think should not exist).
>> >
>> >>
>> >> (3) We can keep sharding, and print directory-by-directory completion
>> >> + it keeps sharding, which may be the most efficient and reliable
>> >> way to run the tests
>> >> + it keeps directory completion
>> >> - we can't print the "." output since there's no sensible ordering
>> >> of the parallel streams (as explained above)
>> >
>> > Agreed.
>> >
>> >>
>> >> - it is not that compatible with the webkit output, since the
>> >> directories will be completing out of order.
>> >
>> > This seems fine.
>> >
>> >>
>> >> - printing out realtime failures or passes gets a little weird since
>> >> that'll be interleaved with directory completion as well
>> >
>> > This doesn't seem like a big deal to me and is what we do today in
>> verbose
>> > mode.
>> >
>> >>
>> >> - due to the way the output is currently structured, there is no
>> >> easy way to keep track of the shards or the progress through each
>> >> shard or directory on the output thread. I think adding this will add
>> >> a fair amount of complexity.
>> >>
>> >> (3a) we can keep the "# remaining" meter as well
>> >
>> > I agree. I don't think keeping track of the progress per-shard is worth
>> the
>> > development effort. Just printing out when we start/finish each shard
>> and
>> > the #remaining is sufficient.
>> >
>> >>
>> >> Obviously, we can, given an arbitrary amount of time and space,
>> >> implement all of the above, at increased maintenance cost and at a
>> >> questionable return on investment. Pragmatically, I don't think we
>> >> want to support that many different options, and at some point we're
>> >> bikeshedding on individual preference.
>> >
>> > Agreed.
>> >>
>> >> Ojan, I'm also not sure that I understood what you were trying to say
>> >> with "I'd like to see the distinction be between sharded vs. fully
>> >> parallelized though rather than webkit-style vs. chromium-style. And
>> >> for the sharded version to print out each shard as we start and finish
>> >> it". Did I capture what you were looking for in the options above, or
>> >> are you looking for even more output.
>> >
>> > All I'm really saying is that option #2 makes sense for
>> > --experimental-fully-parallel and option #3 makes sense for the sharded
>> > case. I think it makes sense to do both of those and not have a specific
>> > webkit style output that noone actually uses.
>> >>
>> >> Bear in mind that we should tune the output to the people that
>> >> actually run the tests and look at the output, rather than the people
>> >> that maintain the tests. I don't think anybody but us cares about how
>> >> things are sharded or when they start and complete, and so I'm a bit
>> >> loathe for that to be anything other than debug/--verbose information.
>> >
>> > That's actually not my experience fixing bugs. When I fix an editing
>> bug, I
>> > want to run the whole test suite, but I pay extra attention to the
>> editing
>> > directory as it's running. It's very useful to me to see which parts of
>> the
>> > test suite have run.
>> >
>> >>
>> >> So, given all that, I'd vote for and of the (1) or (2) variants, but
>> >> not do (3). Can we do one of those for now and, if you guys want
>> >> additional stuff, have one of you do the work to see if you can get
>> >> the output the way you want it? Ojan, maybe you can see a way to
>> >> implement sharded output and directory completion that's easier than
>> >> what I'm seeing ...
>> >
>> > I'd say go ahead with finishing #2. I can do #3 later.
>> > Ojan
>>
>
>
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
