[Lldb-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

[Nuno Lopes via Phabricator via lldb-commits]

nlopes added a comment.

This patch makes the order of the list of failing tests non-deterministic. This 
is extremely annoying because you can't do a simple diff between test dumps 
anymore.
Before the list of failed tests used to be sorted.

Can we revert to the previous behavior please? The current behavior is not user 
friendly. Thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

[Nuno Lopes via Phabricator via lldb-commits]

nlopes added a comment.

In D98179#2638801 , @mehdi_amini wrote:

>> Can we revert to the previous behavior please? The current behavior is not 
>> user friendly. Thanks!
>
> To clarify: you care about the order in the final summary, not the actual 
> execution order, right? (the goal of this patch is the latter, if it changes 
> the former this is just a side-effect I believe)

I don't love that the default test order was changed. For our use case that 
isn't good, but I can just delete the `.lit_test_times.txt` file and get the 
old fixed order. (the reason why having a fixed order is good is because of 
timeouts. Also, slow tests often consume more memory in our setting, so it's 
not good to run all the slow tests at the same time).
So I'm just referring to the final summary, yes.

> How are you diffing? Copy-pasting the terminal output to a file? (I alway 
> pipe these kind of list through `sort` before diffing them)

No, I'm running lit and dumping into a file. sorting is not as easy as piping 
through `sort`, as I'm running with -vv (required). Anyway, sorting the list on 
lit side should be 1 line for some that knows where that line goes :)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

[Nuno Lopes via Phabricator via lldb-commits]

nlopes added a comment.

In D98179#2639565 , @davezarzycki 
wrote:

> In D98179#2639554 , @nlopes wrote:
>
>> In D98179#2639491 , @davezarzycki 
>> wrote:
>>
>>> In D98179#2639476 , @nlopes wrote:
>>>
 Why are timeouts important? Our use case is running Alive2 with the test 
 suite. Alive2 is heavy machinery and runs into timeouts. Running the tests 
 in roughly the same order every time is important to avoid timeout tests 
 flipping to failed or vice-versa. Plus slow tests = tests that consume a 
 lot of memory (in our scenario), so we can't bundle slow tests together.
 Adding a `--ignore-timing-data` would be great, yes! But I still feel that 
 sorting the list of failed tests is important for user experience. I diff 
 these all the time.
>>>
>>> That still sounds incredibly brittle. If there is any variety in test 
>>> machine performance, then any and all attempts at sorting should be futile 
>>> because the underlying hardware will perturb different timeouts. Is this 
>>> not your experience? How do you reconcile hardware performance and 
>>> configuration details (like SMT) with timeout settings?
>>
>> Of course it's brittle :)  Changing from a time-based setting to a 
>> ticks-based system is ongoing work, such that resource exhaustion becomes 
>> deterministic.
>> Nevertheless, on a same machine, we don't see many test flips. It's quite 
>> stable most of the times (just one test flip once in a while).
>
> This seems really beyond the scope and purpose of sorting the tests.
>
> If you don't mind and given that the workaround is trivial (delete the timing 
> data), I'd like to hold off on adding `--ignore-timing-data`. If enough 
> people complain then we can add that option. Is that okay with you?

Sounds good. Thanks for adding support for the summary sort!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

[Nuno Lopes via Phabricator via lldb-commits]

nlopes added a comment.

I'm talking about sorting just the summary of failed tests, not the whole 
output. We need the whole -vv output, but that can be out of order.

Why are timeouts important? Our use case is running Alive2 with the test suite. 
Alive2 is heavy machinery and runs into timeouts. Running the tests in roughly 
the same order every time is important to avoid timeout tests flipping to 
failed or vice-versa. Plus slow tests = tests that consume a lot of memory (in 
our scenario), so we can't bundle slow tests together.
Adding a `--ignore-timing-data` would be great, yes! But I still feel that 
sorting the list of failed tests is important for user experience. I diff these 
all the time.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

[Nuno Lopes via Phabricator via lldb-commits]

nlopes added a comment.

In D98179#2639491 , @davezarzycki 
wrote:

> In D98179#2639476 , @nlopes wrote:
>
>> Why are timeouts important? Our use case is running Alive2 with the test 
>> suite. Alive2 is heavy machinery and runs into timeouts. Running the tests 
>> in roughly the same order every time is important to avoid timeout tests 
>> flipping to failed or vice-versa. Plus slow tests = tests that consume a lot 
>> of memory (in our scenario), so we can't bundle slow tests together.
>> Adding a `--ignore-timing-data` would be great, yes! But I still feel that 
>> sorting the list of failed tests is important for user experience. I diff 
>> these all the time.
>
> That still sounds incredibly brittle. If there is any variety in test machine 
> performance, then any and all attempts at sorting should be futile because 
> the underlying hardware will perturb different timeouts. Is this not your 
> experience? How do you reconcile hardware performance and configuration 
> details (like SMT) with timeout settings?

Of course it's brittle :)  Changing from a time-based setting to a ticks-based 
system is ongoing work, such that resource exhaustion becomes deterministic.
Nevertheless, on a same machine, we don't see many test flips. It's quite 
stable most of the times (just one test flip once in a while).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits