Re: [webkit-dev] Switching open source Debug bots to building and testing with configuration --force-opt=O3

[Geoffrey Garen]

OK.

Geooff

> On Jun 24, 2020, at 12:19 PM, Saam Barati  wrote:
> 
> If we're only doing this for JSC, I don't think we need this, as none of us 
> care.
> 
> If we want to do it for all of WK testing, we should include this for folks 
> who care about having O0-style stack traces.
> 
> - Saam
> 
>> On Jun 24, 2020, at 12:17 PM, Geoffrey Garen > > wrote:
>> 
>> Is "-fno-inline -fno-optimize-sibling-calls” still on the table?
>> 
>> Thanks,
>> Geoff
>> 
>>> On Jun 24, 2020, at 10:28 AM, Mark Lam >> > wrote:
>>> 
>>> I forgot to add ...
>>> 
 On Jun 24, 2020, at 10:26 AM, Mark Lam >>> > wrote:
 
 Based on all the feedback given so far, it looks like we can move forward 
 with the following plan:
 1. JSC Debug test bots will build their own local jsc with O3 before 
 running the tests.
>>> 
>>> 1.5 JSC EWS bot will also run with an O3 Debug build.
>>> 
>>> Mark
>>> 
 2. The rest of the build and test bots will remain unchanged.
 
 Let's move forward with this and get the Debug JSC test bot functional 
 again.
 
 Thanks.
 
 Mark
 
 
> On Jun 19, 2020, at 2:58 PM, Alexey Proskuryakov  > wrote:
> 
> 
> 
>> 19 июня 2020 г., в 1:11 PM, Mark Lam > > написал(а):
>> 
>> 
>> 
>>> On Jun 19, 2020, at 10:24 AM, Geoffrey Garen >> > wrote:
>>> 
> On Jun 19, 2020, at 8:04 AM, Geoffrey Garen  > wrote:
> 
> Can you explain more about what "O3 with no-inlining” is? How does 
> --force-opt=O3 avoid inlining? Would this fully resolve Simon concern 
> about stack traces, or would something still be different about stack 
> traces?
 There doesn't exist a way to do this now, but it'd be trivial to add a 
 way. I won't claim it fixes all stack traces differences, but I'd 
 think compiling using "-fno-inline -fno-optimize-sibling-calls" would 
 get us pretty far in crashing stack traces being similar enough.
>>> 
>>> Sounds good.
>>> 
>>> I think we should try to refine the proposal along these lines, to 
>>> minimize the downsides. I won’t speak for Simon, but for me, being able 
>>> to ensure a clear backtrace from a bot would be a big improvement.
>>> 
> And again, I think this discussion would get a lot more focused if 
> the change could apply only to JSC code, and not also to WTF code.
 I believe Mark's proposal, initially, is just to make JSC do this. So 
 I don't see the point of compiling WTF differently. JSC can kick off 
 its own build, and run Debug+O3 tests faster than it can run Debug+O0 
 tests. Given people working on JSC want this, and people working on 
 JSC defend these tests, and that these test results are more stable 
 (see below), we should make this change for JSC.
 
 I was trying to convince folks defending non-JSC testing, that they 
 too, should want this. I'm not going to pull teeth here. If folks want 
 their tests to take ~10x longer to run, they're entitled to make that 
 tradeoff.
>>> 
>>> Got it.
>> 
>> I'm of the same mind as Saam.  We want this change for the JSC bots, and 
>> from the time measurements I’ve collected, we can afford to do a clean 
>> build for the JSC Debug test runs using O3, and still come out way ahead.
> 
> This seems like a reasonable plan. You didn't mention what hardware you 
> measured with, but it seems certain to be beneficial on any current 
> hardware.
> 
> I need to mention that we saw unexplained and very large impact on JSC 
> test speed from enabling/disabling TCSM. That may be a good thing to look 
> into while optimizing JSC test speed.
> 
> - Alexey
> 
> 
>> As for non-JSC test runs, I have not actually measured what the time 
>> savings are.  Given there is resistance to going with O3 there, we don’t 
>> have to share the build artifacts for running the tests.  JSC test runs 
>> should be able to just build JSC for each O3 Debug JSC test run and it 
>> is still a win over the current status quo i.e. test runs never complete.
>> 
>> Regarding Geoff’s earlier question about whether I know for sure that 
>> switching to O3 will fix the current Debug JSC bot failures to run 
>> tests, the answer is I’m not certain.  The failure is a timeout due to 
>> the master bot not seeing any output from the tester bot for more than 
>> 20 minutes.  I’ve not been able to reproduce this yet.  But with a Debug 
>> build test run taking 4+ hours, it can only be a progression to switch 
>> the Debug JSC test bots to O3.
>> 
>> Mark
>> 
>

Re: [webkit-dev] Switching open source Debug bots to building and testing with configuration --force-opt=O3

[Saam Barati]

If we're only doing this for JSC, I don't think we need this, as none of us 
care.

If we want to do it for all of WK testing, we should include this for folks who 
care about having O0-style stack traces.

- Saam

> On Jun 24, 2020, at 12:17 PM, Geoffrey Garen  wrote:
> 
> Is "-fno-inline -fno-optimize-sibling-calls” still on the table?
> 
> Thanks,
> Geoff
> 
>> On Jun 24, 2020, at 10:28 AM, Mark Lam > > wrote:
>> 
>> I forgot to add ...
>> 
>>> On Jun 24, 2020, at 10:26 AM, Mark Lam >> > wrote:
>>> 
>>> Based on all the feedback given so far, it looks like we can move forward 
>>> with the following plan:
>>> 1. JSC Debug test bots will build their own local jsc with O3 before 
>>> running the tests.
>> 
>> 1.5 JSC EWS bot will also run with an O3 Debug build.
>> 
>> Mark
>> 
>>> 2. The rest of the build and test bots will remain unchanged.
>>> 
>>> Let's move forward with this and get the Debug JSC test bot functional 
>>> again.
>>> 
>>> Thanks.
>>> 
>>> Mark
>>> 
>>> 
 On Jun 19, 2020, at 2:58 PM, Alexey Proskuryakov >>> > wrote:
 
 
 
> 19 июня 2020 г., в 1:11 PM, Mark Lam  > написал(а):
> 
> 
> 
>> On Jun 19, 2020, at 10:24 AM, Geoffrey Garen > > wrote:
>> 
 On Jun 19, 2020, at 8:04 AM, Geoffrey Garen >>> > wrote:
 
 Can you explain more about what "O3 with no-inlining” is? How does 
 --force-opt=O3 avoid inlining? Would this fully resolve Simon concern 
 about stack traces, or would something still be different about stack 
 traces?
>>> There doesn't exist a way to do this now, but it'd be trivial to add a 
>>> way. I won't claim it fixes all stack traces differences, but I'd think 
>>> compiling using "-fno-inline -fno-optimize-sibling-calls" would get us 
>>> pretty far in crashing stack traces being similar enough.
>> 
>> Sounds good.
>> 
>> I think we should try to refine the proposal along these lines, to 
>> minimize the downsides. I won’t speak for Simon, but for me, being able 
>> to ensure a clear backtrace from a bot would be a big improvement.
>> 
 And again, I think this discussion would get a lot more focused if the 
 change could apply only to JSC code, and not also to WTF code.
>>> I believe Mark's proposal, initially, is just to make JSC do this. So I 
>>> don't see the point of compiling WTF differently. JSC can kick off its 
>>> own build, and run Debug+O3 tests faster than it can run Debug+O0 
>>> tests. Given people working on JSC want this, and people working on JSC 
>>> defend these tests, and that these test results are more stable (see 
>>> below), we should make this change for JSC.
>>> 
>>> I was trying to convince folks defending non-JSC testing, that they 
>>> too, should want this. I'm not going to pull teeth here. If folks want 
>>> their tests to take ~10x longer to run, they're entitled to make that 
>>> tradeoff.
>> 
>> Got it.
> 
> I'm of the same mind as Saam.  We want this change for the JSC bots, and 
> from the time measurements I’ve collected, we can afford to do a clean 
> build for the JSC Debug test runs using O3, and still come out way ahead.
 
 This seems like a reasonable plan. You didn't mention what hardware you 
 measured with, but it seems certain to be beneficial on any current 
 hardware.
 
 I need to mention that we saw unexplained and very large impact on JSC 
 test speed from enabling/disabling TCSM. That may be a good thing to look 
 into while optimizing JSC test speed.
 
 - Alexey
 
 
> As for non-JSC test runs, I have not actually measured what the time 
> savings are.  Given there is resistance to going with O3 there, we don’t 
> have to share the build artifacts for running the tests.  JSC test runs 
> should be able to just build JSC for each O3 Debug JSC test run and it is 
> still a win over the current status quo i.e. test runs never complete.
> 
> Regarding Geoff’s earlier question about whether I know for sure that 
> switching to O3 will fix the current Debug JSC bot failures to run tests, 
> the answer is I’m not certain.  The failure is a timeout due to the 
> master bot not seeing any output from the tester bot for more than 20 
> minutes.  I’ve not been able to reproduce this yet.  But with a Debug 
> build test run taking 4+ hours, it can only be a progression to switch 
> the Debug JSC test bots to O3.
> 
> Mark
> 
> 
>> 
 And again, on the run more tests front, it would be helpful to know 
 whether this change was enough to get the stress tests running or not.
>>> My experience running the tests locally supports

Re: [webkit-dev] Switching open source Debug bots to building and testing with configuration --force-opt=O3

[Geoffrey Garen]

Is "-fno-inline -fno-optimize-sibling-calls” still on the table?

Thanks,
Geoff

> On Jun 24, 2020, at 10:28 AM, Mark Lam  wrote:
> 
> I forgot to add ...
> 
>> On Jun 24, 2020, at 10:26 AM, Mark Lam > > wrote:
>> 
>> Based on all the feedback given so far, it looks like we can move forward 
>> with the following plan:
>> 1. JSC Debug test bots will build their own local jsc with O3 before running 
>> the tests.
> 
> 1.5 JSC EWS bot will also run with an O3 Debug build.
> 
> Mark
> 
>> 2. The rest of the build and test bots will remain unchanged.
>> 
>> Let's move forward with this and get the Debug JSC test bot functional again.
>> 
>> Thanks.
>> 
>> Mark
>> 
>> 
>>> On Jun 19, 2020, at 2:58 PM, Alexey Proskuryakov >> > wrote:
>>> 
>>> 
>>> 
 19 июня 2020 г., в 1:11 PM, Mark Lam >>> > написал(а):
 
 
 
> On Jun 19, 2020, at 10:24 AM, Geoffrey Garen  > wrote:
> 
>>> On Jun 19, 2020, at 8:04 AM, Geoffrey Garen >> > wrote:
>>> 
>>> Can you explain more about what "O3 with no-inlining” is? How does 
>>> --force-opt=O3 avoid inlining? Would this fully resolve Simon concern 
>>> about stack traces, or would something still be different about stack 
>>> traces?
>> There doesn't exist a way to do this now, but it'd be trivial to add a 
>> way. I won't claim it fixes all stack traces differences, but I'd think 
>> compiling using "-fno-inline -fno-optimize-sibling-calls" would get us 
>> pretty far in crashing stack traces being similar enough.
> 
> Sounds good.
> 
> I think we should try to refine the proposal along these lines, to 
> minimize the downsides. I won’t speak for Simon, but for me, being able 
> to ensure a clear backtrace from a bot would be a big improvement.
> 
>>> And again, I think this discussion would get a lot more focused if the 
>>> change could apply only to JSC code, and not also to WTF code.
>> I believe Mark's proposal, initially, is just to make JSC do this. So I 
>> don't see the point of compiling WTF differently. JSC can kick off its 
>> own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. 
>> Given people working on JSC want this, and people working on JSC defend 
>> these tests, and that these test results are more stable (see below), we 
>> should make this change for JSC.
>> 
>> I was trying to convince folks defending non-JSC testing, that they too, 
>> should want this. I'm not going to pull teeth here. If folks want their 
>> tests to take ~10x longer to run, they're entitled to make that tradeoff.
> 
> Got it.
 
 I'm of the same mind as Saam.  We want this change for the JSC bots, and 
 from the time measurements I’ve collected, we can afford to do a clean 
 build for the JSC Debug test runs using O3, and still come out way ahead.
>>> 
>>> This seems like a reasonable plan. You didn't mention what hardware you 
>>> measured with, but it seems certain to be beneficial on any current 
>>> hardware.
>>> 
>>> I need to mention that we saw unexplained and very large impact on JSC test 
>>> speed from enabling/disabling TCSM. That may be a good thing to look into 
>>> while optimizing JSC test speed.
>>> 
>>> - Alexey
>>> 
>>> 
 As for non-JSC test runs, I have not actually measured what the time 
 savings are.  Given there is resistance to going with O3 there, we don’t 
 have to share the build artifacts for running the tests.  JSC test runs 
 should be able to just build JSC for each O3 Debug JSC test run and it is 
 still a win over the current status quo i.e. test runs never complete.
 
 Regarding Geoff’s earlier question about whether I know for sure that 
 switching to O3 will fix the current Debug JSC bot failures to run tests, 
 the answer is I’m not certain.  The failure is a timeout due to the master 
 bot not seeing any output from the tester bot for more than 20 minutes.  
 I’ve not been able to reproduce this yet.  But with a Debug build test run 
 taking 4+ hours, it can only be a progression to switch the Debug JSC test 
 bots to O3.
 
 Mark
 
 
> 
>>> And again, on the run more tests front, it would be helpful to know 
>>> whether this change was enough to get the stress tests running or not.
>> My experience running the tests locally supports this fully. I don't get 
>> timeouts when running O3+Debug locally. When running Debug+O0 locally, 
>> I'd get timeouts all the time, and the total test run would take ~4-8 
>> hours. We can wait for official confirmation from Mark.
> 
> Alexey, do the JSC stress tests run now on bots? If not, how fast would 
> they need to run in order to be eligible to run on bots?
> 
> Thanks,
> Geoff
> 

Re: [webkit-dev] Switching open source Debug bots to building and testing with configuration --force-opt=O3

[Mark Lam]

I forgot to add ...

> On Jun 24, 2020, at 10:26 AM, Mark Lam  wrote:
> 
> Based on all the feedback given so far, it looks like we can move forward 
> with the following plan:
> 1. JSC Debug test bots will build their own local jsc with O3 before running 
> the tests.

1.5 JSC EWS bot will also run with an O3 Debug build.

Mark

> 2. The rest of the build and test bots will remain unchanged.
> 
> Let's move forward with this and get the Debug JSC test bot functional again.
> 
> Thanks.
> 
> Mark
> 
> 
>> On Jun 19, 2020, at 2:58 PM, Alexey Proskuryakov > > wrote:
>> 
>> 
>> 
>>> 19 июня 2020 г., в 1:11 PM, Mark Lam >> > написал(а):
>>> 
>>> 
>>> 
 On Jun 19, 2020, at 10:24 AM, Geoffrey Garen >>> > wrote:
 
>> On Jun 19, 2020, at 8:04 AM, Geoffrey Garen > > wrote:
>> 
>> Can you explain more about what "O3 with no-inlining” is? How does 
>> --force-opt=O3 avoid inlining? Would this fully resolve Simon concern 
>> about stack traces, or would something still be different about stack 
>> traces?
> There doesn't exist a way to do this now, but it'd be trivial to add a 
> way. I won't claim it fixes all stack traces differences, but I'd think 
> compiling using "-fno-inline -fno-optimize-sibling-calls" would get us 
> pretty far in crashing stack traces being similar enough.
 
 Sounds good.
 
 I think we should try to refine the proposal along these lines, to 
 minimize the downsides. I won’t speak for Simon, but for me, being able to 
 ensure a clear backtrace from a bot would be a big improvement.
 
>> And again, I think this discussion would get a lot more focused if the 
>> change could apply only to JSC code, and not also to WTF code.
> I believe Mark's proposal, initially, is just to make JSC do this. So I 
> don't see the point of compiling WTF differently. JSC can kick off its 
> own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. 
> Given people working on JSC want this, and people working on JSC defend 
> these tests, and that these test results are more stable (see below), we 
> should make this change for JSC.
> 
> I was trying to convince folks defending non-JSC testing, that they too, 
> should want this. I'm not going to pull teeth here. If folks want their 
> tests to take ~10x longer to run, they're entitled to make that tradeoff.
 
 Got it.
>>> 
>>> I'm of the same mind as Saam.  We want this change for the JSC bots, and 
>>> from the time measurements I’ve collected, we can afford to do a clean 
>>> build for the JSC Debug test runs using O3, and still come out way ahead.
>> 
>> This seems like a reasonable plan. You didn't mention what hardware you 
>> measured with, but it seems certain to be beneficial on any current hardware.
>> 
>> I need to mention that we saw unexplained and very large impact on JSC test 
>> speed from enabling/disabling TCSM. That may be a good thing to look into 
>> while optimizing JSC test speed.
>> 
>> - Alexey
>> 
>> 
>>> As for non-JSC test runs, I have not actually measured what the time 
>>> savings are.  Given there is resistance to going with O3 there, we don’t 
>>> have to share the build artifacts for running the tests.  JSC test runs 
>>> should be able to just build JSC for each O3 Debug JSC test run and it is 
>>> still a win over the current status quo i.e. test runs never complete.
>>> 
>>> Regarding Geoff’s earlier question about whether I know for sure that 
>>> switching to O3 will fix the current Debug JSC bot failures to run tests, 
>>> the answer is I’m not certain.  The failure is a timeout due to the master 
>>> bot not seeing any output from the tester bot for more than 20 minutes.  
>>> I’ve not been able to reproduce this yet.  But with a Debug build test run 
>>> taking 4+ hours, it can only be a progression to switch the Debug JSC test 
>>> bots to O3.
>>> 
>>> Mark
>>> 
>>> 
 
>> And again, on the run more tests front, it would be helpful to know 
>> whether this change was enough to get the stress tests running or not.
> My experience running the tests locally supports this fully. I don't get 
> timeouts when running O3+Debug locally. When running Debug+O0 locally, 
> I'd get timeouts all the time, and the total test run would take ~4-8 
> hours. We can wait for official confirmation from Mark.
 
 Alexey, do the JSC stress tests run now on bots? If not, how fast would 
 they need to run in order to be eligible to run on bots?
 
 Thanks,
 Geoff
 
> 
> - Saam
> 
>> 
>> Thanks,
>> Geoff
>> 
>>> On Jun 18, 2020, at 9:30 PM, Saam Barati >> > wrote:
>>> 
>>> Why are we insisting on doing something on the bots that takes ~10x 
>>> longer to run than necessary? I

Re: [webkit-dev] Switching open source Debug bots to building and testing with configuration --force-opt=O3

[Mark Lam]

Based on all the feedback given so far, it looks like we can move forward with 
the following plan:
1. JSC Debug test bots will build their own local jsc with O3 before running 
the tests.
2. The rest of the build and test bots will remain unchanged.

Let's move forward with this and get the Debug JSC test bot functional again.

Thanks.

Mark


> On Jun 19, 2020, at 2:58 PM, Alexey Proskuryakov  wrote:
> 
> 
> 
>> 19 июня 2020 г., в 1:11 PM, Mark Lam > > написал(а):
>> 
>> 
>> 
>>> On Jun 19, 2020, at 10:24 AM, Geoffrey Garen >> > wrote:
>>> 
> On Jun 19, 2020, at 8:04 AM, Geoffrey Garen  > wrote:
> 
> Can you explain more about what "O3 with no-inlining” is? How does 
> --force-opt=O3 avoid inlining? Would this fully resolve Simon concern 
> about stack traces, or would something still be different about stack 
> traces?
 There doesn't exist a way to do this now, but it'd be trivial to add a 
 way. I won't claim it fixes all stack traces differences, but I'd think 
 compiling using "-fno-inline -fno-optimize-sibling-calls" would get us 
 pretty far in crashing stack traces being similar enough.
>>> 
>>> Sounds good.
>>> 
>>> I think we should try to refine the proposal along these lines, to minimize 
>>> the downsides. I won’t speak for Simon, but for me, being able to ensure a 
>>> clear backtrace from a bot would be a big improvement.
>>> 
> And again, I think this discussion would get a lot more focused if the 
> change could apply only to JSC code, and not also to WTF code.
 I believe Mark's proposal, initially, is just to make JSC do this. So I 
 don't see the point of compiling WTF differently. JSC can kick off its own 
 build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given 
 people working on JSC want this, and people working on JSC defend these 
 tests, and that these test results are more stable (see below), we should 
 make this change for JSC.
 
 I was trying to convince folks defending non-JSC testing, that they too, 
 should want this. I'm not going to pull teeth here. If folks want their 
 tests to take ~10x longer to run, they're entitled to make that tradeoff.
>>> 
>>> Got it.
>> 
>> I'm of the same mind as Saam.  We want this change for the JSC bots, and 
>> from the time measurements I’ve collected, we can afford to do a clean build 
>> for the JSC Debug test runs using O3, and still come out way ahead.
> 
> This seems like a reasonable plan. You didn't mention what hardware you 
> measured with, but it seems certain to be beneficial on any current hardware.
> 
> I need to mention that we saw unexplained and very large impact on JSC test 
> speed from enabling/disabling TCSM. That may be a good thing to look into 
> while optimizing JSC test speed.
> 
> - Alexey
> 
> 
>> As for non-JSC test runs, I have not actually measured what the time savings 
>> are.  Given there is resistance to going with O3 there, we don’t have to 
>> share the build artifacts for running the tests.  JSC test runs should be 
>> able to just build JSC for each O3 Debug JSC test run and it is still a win 
>> over the current status quo i.e. test runs never complete.
>> 
>> Regarding Geoff’s earlier question about whether I know for sure that 
>> switching to O3 will fix the current Debug JSC bot failures to run tests, 
>> the answer is I’m not certain.  The failure is a timeout due to the master 
>> bot not seeing any output from the tester bot for more than 20 minutes.  
>> I’ve not been able to reproduce this yet.  But with a Debug build test run 
>> taking 4+ hours, it can only be a progression to switch the Debug JSC test 
>> bots to O3.
>> 
>> Mark
>> 
>> 
>>> 
> And again, on the run more tests front, it would be helpful to know 
> whether this change was enough to get the stress tests running or not.
 My experience running the tests locally supports this fully. I don't get 
 timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd 
 get timeouts all the time, and the total test run would take ~4-8 hours. 
 We can wait for official confirmation from Mark.
>>> 
>>> Alexey, do the JSC stress tests run now on bots? If not, how fast would 
>>> they need to run in order to be eligible to run on bots?
>>> 
>>> Thanks,
>>> Geoff
>>> 
 
 - Saam
 
> 
> Thanks,
> Geoff
> 
>> On Jun 18, 2020, at 9:30 PM, Saam Barati > > wrote:
>> 
>> Why are we insisting on doing something on the bots that takes ~10x 
>> longer to run than necessary? I’d rather have that time spent running 
>> more tests.
>> 
>> Overall, how we’re doing things now feels like a bad allocation of bot 
>> resources. The differences I see between O3 with no-inlining vs O0 is:
>> - Some race conditions will behave differently. Race condit