Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Eric Seidel]

This issue has resurfaced again today.  We had 3 long breaks from 3
different people all related to JSC changes in the last 24 hours. :(

FYI, the WinEWS bot is up and running again (this time for good!) as
of 48 hours ago.  I don't think it's quite caught up yet, and it takes
about 60 mins to cycle for big builds.  However, it's at least able to
provide some information for patches which are up for review for an
hour or more.

-eric

On Wed, Apr 21, 2010 at 3:27 PM, Gavin Barraclough
 wrote:
> Hi Eric,
>
> Many apologies for the redness.  These changes are pretty much complete now,
> so hopefully there shouldn't be any more big file moves like this too soon.
>
> One thing that was hugely useful in minimizing the breakage as much as
> possible while making these changes was the ews bots – these generally
> helped me to get my patches building cleanly on all platforms bar Windows
> before committing.  It is a real shame that an ews bot isn't available for
> Windows, since this would be particularly useful - JSC changes frequently
> break Windows builds due to the .def files.
>
> I believe a big problem that caused the extended periods of redness was the
> slowness of the Windows test queues.  These can lag badly behind the builds,
> making failures here very are easy to miss - having landed a large change,
> and waited to watch the waterfall stay green for an extended period of time,
> it was easy to be under the misapprehension that everything was okay.  Only
> later would I discover windows test had started to fail.  Clearly there is a
> lesson I've learned here, but maybe we can find some more hardware to throw
> at these queues, to help them avoid getting quite so far behind.
>
> cheers,
> G.
>
>
> On Apr 21, 2010, at 1:39 PM, Eric Seidel wrote:
>
>> A large portion of the tree redness in the last 3 days is due to JSC
>> string re-factoring.
>>
>> We need to build some better tools, or find some better method to land
>> these changes w/o hosing the tree.  I'm happy to help with building of
>> said tools if folks have requests/suggestions.
>>
>> Broken in 58001  Fixes: 58003, 58006, 58007, 58008, 58010
>> Time: 55m
>>
>> Broken in 57904   Fixes: 57908, 57911, 57912, 57917
>> Time 1hr 45m
>>
>> Broken in 57829   Attempted fix: 57835, Rolled out in:57853
>> Time: 3h 21m
>>
>> Re-broke in 57879   Fixes: 57883, 57884
>> Time: 3h 3m
>>
>> Getting 57829 landed resulted in nearly a full work-day of tree
>> redness. :(  Also, even once a change is fixed, it will take 15 mins
>> or so for all the bots to cycle green.
>>
>> -eric
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 3:32 PM, Adam Roben wrote:

> On Apr 22, 2010, at 3:02 PM, Ojan Vafai wrote:
> 
>> On Thu, Apr 22, 2010 at 11:44 AM, Brian Weinstein  
>> wrote:
>> What is going there is that both machines are running Windows Debug Tests 
>> (or Windows Release Tests).
>> 
>> We have two machines that run both Windows Debug Tests, and Windows Release 
>> Tests, so there are some times where they are both running the same kind of 
>> test, and it looks like the other kind of test is idle, when both machines 
>> are actually busy.
>> 
>> I get that in theory, but that still doesn't explain huge gaps.
>> 
>> For example:
>> Build 12537 and Build 12538 were running on Windows Debug at the same time 
>> using apple-windows-3 and apple-windows-4 respectively. I would expect that 
>> as soon as 12537 finished, the windows release bot would have started Build 
>> 11615 on apple-windows-3, but it actually waited 8-10 minutes. As best I can 
>> tell, that's 8-10 minutes where the apple-windows-3 slave was actually just 
>> idle.
>> 
>> Am I reading the data wrong?
> 
> I think you're interpreting it correctly.
> 
> Maybe this is an artifact of having a single machine switch between two 
> different logical builders?

It looks like the Leopard builders are set up this way, too (there are 2 bots, 
both of which each to Release and Debug builds). In some cases I see a gap 
between successive builds on different logical builders, but it's much smaller 
than the 8-10 minute gap Ojan cited above. (E.g., build 13929 (Leopard Release) 
and build 13680 (Leopard Debug), which were both done on apple-xserve-1; 
there's a ~1 minute gap between the end of the first build and the start of the 
second.)

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 3:02 PM, Ojan Vafai wrote:

> On Thu, Apr 22, 2010 at 11:44 AM, Brian Weinstein  
> wrote:
> What is going there is that both machines are running Windows Debug Tests (or 
> Windows Release Tests).
> 
> We have two machines that run both Windows Debug Tests, and Windows Release 
> Tests, so there are some times where they are both running the same kind of 
> test, and it looks like the other kind of test is idle, when both machines 
> are actually busy.
> 
> I get that in theory, but that still doesn't explain huge gaps.
> 
> For example:
> Build 12537 and Build 12538 were running on Windows Debug at the same time 
> using apple-windows-3 and apple-windows-4 respectively. I would expect that 
> as soon as 12537 finished, the windows release bot would have started Build 
> 11615 on apple-windows-3, but it actually waited 8-10 minutes. As best I can 
> tell, that's 8-10 minutes where the apple-windows-3 slave was actually just 
> idle.
> 
> Am I reading the data wrong?

I think you're interpreting it correctly.

Maybe this is an artifact of having a single machine switch between two 
different logical builders?

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Ojan Vafai]

On Thu, Apr 22, 2010 at 11:44 AM, Brian Weinstein wrote:

> What is going there is that both machines are running Windows Debug Tests
> (or Windows Release Tests).
>
> We have two machines that run both Windows Debug Tests, and Windows Release
> Tests, so there are some times where they are both running the same kind of
> test, and it looks like the other kind of test is idle, when both machines
> are actually busy.
>

I get that in theory, but that still doesn't explain huge gaps.

For example:
Build 12537 and Build 12538 were running on Windows Debug at the same time
using apple-windows-3 and apple-windows-4 respectively. I would expect that
as soon as 12537 finished, the windows release bot would have started Build
11615 on apple-windows-3, but it actually waited 8-10 minutes. As best I can
tell, that's 8-10 minutes where the apple-windows-3 slave was actually just
idle.

Am I reading the data wrong?

Ojan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Brian Weinstein]

Ojan,

What is going there is that both machines are running Windows Debug Tests (or 
Windows Release Tests).

We have two machines that run both Windows Debug Tests, and Windows Release 
Tests, so there are some times where they are both running the same kind of 
test, and it looks like the other kind of test is idle, when both machines are 
actually busy.

Brian

On Apr 22, 2010, at 11:42 AM, Ojan Vafai wrote:

> On Thu, Apr 22, 2010 at 11:32 AM, Adam Roben  wrote:
> On Apr 22, 2010, at 2:30 PM, Ojan Vafai wrote:
> Builds 11611 and 11612 took 89 and 93 seconds, respectively. Seems like 
> there's a fair amount of variance. In any case it sounds like the svn step is 
> even less of an issue than we thought.
> 
> In addition, those builds took less than 14 minutes overall, which is not so 
> far off from the Leopard Release test bots. So what's causing the slowness on 
> Windows?
> 
> Do we just have fewer machines running Windows tests? Also, there are huge 
> gaps where both the release and debug Windows Tests bots have huge idle 
> periods despite having pending builds to test.
> 
> For example: 
> http://build.webkit.org/waterfall?builder=Windows%20Debug%20(Build)&builder=Windows%20Debug%20(Tests)&builder=Windows%20Release%20(Build)&builder=Windows%20Release%20(Tests)
> 
> 11:06:15-11:31:39. The debug tests bot is just idle for 25 minutes despite 
> having 32 pending builds to test. 
> 
> With 32 pending builds, anyone checking in now will have to wait 8 hours to 
> see the results of their checkin on the Windows Tests bots.
> 
> Ojan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Ojan Vafai]

On Thu, Apr 22, 2010 at 11:32 AM, Adam Roben  wrote:

> On Apr 22, 2010, at 2:30 PM, Ojan Vafai wrote:
> Builds 11611 and 11612 took 89 and 93 seconds, respectively. Seems like
> there's a fair amount of variance. In any case it sounds like the svn step
> is even less of an issue than we thought.
>
> In addition, those builds took less than 14 minutes overall, which is not
> so far off from the Leopard Release test bots. So what's causing the
> slowness on Windows?
>

Do we just have fewer machines running Windows tests? Also, there are huge
gaps where both the release and debug Windows Tests bots have huge idle
periods despite having pending builds to test.

For example:
http://build.webkit.org/waterfall?builder=Windows%20Debug%20(Build)&builder=Windows%20Debug%20(Tests)&builder=Windows%20Release%20(Build)&builder=Windows%20Release%20(Tests)

11:06:15-11:31:39. The debug tests bot is just idle for 25 minutes despite
having 32 pending builds to test.

With 32 pending builds, anyone checking in now will have to wait 8 hours to
see the results of their checkin on the Windows Tests bots.

Ojan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 2:30 PM, Ojan Vafai wrote:

> On Thu, Apr 22, 2010 at 11:25 AM, Adam Barth  wrote:
> On Thu, Apr 22, 2010 at 11:23 AM, Adam Roben  wrote:
> > On Apr 22, 2010, at 2:19 PM, Adam Roben wrote:
> >> The "svn" step of the test builders takes about 4.5 minutes, compared to 
> >> about 15 minutes to run the tests. I don't know if having to update two 
> >> different source trees is the main problem here.
> >
> > Note that this data was for a revision where only 5 files changed. If many 
> > more files were changing, I'd expect the "svn" step to take a somewhat 
> > greater proportion of the time (though I don't have data to back this up).
> 
> Why does the svn step take five minutes when only five files change?
> That seems obscenely slow.
> 
> Agreed. Looking at the windows bots right now, the svn step is taking 20-40 
> seconds. Maybe the 5 minutes measurement was just an outlier?

Builds 11611 and 11612 took 89 and 93 seconds, respectively. Seems like there's 
a fair amount of variance. In any case it sounds like the svn step is even less 
of an issue than we thought.

In addition, those builds took less than 14 minutes overall, which is not so 
far off from the Leopard Release test bots. So what's causing the slowness on 
Windows?

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Barth]

On Thu, Apr 22, 2010 at 11:23 AM, Adam Roben  wrote:
> On Apr 22, 2010, at 2:19 PM, Adam Roben wrote:
>> The "svn" step of the test builders takes about 4.5 minutes, compared to 
>> about 15 minutes to run the tests. I don't know if having to update two 
>> different source trees is the main problem here.
>
> Note that this data was for a revision where only 5 files changed. If many 
> more files were changing, I'd expect the "svn" step to take a somewhat 
> greater proportion of the time (though I don't have data to back this up).

Why does the svn step take five minutes when only five files change?
That seems obscenely slow.

Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 2:19 PM, Adam Roben wrote:

> The "svn" step of the test builders takes about 4.5 minutes, compared to 
> about 15 minutes to run the tests. I don't know if having to update two 
> different source trees is the main problem here.

Note that this data was for a revision where only 5 files changed. If many more 
files were changing, I'd expect the "svn" step to take a somewhat greater 
proportion of the time (though I don't have data to back this up).

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 2:04 PM, Brian Weinstein wrote:

> This means that they both have to keep two trees up to date (the debug and 
> release trees),

The "svn" step of the test builders takes about 4.5 minutes, compared to about 
15 minutes to run the tests. I don't know if having to update two different 
source trees is the main problem here.

> and that ~2/3 of the
> time on both machines will be spend running the slower debug tests, causing 
> them both to fall behind.

I guess you're implying that having one more-up-to-date bot and one 
even-farther-behind bot would be better than the current situation. 
Unfortunately, right now the Debug tests are far more stable than the Release 
ones, so your proposed change would presumably make the stable, Debug tests be 
even farther behind than they are today.

> I can upload a patch to bugzilla if that's the easiest way to handle it.

That probably is the best way to do it, if we decide this is a good change to 
make.

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Brian Weinstein]

I sent an email to webkit-dev a couple of weeks ago, explaining my logic behind 
why I thought the testers were slow.



My theory as to what is happening is that:

- The Windows Release tests take ~500 seconds to run.
- The Windows Debug tests take ~950 seconds to run.

The master is currently configured so that both Windows test bots 
(apple-windows-3 and apple-windows-4 - as seen in 
WebKitTools/BuildSlaveSupport/build.webkit.org-config/config.json) both 
currently run Debug and Release Tests. This means that they both have to keep 
two trees up to date (the debug and release trees), and that ~2/3 of the
time on both machines will be spend running the slower debug tests, causing 
them both to fall behind.

My idea for a solution to this is to reconfigure the master so that one 
computer is running one kind of test (3 runs debug, 4 runs release or vice 
versa). This would mean that the bots would only need to keep one tree up to 
date (dropping the amount of time they need to spend in the svn step (which is 
surprisingly slow on inspection)), and it would mean if the debug bot fell 
behind, the release bot would be able to stay caught up with the commits.

If there are any objections to this proposal, reply back on webkit-dev, if not, 
it would be great to get the master changed to make this happen.

I think another reason it was especially bad today was that in the past 24 
hours there have been 100+ commits, which is much higher than the average.

Thanks,
Brian Weinstein



I think we need to make this change on the master, and didn't hear any 
objections to it when I brought it up last time. I can upload a patch to 
bugzilla if that's the easiest way to handle it.

Thanks,
Brian
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Ojan Vafai]

On Thu, Apr 22, 2010 at 10:46 AM, Adam Roben  wrote:

> On Apr 22, 2010, at 1:31 PM, Maciej Stachowiak wrote:
> > I suspect Windows builders fall behind whenever there is a sequence of
> changes that each requires a long rebuild.
>
> That might be right, though I'm having trouble finding data on this because
> all the recent changes that would have caused long builds broke things on
> one platform or the other.
>

The slowness of the builders only means that we do fewer builds. So when the
bot turns red, the regression window is large. For the most part, we don't
actually fall behind on the builders. The problem is that the build is
*much* faster than running the tests. So we often end up in a situation
where the build finishes in a reasonable amount of time, but the tests don't
run for hours because we have a number of builds queued up. I've literally
waited 10 hours after the build has completed for the tests to run.

The build time could use an improvement, but it's minor in comparison to the
time to run tests.

Ojan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 1:46 PM, Adam Roben wrote:

>> and I believe that is because they are not doing parallel builds that take 
>> advantage of their multiple cores.
> 
> Yes, this is because pdevenv doesn't currently work with VC++ Express (which 
> the bots are using to build). There might be ways to work around this, 
> though. I filed  about this.

Actually, the bots have Visual Studio Professional installed and *are* doing 
parallel builds. It will be interesting to see how the build times compare to 
OS X once we have some better data.

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 22, 2010, at 1:31 PM, Maciej Stachowiak wrote:

> On Apr 21, 2010, at 3:35 PM, Ojan Vafai wrote:
> 
>> On Wed, Apr 21, 2010 at 3:27 PM, Gavin Barraclough  
>> wrote:
>> I believe a big problem that caused the extended periods of redness was the 
>> slowness of the Windows test queues.  These can lag badly behind the builds, 
>> making failures here very are easy to miss - having landed a large change, 
>> and waited to watch the waterfall stay green for an extended period of time, 
>> it was easy to be under the misapprehension that everything was okay.  Only 
>> later would I discover windows test had started to fail.  Clearly there is a 
>> lesson I've learned here, but maybe we can find some more hardware to throw 
>> at these queues, to help them avoid getting quite so far behind.
>> 
>> I also have had this problem many times. Throwing more hardware at it would 
>> be great.
>> 
>> Also, maybe we could use one of the Windows test bots for the initial trial 
>> of new-run-webkit-tests. Do we know how may cores are on those windows 
>> machines? The improvement in running the tests will be roughly proportional 
>> to the number of cores on the machine.
> 
> That will improve the time running the tests, but I think the slow thing on 
> Windows bots is building,

It looks like test runs take about 19 minutes on the Windows Release bot, vs 11 
minutes on the Leopard Release bot. That's almost twice as slow!

> and I believe that is because they are not doing parallel builds that take 
> advantage of their multiple cores.

Yes, this is because pdevenv doesn't currently work with VC++ Express (which 
the bots are using to build). There might be ways to work around this, though. 
I filed  about this.

> I suspect Windows builders fall behind whenever there is a sequence of 
> changes that each requires a long rebuild.

That might be right, though I'm having trouble finding data on this because all 
the recent changes that would have caused long builds broke things on one 
platform or the other.

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Maciej Stachowiak]

On Apr 21, 2010, at 3:35 PM, Ojan Vafai wrote:

On Wed, Apr 21, 2010 at 3:27 PM, Gavin Barraclough > wrote:
I believe a big problem that caused the extended periods of redness  
was the slowness of the Windows test queues.  These can lag badly  
behind the builds, making failures here very are easy to miss -  
having landed a large change, and waited to watch the waterfall stay  
green for an extended period of time, it was easy to be under the  
misapprehension that everything was okay.  Only later would I  
discover windows test had started to fail.  Clearly there is a  
lesson I've learned here, but maybe we can find some more hardware  
to throw at these queues, to help them avoid getting quite so far  
behind.


I also have had this problem many times. Throwing more hardware at  
it would be great.


Also, maybe we could use one of the Windows test bots for the  
initial trial of new-run-webkit-tests. Do we know how may cores are  
on those windows machines? The improvement in running the tests will  
be roughly proportional to the number of cores on the machine.


That will improve the time running the tests, but I think the slow  
thing on Windows bots is building, and I believe that is because they  
are not doing parallel builds that take advantage of their multiple  
cores. I suspect Windows builders fall behind whenever there is a  
sequence of changes that each requires a long rebuild.


Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[David Levin]

On Thu, Apr 22, 2010 at 8:16 AM, Adam Roben  wrote:

> On Apr 21, 2010, at 8:37 PM, Eric Seidel wrote:
>
> > I don't
> > think NRWT works on Windows yet.  Adam Roben had some half-working
> > patches to that affect, but I don't think they're done yet.  Mostly
> > they involve ripping out chromium-specific windows assumptions. :)
>
> I'm planning to clean up these patches and post them either today or
> tomorrow. Unfortunately, there are still problems with Apache on Vista and
> Windows 7 (it crashes constantly due to bad interactions between Cygwin and
> ASLR), and I was also seeing a lot of hangs on XP in DumpRenderTree.
> Hopefully we can iron out these issues quickly!
>

Regarding hangs in DRT, I don't know how similar the windows code is to the
osx side but I was seeing a lot more of these (when I ran the drt in
parallel on osx) before I did this patch
trac.webkit.org/changeset/43721
and then set the home directory to different places for each drt that was
run in parallel. (Now there is another environment variable that was
introduced for this).


> -Adam
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 21, 2010, at 6:35 PM, Ojan Vafai wrote:

> Do we know how may cores are on those windows machines?

4 each.

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Roben]

On Apr 21, 2010, at 8:37 PM, Eric Seidel wrote:

> I don't
> think NRWT works on Windows yet.  Adam Roben had some half-working
> patches to that affect, but I don't think they're done yet.  Mostly
> they involve ripping out chromium-specific windows assumptions. :)

I'm planning to clean up these patches and post them either today or tomorrow. 
Unfortunately, there are still problems with Apache on Vista and Windows 7 (it 
crashes constantly due to bad interactions between Cygwin and ASLR), and I was 
also seeing a lot of hangs on XP in DumpRenderTree. Hopefully we can iron out 
these issues quickly!

-Adam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Eric Seidel]

On Wed, Apr 21, 2010 at 3:35 PM, Ojan Vafai  wrote:
> Also, maybe we could use one of the Windows test bots for the initial trial
> of new-run-webkit-tests. Do we know how may cores are on those windows
> machines? The improvement in running the tests will be roughly proportional
> to the number of cores on the machine.

We'll be starting NRWT testing on the bots tonight.  However, I don't
think NRWT works on Windows yet.  Adam Roben had some half-working
patches to that affect, but I don't think they're done yet.  Mostly
they involve ripping out chromium-specific windows assumptions. :)

-eric
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Adam Barth]

I'll work on finishing the Windows EWS.  The remaining blocking issue
is working around the case insensitivity of the Windows file system.

Adam


On Wed, Apr 21, 2010 at 3:27 PM, Gavin Barraclough
 wrote:
> Hi Eric,
>
> Many apologies for the redness.  These changes are pretty much complete now,
> so hopefully there shouldn't be any more big file moves like this too soon.
>
> One thing that was hugely useful in minimizing the breakage as much as
> possible while making these changes was the ews bots – these generally
> helped me to get my patches building cleanly on all platforms bar Windows
> before committing.  It is a real shame that an ews bot isn't available for
> Windows, since this would be particularly useful - JSC changes frequently
> break Windows builds due to the .def files.
>
> I believe a big problem that caused the extended periods of redness was the
> slowness of the Windows test queues.  These can lag badly behind the builds,
> making failures here very are easy to miss - having landed a large change,
> and waited to watch the waterfall stay green for an extended period of time,
> it was easy to be under the misapprehension that everything was okay.  Only
> later would I discover windows test had started to fail.  Clearly there is a
> lesson I've learned here, but maybe we can find some more hardware to throw
> at these queues, to help them avoid getting quite so far behind.
>
> cheers,
> G.
>
>
> On Apr 21, 2010, at 1:39 PM, Eric Seidel wrote:
>
>> A large portion of the tree redness in the last 3 days is due to JSC
>> string re-factoring.
>>
>> We need to build some better tools, or find some better method to land
>> these changes w/o hosing the tree.  I'm happy to help with building of
>> said tools if folks have requests/suggestions.
>>
>> Broken in 58001  Fixes: 58003, 58006, 58007, 58008, 58010
>> Time: 55m
>>
>> Broken in 57904   Fixes: 57908, 57911, 57912, 57917
>> Time 1hr 45m
>>
>> Broken in 57829   Attempted fix: 57835, Rolled out in:57853
>> Time: 3h 21m
>>
>> Re-broke in 57879   Fixes: 57883, 57884
>> Time: 3h 3m
>>
>> Getting 57829 landed resulted in nearly a full work-day of tree
>> redness. :(  Also, even once a change is fixed, it will take 15 mins
>> or so for all the bots to cycle green.
>>
>> -eric
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Ojan Vafai]

On Wed, Apr 21, 2010 at 3:27 PM, Gavin Barraclough wrote:

> I believe a big problem that caused the extended periods of redness was the
> slowness of the Windows test queues.  These can lag badly behind the builds,
> making failures here very are easy to miss - having landed a large change,
> and waited to watch the waterfall stay green for an extended period of time,
> it was easy to be under the misapprehension that everything was okay.  Only
> later would I discover windows test had started to fail.  Clearly there is a
> lesson I've learned here, but maybe we can find some more hardware to throw
> at these queues, to help them avoid getting quite so far behind.
>

I also have had this problem many times. Throwing more hardware at it would
be great.

Also, maybe we could use one of the Windows test bots for the initial trial
of new-run-webkit-tests. Do we know how may cores are on those windows
machines? The improvement in running the tests will be roughly proportional
to the number of cores on the machine.

Ojan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Kenneth Christiansen]

That said, the Tiger bot is often lacking behind as well, which I'm
not sure should be acceptable for core builders.

Kenneth

On Wed, Apr 21, 2010 at 7:27 PM, Gavin Barraclough
 wrote:
> Hi Eric,
>
> Many apologies for the redness.  These changes are pretty much complete now,
> so hopefully there shouldn't be any more big file moves like this too soon.
>
> One thing that was hugely useful in minimizing the breakage as much as
> possible while making these changes was the ews bots – these generally
> helped me to get my patches building cleanly on all platforms bar Windows
> before committing.  It is a real shame that an ews bot isn't available for
> Windows, since this would be particularly useful - JSC changes frequently
> break Windows builds due to the .def files.
>
> I believe a big problem that caused the extended periods of redness was the
> slowness of the Windows test queues.  These can lag badly behind the builds,
> making failures here very are easy to miss - having landed a large change,
> and waited to watch the waterfall stay green for an extended period of time,
> it was easy to be under the misapprehension that everything was okay.  Only
> later would I discover windows test had started to fail.  Clearly there is a
> lesson I've learned here, but maybe we can find some more hardware to throw
> at these queues, to help them avoid getting quite so far behind.
>
> cheers,
> G.
>
>
> On Apr 21, 2010, at 1:39 PM, Eric Seidel wrote:
>
>> A large portion of the tree redness in the last 3 days is due to JSC
>> string re-factoring.
>>
>> We need to build some better tools, or find some better method to land
>> these changes w/o hosing the tree.  I'm happy to help with building of
>> said tools if folks have requests/suggestions.
>>
>> Broken in 58001  Fixes: 58003, 58006, 58007, 58008, 58010
>> Time: 55m
>>
>> Broken in 57904   Fixes: 57908, 57911, 57912, 57917
>> Time 1hr 45m
>>
>> Broken in 57829   Attempted fix: 57835, Rolled out in:57853
>> Time: 3h 21m
>>
>> Re-broke in 57879   Fixes: 57883, 57884
>> Time: 3h 3m
>>
>> Getting 57829 landed resulted in nearly a full work-day of tree
>> redness. :(  Also, even once a change is fixed, it will take 15 mins
>> or so for all the bots to cycle green.
>>
>> -eric
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>



-- 
Kenneth Rohde Christiansen
Technical Lead / Senior Software Engineer
Qt Labs Americas, Nokia Technology Institute, INdT
Phone  +55 81 8895 6002 / E-mail kenneth.christiansen at openbossa.org
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] JSC String re-factorings are hosing the tree.

[Gavin Barraclough]

Hi Eric,

Many apologies for the redness.  These changes are pretty much  
complete now, so hopefully there shouldn't be any more big file moves  
like this too soon.


One thing that was hugely useful in minimizing the breakage as much as  
possible while making these changes was the ews bots – these generally  
helped me to get my patches building cleanly on all platforms bar  
Windows before committing.  It is a real shame that an ews bot isn't  
available for Windows, since this would be particularly useful - JSC  
changes frequently break Windows builds due to the .def files.


I believe a big problem that caused the extended periods of redness  
was the slowness of the Windows test queues.  These can lag badly  
behind the builds, making failures here very are easy to miss - having  
landed a large change, and waited to watch the waterfall stay green  
for an extended period of time, it was easy to be under the  
misapprehension that everything was okay.  Only later would I discover  
windows test had started to fail.  Clearly there is a lesson I've  
learned here, but maybe we can find some more hardware to throw at  
these queues, to help them avoid getting quite so far behind.


cheers,
G.


On Apr 21, 2010, at 1:39 PM, Eric Seidel wrote:


A large portion of the tree redness in the last 3 days is due to JSC
string re-factoring.

We need to build some better tools, or find some better method to land
these changes w/o hosing the tree.  I'm happy to help with building of
said tools if folks have requests/suggestions.

Broken in 58001  Fixes: 58003, 58006, 58007, 58008, 58010
Time: 55m

Broken in 57904   Fixes: 57908, 57911, 57912, 57917
Time 1hr 45m

Broken in 57829   Attempted fix: 57835, Rolled out in:57853
Time: 3h 21m

Re-broke in 57879   Fixes: 57883, 57884
Time: 3h 3m

Getting 57829 landed resulted in nearly a full work-day of tree
redness. :(  Also, even once a change is fixed, it will take 15 mins
or so for all the bots to cycle green.

-eric
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] JSC String re-factorings are hosing the tree.

[Eric Seidel]

A large portion of the tree redness in the last 3 days is due to JSC
string re-factoring.

We need to build some better tools, or find some better method to land
these changes w/o hosing the tree.  I'm happy to help with building of
said tools if folks have requests/suggestions.

Broken in 58001  Fixes: 58003, 58006, 58007, 58008, 58010
Time: 55m

Broken in 57904   Fixes: 57908, 57911, 57912, 57917
Time 1hr 45m

Broken in 57829   Attempted fix: 57835, Rolled out in:57853
Time: 3h 21m

Re-broke in 57879   Fixes: 57883, 57884
Time: 3h 3m

Getting 57829 landed resulted in nearly a full work-day of tree
redness. :(  Also, even once a change is fixed, it will take 15 mins
or so for all the bots to cycle green.

-eric
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev