Re: [webkit-dev] git.webkit.org problem
Here are some things that may help; *#1 svn server overloading when all the slaves bootstrap at the same time* The only good work around is to keep an internal read-only svn-mirror that the slaves use instead of the real svn server. This means that builds must be triggered on the svn-mirror, which may incurs a delay of a few seconds, due to svnsync polling. My hypothesis is that svn has repo-wide locks while committing, so when there are a lot of readers, they are all starved whenever some action is happening. Doing the svn-mirror trick helped us to scale in the range past *several hundreds of simultaneous active connections*. As a guidance, gclient sync does 8 svn connections per slave by default. Scale that to hundreds of slaves waking up at the same time... Also, serving the read-only svn server from a set of apache frontends helps. We do that with src.chromium.org. To be clear, above I was talking about a DMZ-only mirror. It is a different one from the apache frontends. Implementing both has the biggest benefit. *#2 Using Chromium's git svn clone of webkit* We have a few read only git frontends that are relatively speedy. I just verified and both master@HEAD matches: git remote add chromium http://git.chromium.org/external/Webkit.git git fetch chromium Once you verified that the hash matches locally with: git fetch --all git log -1 origin/master git log -1 chromium/master Then you may want to; git config remote.chromium.fetch +refs/heads/*:refs/remotes/origin/* git branch -r -D chromium/master I give no guarantee what-so-ever about freshness, lack of corruption, consistent performance, availability, YMMV, etc. Note the url has a lower K! Note that we also have WebKit_trimmed.git (note the upper case K!) that is much smaller due to removal of many files chromium developers don't need but it has invalid hashes so *do not fetch this one in your current webkit git-svn clone*. *#3 Buildbot's (absence of) performance* Stefan (cc'ed) is also looking at this and has been profiling our slowest masters for a while. In our case, we are CPU-bound and it looks like we have to tune our caching. Stefan can explain it better than me. You guys probably want to talk together. Hope this is useful, M-A Le 29 février 2012 21:50, William Siegrist wsiegr...@apple.com a écrit : We're back to the slaves overloading svn. The Apple slaves are blocked temporarily while everyone else catches up. Sorry for the inconvenience. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
Hi, Marc-Antoine Ruel írta: Here are some things that may help; *#1 svn server overloading when all the slaves bootstrap at the same time* The only good work around is to keep an internal read-only svn-mirror that the slaves use instead of the real svn server. This means that builds must be triggered on the svn-mirror, which may incurs a delay of a few seconds, due to svnsync polling. My hypothesis is that svn has repo-wide locks while committing, so when there are a lot of readers, they are all starved whenever some action is happening. Doing the svn-mirror trick helped us to scale in the range past /several hundreds of simultaneous active connections/. As a guidance, gclient sync does 8 svn connections per slave by default. Scale that to hundreds of slaves waking up at the same time... Also, serving the read-only svn server from a set of apache frontends helps. We do that with src.chromium.org http://src.chromium.org. To be clear, above I was talking about a DMZ-only mirror. It is a different one from the apache frontends. Implementing both has the biggest benefit. We like the idea to setup svn mirrors, and we are ready to setup an svn mirror for Qt buildslaves hosted in Szeged. (6 slaves connected to build.webkit.org master and 13 slaves connected to build.webkit.sed.hu master) I think this mirror would help to reduce the load of svn.webkit.org and avoid rm -rf because of slow and not so stable trans-atlantic network connection. If there isn't any objection against adding an svn.webkit.sed.hu mirror, we will start to setup the server and prepare the master.cfg change to make slaves to be able to use svn mirrors, and add new scheduler to trigger slaves depends on their svn mirror used. And one more technical question. Could you add an svn post-commit hook to do svnsync on the mirror svn server if we finished setting up the server? br, Ossy ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Mar 1, 2012, at 7:56 AM, Osztrogonac Csaba wrote: And one more technical question. Could you add an svn post-commit hook to do svnsync on the mirror svn server if we finished setting up the server? How would you want svn.webkit.org to signal the mirror? I think having svn.webkit.org request a specific HTTP URL would work. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
Hi, William Siegrist írta: On Mar 1, 2012, at 7:56 AM, Osztrogonac Csaba wrote: And one more technical question. Could you add an svn post-commit hook to do svnsync on the mirror svn server if we finished setting up the server? How would you want svn.webkit.org to signal the mirror? I think having svn.webkit.org request a specific HTTP URL would work. My first idea was that I gives you ssh or something else access to the svn mirror and the following post-commit hook will do the synchronization: /usr/local/bin/svnsync synchronize --username svnsync svn+ssh://svnsync@remote/home/svnsync/svn But your suggestion is reasonable and more safe than mine, when svn.webkit.org does a specific HTTP request and then our svn mirror will start the synchronization. Let's talk about it in private mail after we finished setting up the server. PS. So could you unban our subnet in the near future, please? br, Ossy ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Mar 1, 2012, at 9:36 AM, Osztrogonac Csaba o...@inf.u-szeged.hu wrote: Hi, William Siegrist írta: On Mar 1, 2012, at 7:56 AM, Osztrogonac Csaba wrote: And one more technical question. Could you add an svn post-commit hook to do svnsync on the mirror svn server if we finished setting up the server? How would you want svn.webkit.org to signal the mirror? I think having svn.webkit.org request a specific HTTP URL would work. My first idea was that I gives you ssh or something else access to the svn mirror and the following post-commit hook will do the synchronization: /usr/local/bin/svnsync synchronize --username svnsync svn+ssh://svnsync@remote/home/svnsync/svn But your suggestion is reasonable and more safe than mine, when svn.webkit.org does a specific HTTP request and then our svn mirror will start the synchronization. Let's talk about it in private mail after we finished setting up the server. PS. So could you unban our subnet in the near future, please? We're going to try slowly bring on more slaves soon. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Mar 1, 2012, at 11:13 AM, William Siegrist wsiegr...@apple.com wrote: On Mar 1, 2012, at 9:36 AM, Osztrogonac Csaba o...@inf.u-szeged.hu wrote: Hi, William Siegrist írta: On Mar 1, 2012, at 7:56 AM, Osztrogonac Csaba wrote: And one more technical question. Could you add an svn post-commit hook to do svnsync on the mirror svn server if we finished setting up the server? How would you want svn.webkit.org to signal the mirror? I think having svn.webkit.org request a specific HTTP URL would work. My first idea was that I gives you ssh or something else access to the svn mirror and the following post-commit hook will do the synchronization: /usr/local/bin/svnsync synchronize --username svnsync svn+ssh://svnsync@remote/home/svnsync/svn But your suggestion is reasonable and more safe than mine, when svn.webkit.org does a specific HTTP request and then our svn mirror will start the synchronization. Let's talk about it in private mail after we finished setting up the server. PS. So could you unban our subnet in the near future, please? We're going to try slowly bring on more slaves soon. The Szeged slaves are unblocked. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
William Siegrist írta: The Szeged slaves are unblocked. -Bill Many thanks, I copied up-to-date svn working copy to them and then started. I have only one more little request. Could you kick git.webkit.org too? It seems it is still stay on r109303. Our EWS would be happy if it works again. PS. Tomorrow we are going to setup the Szeged svn mirror. ;) br, Ossy ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Mar 1, 2012, at 12:54 PM, Osztrogonac Csaba o...@inf.u-szeged.hu wrote: William Siegrist írta: The Szeged slaves are unblocked. -Bill Many thanks, I copied up-to-date svn working copy to them and then started. I have only one more little request. Could you kick git.webkit.org too? It seems it is still stay on r109303. Our EWS would be happy if it works again. Thanks for the reminder. It was next on my list to clean up. It should be caught up in a few minutes. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
Is it any trick to update my git-svn repo from svn? I set up my git svn repo about $http://trac.webkit.org/wiki/UsingGitWithWebKit : - git clone git://git.webkit.org/WebKit.git WebKit - git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit - git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master Usually, the only missing step here is 'git svn fetch'. However, when I run that now I get: RA layer request failed: OPTIONS of 'http://svn.webkit.org/repository/webkit': could not connect to server (http://svn.webkit.org) at /Users/davidbarr/libexec/git-core/git-svn line 2139 -- David Barr. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
This should be fixed now. -Bill On Feb 29, 2012, at 3:19 AM, David Barr wrote: Is it any trick to update my git-svn repo from svn? I set up my git svn repo about $http://trac.webkit.org/wiki/UsingGitWithWebKit : - git clone git://git.webkit.org/WebKit.git WebKit - git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit - git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master Usually, the only missing step here is 'git svn fetch'. However, when I run that now I get: RA layer request failed: OPTIONS of 'http://svn.webkit.org/repository/webkit': could not connect to server (http://svn.webkit.org) at /Users/davidbarr/libexec/git-core/git-svn line 2139 -- David Barr. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
Hi Bill, Would it be possible to get some more information on your/Lucas' plans on improving stability of the infrastructure? This was the third significant interruption within a week, and especially for us non-PST folks it's becoming quite inconvenient. Thanks, Peter On Wed, Feb 29, 2012 at 15:00, William Siegrist wsiegr...@apple.com wrote: This should be fixed now. -Bill On Feb 29, 2012, at 3:19 AM, David Barr wrote: Is it any trick to update my git-svn repo from svn? I set up my git svn repo about $http://trac.webkit.org/wiki/UsingGitWithWebKit : - git clone git://git.webkit.org/WebKit.git WebKit - git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit - git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master Usually, the only missing step here is 'git svn fetch'. However, when I run that now I get: RA layer request failed: OPTIONS of 'http://svn.webkit.org/repository/webkit': could not connect to server (http://svn.webkit.org) at /Users/davidbarr/libexec/git-core/git-svn line 2139 -- David Barr. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
Hi Lucas, Thank you very much for your answer! One change that was applied to Chromium's build bot console[1] recently was to cache the overview page for a minute. This may be relevant to the Apache caching layer you mentioned. If requests are really handled serially then updating to version 0.8.5 would hopefully yield performance improvements as well. Peter [1] http://chromium-build-master.appspot.com/p/chromium/console On Wed, Feb 29, 2012 at 19:32, Lucas Forschler lforsch...@apple.com wrote: Hi Peter, I am looking into improving performance of build.webkit.org. After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests. I believe requests are processed serially, and not in parallel. The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers. As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity. We may have reached a limit on the number of slaves we can handle with our current configuration. Some of the things I have been researching to increase build.webkit.orgperformance include the following: -Upgrades to hardware. It's possible faster hardware could alleviate some problems. It is unclear if this is really the limiting factor, or how much additional capacity this would provide. -Updating the way files are transferred (https://bugs.webkit.org/show_bug.cgi?id=73484) -Upgrade the version of buildbot. (0.8.3 - 0.8.5). Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. -Upgrading the version of twisted running on the sever. We are currently at version 8.2, and version 12 is now available. -Investigating some kind of apache caching layer on top of buildbot -Adding a second master to the buildbot system. buildbot supports multi-master mode. We could potentially split out build and test onto two separate masters. build.webkit.org and test.webkit.org ? I would be interested in hearing any other thoughts on increasing build.webkit.org performance. Lucas On Feb 29, 2012, at 7:08 AM, Peter Beverloo wrote: Hi Bill, Would it be possible to get some more information on your/Lucas' plans on improving stability of the infrastructure? This was the third significant interruption within a week, and especially for us non-PST folks it's becoming quite inconvenient. Thanks, Peter On Wed, Feb 29, 2012 at 15:00, William Siegrist wsiegr...@apple.comwrote: This should be fixed now. -Bill On Feb 29, 2012, at 3:19 AM, David Barr wrote: Is it any trick to update my git-svn repo from svn? I set up my git svn repo about $http://trac.webkit.org/wiki/UsingGitWithWebKit : - git clone git://git.webkit.org/WebKit.git WebKit - git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit - git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master Usually, the only missing step here is 'git svn fetch'. However, when I run that now I get: RA layer request failed: OPTIONS of 'http://svn.webkit.org/repository/webkit': could not connect to server (http://svn.webkit.org) at /Users/davidbarr/libexec/git-core/git-svn line 2139 -- David Barr. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Wed, Feb 29, 2012 at 11:32 AM, Lucas Forschler lforsch...@apple.comwrote: I am looking into improving performance of build.webkit.org. After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests. I believe requests are processed serially, and not in parallel. The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers. As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity. We may have reached a limit on the number of slaves we can handle with our current configuration. On Chromium side, buildbot appears to leak a great amount of memory. e.g. we end up running out of memory at some point because buildbot keeps leaking some memory over time :( Do you see the same issue on your buildbot? -Upgrades to hardware. It's possible faster hardware could alleviate some problems. It is unclear if this is really the limiting factor, or how much additional capacity this would provide. My understanding is that we use a really fast server but still experience a similar problem. -Updating the way files are transferred (https://bugs.webkit.org/show_bug.cgi?id=73484) Would it make sense to have a separate apache server, etc... for receiving these files? twistd doesn't appear to be a cut out for uploading/downloading megabytes of data. -Upgrade the version of buildbot. (0.8.3 - 0.8.5). Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. I've been told that upgrading buildbot didn't improve the performance. But upgrading it to 0.8.5 is probably a good nonetheless since it fixes various bugs I've encountered while maintaining Chromium bots and also when I ran buildbot locally. -Investigating some kind of apache caching layer on top of buildbot MUST DO THIS. This will greatly reduce the latency for most people. -Adding a second master to the buildbot system. buildbot supports multi-master mode. We could potentially split out build and test onto two separate masters. build.webkit.org and test.webkit.org ? build.chromium.org takes this approach. But then we won't be able to see both build and test step at once though... Would it make more sense to split per port? e.g. Apple Mac Apple Win, Qt Gtk, Chromium misc or something along that line? - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Wed, Feb 29, 2012 at 3:38 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Feb 29, 2012 at 11:32 AM, Lucas Forschler lforsch...@apple.comwrote: I am looking into improving performance of build.webkit.org. After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests. I believe requests are processed serially, and not in parallel. The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers. As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity. We may have reached a limit on the number of slaves we can handle with our current configuration. On Chromium side, buildbot appears to leak a great amount of memory. e.g. we end up running out of memory at some point because buildbot keeps leaking some memory over time :( Do you see the same issue on your buildbot? -Upgrades to hardware. It's possible faster hardware could alleviate some problems. It is unclear if this is really the limiting factor, or how much additional capacity this would provide. My understanding is that we use a really fast server but still experience a similar problem. -Updating the way files are transferred (https://bugs.webkit.org/show_bug.cgi?id=73484) Would it make sense to have a separate apache server, etc... for receiving these files? twistd doesn't appear to be a cut out for uploading/downloading megabytes of data. -Upgrade the version of buildbot. (0.8.3 - 0.8.5). Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. I've been told that upgrading buildbot didn't improve the performance. But upgrading it to 0.8.5 is probably a good nonetheless since it fixes various bugs I've encountered while maintaining Chromium bots and also when I ran buildbot locally. -Investigating some kind of apache caching layer on top of buildbot MUST DO THIS. This will greatly reduce the latency for most people. -Adding a second master to the buildbot system. buildbot supports multi-master mode. We could potentially split out build and test onto two separate masters. build.webkit.org and test.webkit.org ? build.chromium.org takes this approach. But then we won't be able to see both build and test step at once though... Would it make more sense to split per port? e.g. Apple Mac Apple Win, Qt Gtk, Chromium misc or something along that line? If this is pursued, then I would tend to agree with a per-port split vs a vertical split. But, it would mean more masters and more places to keep an eye on when doing cross-port changes, which could get out of hand. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
I'm currenlty upgrading build.webkit.org from 0.8.3 to 0.8.5. It should be back online shortly. Lucas On Feb 29, 2012, at 12:38 PM, Ryosuke Niwa wrote: On Wed, Feb 29, 2012 at 11:32 AM, Lucas Forschler lforsch...@apple.com wrote: I am looking into improving performance of build.webkit.org. After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests. I believe requests are processed serially, and not in parallel. The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers. As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity. We may have reached a limit on the number of slaves we can handle with our current configuration. On Chromium side, buildbot appears to leak a great amount of memory. e.g. we end up running out of memory at some point because buildbot keeps leaking some memory over time :( Do you see the same issue on your buildbot? -Upgrades to hardware. It's possible faster hardware could alleviate some problems. It is unclear if this is really the limiting factor, or how much additional capacity this would provide. My understanding is that we use a really fast server but still experience a similar problem. -Updating the way files are transferred (https://bugs.webkit.org/show_bug.cgi?id=73484) Would it make sense to have a separate apache server, etc... for receiving these files? twistd doesn't appear to be a cut out for uploading/downloading megabytes of data. -Upgrade the version of buildbot. (0.8.3 - 0.8.5). Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. I've been told that upgrading buildbot didn't improve the performance. But upgrading it to 0.8.5 is probably a good nonetheless since it fixes various bugs I've encountered while maintaining Chromium bots and also when I ran buildbot locally. -Investigating some kind of apache caching layer on top of buildbot MUST DO THIS. This will greatly reduce the latency for most people. -Adding a second master to the buildbot system. buildbot supports multi-master mode. We could potentially split out build and test onto two separate masters. build.webkit.org and test.webkit.org ? build.chromium.org takes this approach. But then we won't be able to see both build and test step at once though... Would it make more sense to split per port? e.g. Apple Mac Apple Win, Qt Gtk, Chromium misc or something along that line? - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
build.webkit.org should be back online now with 0.8.5. Thanks for your patience! Lucas On Feb 29, 2012, at 3:44 PM, Lucas Forschler wrote: I'm currenlty upgrading build.webkit.org from 0.8.3 to 0.8.5. It should be back online shortly. Lucas On Feb 29, 2012, at 12:38 PM, Ryosuke Niwa wrote: On Wed, Feb 29, 2012 at 11:32 AM, Lucas Forschler lforsch...@apple.com wrote: I am looking into improving performance of build.webkit.org. After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests. I believe requests are processed serially, and not in parallel. The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers. As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity. We may have reached a limit on the number of slaves we can handle with our current configuration. On Chromium side, buildbot appears to leak a great amount of memory. e.g. we end up running out of memory at some point because buildbot keeps leaking some memory over time :( Do you see the same issue on your buildbot? -Upgrades to hardware. It's possible faster hardware could alleviate some problems. It is unclear if this is really the limiting factor, or how much additional capacity this would provide. My understanding is that we use a really fast server but still experience a similar problem. -Updating the way files are transferred (https://bugs.webkit.org/show_bug.cgi?id=73484) Would it make sense to have a separate apache server, etc... for receiving these files? twistd doesn't appear to be a cut out for uploading/downloading megabytes of data. -Upgrade the version of buildbot. (0.8.3 - 0.8.5). Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. I've been told that upgrading buildbot didn't improve the performance. But upgrading it to 0.8.5 is probably a good nonetheless since it fixes various bugs I've encountered while maintaining Chromium bots and also when I ran buildbot locally. -Investigating some kind of apache caching layer on top of buildbot MUST DO THIS. This will greatly reduce the latency for most people. -Adding a second master to the buildbot system. buildbot supports multi-master mode. We could potentially split out build and test onto two separate masters. build.webkit.org and test.webkit.org ? build.chromium.org takes this approach. But then we won't be able to see both build and test step at once though... Would it make more sense to split per port? e.g. Apple Mac Apple Win, Qt Gtk, Chromium misc or something along that line? - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On 2012-02-29, at 17:05, Lucas Forschler lforsch...@apple.com wrote: build.webkit.org should be back online now with 0.8.5. Thanks for your patience! For anyone following along at home, upgrading to Buildbot v0.8.5 alone actually made build.webkit.org substantially slower. After profiling it for a while we noticed that it was spending an exceptionally long time performing queries against its SQLite database. Some analysis on the queries that were being run showed that the database lacked appropriate indices for the queries, causing full table scans across rather large tables. Ouch. A few CREATE INDEX … statements later and now build.webkit.org is responding much more quickly than it has been in quite some time. If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know. - Mark. Lucas On Feb 29, 2012, at 3:44 PM, Lucas Forschler wrote: I'm currenlty upgrading build.webkit.org from 0.8.3 to 0.8.5. It should be back online shortly. Lucas On Feb 29, 2012, at 12:38 PM, Ryosuke Niwa wrote: On Wed, Feb 29, 2012 at 11:32 AM, Lucas Forschler lforsch...@apple.com wrote: I am looking into improving performance of build.webkit.org. After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests. I believe requests are processed serially, and not in parallel. The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers. As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity. We may have reached a limit on the number of slaves we can handle with our current configuration. On Chromium side, buildbot appears to leak a great amount of memory. e.g. we end up running out of memory at some point because buildbot keeps leaking some memory over time :( Do you see the same issue on your buildbot? -Upgrades to hardware. It's possible faster hardware could alleviate some problems. It is unclear if this is really the limiting factor, or how much additional capacity this would provide. My understanding is that we use a really fast server but still experience a similar problem. -Updating the way files are transferred (https://bugs.webkit.org/show_bug.cgi?id=73484) Would it make sense to have a separate apache server, etc... for receiving these files? twistd doesn't appear to be a cut out for uploading/downloading megabytes of data. -Upgrade the version of buildbot. (0.8.3 - 0.8.5). Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. I've been told that upgrading buildbot didn't improve the performance. But upgrading it to 0.8.5 is probably a good nonetheless since it fixes various bugs I've encountered while maintaining Chromium bots and also when I ran buildbot locally. -Investigating some kind of apache caching layer on top of buildbot MUST DO THIS. This will greatly reduce the latency for most people. -Adding a second master to the buildbot system. buildbot supports multi-master mode. We could potentially split out build and test onto two separate masters. build.webkit.org and test.webkit.org ? build.chromium.org takes this approach. But then we won't be able to see both build and test step at once though... Would it make more sense to split per port? e.g. Apple Mac Apple Win, Qt Gtk, Chromium misc or something along that line? - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Wed, Feb 29, 2012 at 6:10 PM, Mark Rowe mr...@apple.com wrote: On 2012-02-29, at 17:05, Lucas Forschler lforsch...@apple.com wrote: build.webkit.org should be back online now with 0.8.5. Thanks for your patience! For anyone following along at home, upgrading to Buildbot v0.8.5 alone actually made build.webkit.org substantially slower. After profiling it for a while we noticed that it was spending an exceptionally long time performing queries against its SQLite database. Some analysis on the queries that were being run showed that the database lacked appropriate indices for the queries, causing full table scans across rather large tables. Ouch. A few CREATE INDEX … statements later and now build.webkit.org is responding much more quickly than it has been in quite some time. If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know. svn.webkit.org (and trac) appears to be very unhappy at the moment; seems like this is likely related? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On 2012-02-29, at 18:20, Dirk Pranke dpra...@chromium.org wrote: On Wed, Feb 29, 2012 at 6:10 PM, Mark Rowe mr...@apple.com wrote: On 2012-02-29, at 17:05, Lucas Forschler lforsch...@apple.com wrote: build.webkit.org should be back online now with 0.8.5. Thanks for your patience! For anyone following along at home, upgrading to Buildbot v0.8.5 alone actually made build.webkit.org substantially slower. After profiling it for a while we noticed that it was spending an exceptionally long time performing queries against its SQLite database. Some analysis on the queries that were being run showed that the database lacked appropriate indices for the queries, causing full table scans across rather large tables. Ouch. A few CREATE INDEX … statements later and now build.webkit.org is responding much more quickly than it has been in quite some time. If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know. svn.webkit.org (and trac) appears to be very unhappy at the moment; seems like this is likely related? It may be that svn.webkit.org is unhappy about the thundering herd of reconnecting build slaves after the downtime. We'll look in to it. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Feb 29, 2012, at 6:22 PM, Mark Rowe wrote: On 2012-02-29, at 18:20, Dirk Pranke dpra...@chromium.org wrote: On Wed, Feb 29, 2012 at 6:10 PM, Mark Rowe mr...@apple.com wrote: On 2012-02-29, at 17:05, Lucas Forschler lforsch...@apple.com wrote: build.webkit.org should be back online now with 0.8.5. Thanks for your patience! For anyone following along at home, upgrading to Buildbot v0.8.5 alone actually made build.webkit.org substantially slower. After profiling it for a while we noticed that it was spending an exceptionally long time performing queries against its SQLite database. Some analysis on the queries that were being run showed that the database lacked appropriate indices for the queries, causing full table scans across rather large tables. Ouch. A few CREATE INDEX … statements later and now build.webkit.org is responding much more quickly than it has been in quite some time. If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know. svn.webkit.org (and trac) appears to be very unhappy at the moment; seems like this is likely related? It may be that svn.webkit.org is unhappy about the thundering herd of reconnecting build slaves after the downtime. We'll look in to it. I'm rebooting it again. I'm not sure if this is the same cause as the overnight problems or the slaves getting new checkouts. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Feb 29, 2012, at 6:26 PM, William Siegrist wrote: On Feb 29, 2012, at 6:22 PM, Mark Rowe wrote: On 2012-02-29, at 18:20, Dirk Pranke dpra...@chromium.org wrote: On Wed, Feb 29, 2012 at 6:10 PM, Mark Rowe mr...@apple.com wrote: On 2012-02-29, at 17:05, Lucas Forschler lforsch...@apple.com wrote: If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know. svn.webkit.org (and trac) appears to be very unhappy at the moment; seems like this is likely related? It may be that svn.webkit.org is unhappy about the thundering herd of reconnecting build slaves after the downtime. We'll look in to it. I'm rebooting it again. I'm not sure if this is the same cause as the overnight problems or the slaves getting new checkouts. Our database pooling could not handle the frequent reboots of the clients, so I'm rebooting most of webkit.org to clean up the mess. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
On Feb 29, 2012, at 6:33 PM, William Siegrist wrote: On Feb 29, 2012, at 6:26 PM, William Siegrist wrote: On Feb 29, 2012, at 6:22 PM, Mark Rowe wrote: On 2012-02-29, at 18:20, Dirk Pranke dpra...@chromium.org wrote: On Wed, Feb 29, 2012 at 6:10 PM, Mark Rowe mr...@apple.com wrote: On 2012-02-29, at 17:05, Lucas Forschler lforsch...@apple.com wrote: If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know. svn.webkit.org (and trac) appears to be very unhappy at the moment; seems like this is likely related? It may be that svn.webkit.org is unhappy about the thundering herd of reconnecting build slaves after the downtime. We'll look in to it. I'm rebooting it again. I'm not sure if this is the same cause as the overnight problems or the slaves getting new checkouts. Our database pooling could not handle the frequent reboots of the clients, so I'm rebooting most of webkit.org to clean up the mess. And we're back up again. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] git.webkit.org problem
We're back to the slaves overloading svn. The Apple slaves are blocked temporarily while everyone else catches up. Sorry for the inconvenience. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
