Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
On Thu, Jan 22, 2015 at 1:56 PM, Darin Adler wrote: > > On Jan 22, 2015, at 12:58 PM, Dirk Pranke wrote: > > > > Of course, Chromium was more willing to incur the cost of keeping all of > the pixel tests up to date when trivial things changed the output > > This comment, and much of the discussion, seems to be about pixel tests. > But what does this have to do with reference tests? > The message has nothing to do with whether the test is a pixel test or a ref test. My reply was intended to explain why we had the "hashes didn't match but diff passed" message, and the discrepancy around fuzzy matching; on the Chromium ports, if run-webkit-tests had printed this message, it would've served as a warning to the user/developer that there was something weird going on. If you all decide you wanted to keep fuzzy matching enabled (and hence that message would happen regularly and could be safely ignored), I would either delete the message outright or at least make it be a debug message. You could also decide you wanted to do fuzzy matching only on ref tests, or only on pixel tests, and change the logic accordingly, of course. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
On Thu, Jan 22, 2015 at 11:48 AM, Alexey Proskuryakov wrote: > > 22 янв. 2015 г., в 11:30, Simon Fraser > написал(а): > > > This happens when the expected and actual images are very close, but not > identical. ImageDiff has some built-in rounding that effectively acts as a > small tolerance, so the hashes are different, but ImageDiff incorrectly > says the images are the same. For example, some of the tests in question > render a green box either via CALayers, or by painting, and there’s a > slight color difference between the two code paths. > > > > My preference for how to fix this would be to fix ImageDiff to remove > its slight built-in tolerance, and then to expose testRunner API to allow a > test to set an explicit tolerance. There are many cases where we’d like to > use ref tests, but are unable to because of slight, justifiable rendering > differences, and having an explicit tolerance would permit the use of ref > tests in these cases. > > One thing about tolerance is that it is super confusing - are we talking > about the number of pixels that are different, or about how different the > pixels are? Also, a lot of failures only cause small differences in pixel > results. Even a 100x100 box that becomes red instead of green is only a > small portion of the 800x600 image, and it's even more the case for tests > that check e.g. text rendering. > > It is not currently known what the root causes are for the tests that say > "ref test hashes didn't match but diff passed". Given that the differences > are very tiny, one guess is that even though compositing and > non-compositing code paths are mathematically equivalent, there are > different rendering steps taken, and rounding at each step adds up to > slight differences. Another theory is that we have actual bugs, such as > with color management. > > If it's just rounding differences, then the right thing to do is probably > to silence the console output, keeping behavior the same otherwise. > Assuming nothing much has changed since I wrote that code, there are a couple of ways you can get that message. As you've found, the primary culprit is probably the fuzzy matching. We turned off fuzzy matching on the Chromium ports because I never liked the idea; I always was worried that it would mask real bugs. As Alexey points out, the problem with fuzzy matching is that it's not clear what exactly the definition of fuzziness is. What may be appropriately fuzzy in one set of tests may be entirely inappropriate for others. Of course, Chromium was more willing to incur the cost of keeping all of the pixel tests up to date when trivial things changed the output, to the detriment of everyone's repo sizes and checkout times :). There may be other things that can also cause mismatches; I remember Simon bugging me from time to time when Chromium would check in a PNG with an alpha channel, which would through off the checksum on the Mac ports (or something like that). I don't know that I would expect you to be hitting those these days, though. Hopefully that provides a little bit of context, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Any old-run-webkit-tests clients left?
iOS was the only one I was aware of that was left. -- Dirk On Mon, Jul 28, 2014 at 12:25 PM, David Farler wrote: > Hi, > > Are there any ports or users of old-run-webkit-tests left besides the iOS > Simulator? I'm on the verge of completing iOS Simulator support for > webkitpy and run-webkit-tests so, once the new port is stable, I'd like to > consider removing this code. > > David > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Anyone else having these problems with run-webkit-tests?
On Mon, Mar 24, 2014 at 3:32 PM, Bem Jones-Bey wrote: > > On Mar 24, 2014, at 14:54 , Dirk Pranke wrote: > > > > > On Sun, Mar 23, 2014 at 12:52 PM, Benjamin Poulain wrote: > >> Same here :( >> >> >> On 3/23/14, 12:15 PM, Darin Adler wrote: >> >>> When I use run-webkit-tests to run the entire test suite, on a debug >>> build of TOT WebKit, on Mavericks, I’m having the following problems: >>> >>> - Towards the end of the run, the tests run slowly, over a second per >>> test on my 3.5GHz i7 iMac with tons of memory, which is about 10X too slow >>> I think. The sample shows the vast majority of the time is spent in >>> JavaScript garbage collection. This is running mostly the svg/custom tests. >>> >> Yep. The time from 31k to 33k is a long as from 0 to 31k :( >> >> - Towards the end of the run, instead of the 8 parallel copies of >>> DumpRenderTree, only 1 copy of DumpRenderTree seems to run. This is running >>> mostly the svg/custom tests. >>> >> I think the problem is we can only run DumpRenderTree in parallel on >> different folders. Toward the end, everything left is one or two slow >> folders. >> >> > The problem is partially that the sharding is folder-at-a-time, > partially that the svg folder is big and slow, and partially that "svg" > comes near the end of the alphabet (i.e., we don't start the big slow > directory until close to the end of run, so there ends up being one big > long pole). > > At some point we added code to run-webkit-tests to have a list of "slow" > directories that got started towards the beginning. I don't remember if we > did that before or after the Blink fork, but it would be easy to port it > over from Blink if it was after. > > > I'm pretty sure this was after the Blink fork. I did a quick look > through the history, and I found this: > > http://src.chromium.org/viewvc/blink?revision=152697&view=revision > > Is that the correct change? If it will make tests run faster in WebKit > land as well, I'm more than happy to port it myself. :-) > > Yup, that's the one. Obviously the whole "virtual test suite" thing is less relevant, but you can use it for any directory you want. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Anyone else having these problems with run-webkit-tests?
On Sun, Mar 23, 2014 at 12:52 PM, Benjamin Poulain wrote: > Same here :( > > > On 3/23/14, 12:15 PM, Darin Adler wrote: > >> When I use run-webkit-tests to run the entire test suite, on a debug >> build of TOT WebKit, on Mavericks, I’m having the following problems: >> >> - Towards the end of the run, the tests run slowly, over a second per >> test on my 3.5GHz i7 iMac with tons of memory, which is about 10X too slow >> I think. The sample shows the vast majority of the time is spent in >> JavaScript garbage collection. This is running mostly the svg/custom tests. >> > Yep. The time from 31k to 33k is a long as from 0 to 31k :( > > - Towards the end of the run, instead of the 8 parallel copies of >> DumpRenderTree, only 1 copy of DumpRenderTree seems to run. This is running >> mostly the svg/custom tests. >> > I think the problem is we can only run DumpRenderTree in parallel on > different folders. Toward the end, everything left is one or two slow > folders. > > The problem is partially that the sharding is folder-at-a-time, partially that the svg folder is big and slow, and partially that "svg" comes near the end of the alphabet (i.e., we don't start the big slow directory until close to the end of run, so there ends up being one big long pole). At some point we added code to run-webkit-tests to have a list of "slow" directories that got started towards the beginning. I don't remember if we did that before or after the Blink fork, but it would be easy to port it over from Blink if it was after. There is also the --experimental-fully-parallel flag that you can use, which shards a test at a time and get you the full parallelization until the last 8 tests. The downside to this is that you'll probably have a lot more flakiness. Hope this is useful, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
Ryosuke and I discussed this a bit over IRC. Ryosuke's main concern was that supporting multiple document roots adds a fair amount of complexity to NRWT. Conceptually, it's probably easier to add support to run the W3C's new server (known as wptserve) and then maybe use it for *all* imported tests from the W3C. If it turns out that wptserve is too slow, and you would prefer to run only some of the directories over http (i.e., the 10k+ CSS tests don't need http), you'll probably need to modify how wptserve is run (and NRWT) as well, anyway, so the patch would be different. Ryosuke, please let me know if I've misstated your thinking at all. -- Dirk On Mon, Jan 6, 2014 at 11:23 AM, Ryosuke Niwa wrote: > I don't think we should do this given that the python server has been > added to W3C testing harness, and they're gonna convert all existing tests > to use that instead: > http://lists.w3.org/Archives/Public/public-test-infra/2014JanMar/.html > > We should simply wait for that effort to take place and add a support for > the python server instead. > > - R. Niwa > > > On Fri, Dec 6, 2013 at 12:25 AM, youenn fablet wrote: > >> As long as the newly imported tests use relative URLs, alias may be used >> as a workaround. I will give it a try. >> Bug entry is at https://bugs.webkit.org/show_bug.cgi?id=125339 >> Any further help appreciated, >> Youenn >> >> >> >> 2013/12/6 Darin Adler >> >>> If that's really ends up being super hard we can always put yet another >>> third-party or imported directory inside the http directory as previously >>> suggested. it's annoying to have three different places for imported tests >>> and code, but not something I want to hold us up for a long time. >>> >>> -- Darin >>> >>> Sent from my iPhone >>> >>> On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa wrote: >>> >>> On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler wrote: >>> On Dec 4, 2013, at 6:48 AM, youenn fablet wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. >>> >>> One trickiness to it is that we don't currently run any HTTP test in >>> parallel and the document root of the HTTP server is set to >>> LayoutTests/http/tests so we might need to modify that or restart the HTTP >>> server whenever we're running HTTP tests outside of LayoutTests/http. >>> >>> - R. Niwa >>> >>> >> > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
The way I got around this when I was first working on it was to simply map imported/w3c onto a subdirectory of the document root in apache; it's a two line change. For some time I've toyed with the idea of changing the DocumentRoot to just be LayoutTests/, so that any test could be run over http directly. A lot of tests and test results would need to be updated for this, but I think it could simplify the test organization a fair amount and open up some interesting possibilities. -- Dirk On Thu, Dec 5, 2013 at 6:25 PM, Darin Adler wrote: > If that's really ends up being super hard we can always put yet another > third-party or imported directory inside the http directory as previously > suggested. it's annoying to have three different places for imported tests > and code, but not something I want to hold us up for a long time. > > -- Darin > > Sent from my iPhone > > On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa wrote: > > On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler wrote: > >> On Dec 4, 2013, at 6:48 AM, youenn fablet wrote: >> >> I am planning to add some XHR tests from >> https://github.com/w3c/web-platform-tests. >> My initial plan was to add them in a subdirectory of >> LayoutTests/http/tests/w3c. >> If adding them into LayoutTests/imported/w3c, that would probably >> require updating the test scripts to start/stop the HTTP test server for >> that particular sub-folder. >> >> Any preference? >> >> >> I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the >> different terminology we are using for “imported” vs. “ThirdParty”, which >> seems like the same concept at the top level of the directory structure. >> > > One trickiness to it is that we don't currently run any HTTP test in > parallel and the document root of the HTTP server is set to > LayoutTests/http/tests so we might need to modify that or restart the HTTP > server whenever we're running HTTP tests outside of LayoutTests/http. > > - R. Niwa > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler wrote: > > On Dec 4, 2013, at 6:48 AM, youenn fablet wrote: > > I am planning to add some XHR tests from > https://github.com/w3c/web-platform-tests. > My initial plan was to add them in a subdirectory of > LayoutTests/http/tests/w3c. > If adding them into LayoutTests/imported/w3c, that would probably require > updating the test scripts to start/stop the HTTP test server for that > particular sub-folder. > > Any preference? > > > I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the > different terminology we are using for “imported” vs. “ThirdParty”, which > seems like the same concept at the top level of the directory structure. > > Might be nice to think about a more elegant way to trigger the HTTP server > when needed. Even requiring some kind of trigger file in each directory > would be better than the hard-coded path rule we use now. > > I'd prefer/recommend LayoutTests/imported/w3c as well (and just modify the test scripts to deal w/ the additional http servers). As you start to pull in more of the w3c's tests, there will probably need to be some overall cleanup and refactoring of how stuff like this works and is shared (e.g., duplication of files under resources/ , paths mapped in the http server, etc.). Blink has the same issues. I'd be happy to help out with this where possible to try and keep things relatively consistent. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Proposed feature: Network Service Discovery
On Fri, Aug 30, 2013 at 3:48 PM, Oliver Hunt wrote: > > On Aug 30, 2013, at 12:44 PM, Dirk Pranke wrote: > > On Fri, Aug 30, 2013 at 10:06 AM, Oliver Hunt wrote: > >> >> On Aug 30, 2013, at 9:15 AM, Brendan Long wrote: >> >> > On 08/29/2013 05:45 PM, Benjamin Poulain wrote: >> >> Can you explain a bit what it is for? What are the common use cases? >> > This would be useful for certain kinds of web apps. For example,a music >> website like Pandora or Spotify could allow users to include music on their >> local network. Or a service like Netflix could include local network movies >> (on networked hard drives, or DVR's) in their search results, and play them >> from the same interface. >> Here's my concern - if you say "a service like " might want to search >> for something, that is better described as "a random website". That may be >> something the user wants, alternatively it could be something evil. It >> could also be something evil embedded in an ad on the site a user "trusts". >> >> My concern here is that as a web spec this essentially acts as a way for >> arbitrary web content from any source to perform a network scan of your >> local machine and get data about your internal network topology and >> services from inside your firewall. That's a really scary concept to me. >> > > While there are certainly security concerns that need to be clearly > thought through and addressed, the spec isn't as broad as you make it > sound. It picks up services that are advertising themselves, after all; you > can't probe. (Unless you've noticed something in the spec I haven't; I've > scanned the spec, but not read it super-carefully). > > > Define advertise? Bonjour like? UPnP? > > Yes (the spec explicitly lists zeroconf, upnp, and dial). > The draft does contain the sentence "Web pages should not be able to > communicate with Local-networked Services that have not been authorized by > the user thereby maintaining the user's privacy" in the use cases section; > this should definite be emphasized and fleshed out, in a security section. > > > How does the user know what they're doing? If there's an ad/unescaped > comment containing something malicious should a remote site be able to know > what services you have in your internal network? > I'm not sure I understand your question, but I'm talking about the user having to opt-in to disclosing services, similar to the opt-ins we do for geolocation, media capture, local files, etc., e.g., "Spotify would like to know if you have any local media receivers", etc. ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Proposed feature: Network Service Discovery
On Fri, Aug 30, 2013 at 10:06 AM, Oliver Hunt wrote: > > On Aug 30, 2013, at 9:15 AM, Brendan Long wrote: > > > On 08/29/2013 05:45 PM, Benjamin Poulain wrote: > >> Can you explain a bit what it is for? What are the common use cases? > > This would be useful for certain kinds of web apps. For example,a music > website like Pandora or Spotify could allow users to include music on their > local network. Or a service like Netflix could include local network movies > (on networked hard drives, or DVR's) in their search results, and play them > from the same interface. > Here's my concern - if you say "a service like " might want to search > for something, that is better described as "a random website". That may be > something the user wants, alternatively it could be something evil. It > could also be something evil embedded in an ad on the site a user "trusts". > > My concern here is that as a web spec this essentially acts as a way for > arbitrary web content from any source to perform a network scan of your > local machine and get data about your internal network topology and > services from inside your firewall. That's a really scary concept to me. > While there are certainly security concerns that need to be clearly thought through and addressed, the spec isn't as broad as you make it sound. It picks up services that are advertising themselves, after all; you can't probe. (Unless you've noticed something in the spec I haven't; I've scanned the spec, but not read it super-carefully). Another use case for this is for devices like AppleTVs and ChromeCast ... receivers advertise themselves on the local network, and the browser (and browser-based apps) can identify available receivers that you can send media to. The draft does contain the sentence "Web pages should not be able to communicate with Local-networked Services that have not been authorized by the user thereby maintaining the user's privacy" in the use cases section; this should definite be emphasized and fleshed out, in a security section. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Adding new entries to TestExpectations can be harmful
I may be missing context, or just slow, but how does that link illustrate things being harmful ? It looks like you're mostly adding the bug numbers inline and/or taking skipped tests and making them flaky ? -- Dirk On Wed, May 22, 2013 at 6:11 PM, Ryosuke Niwa wrote: > And there is an overwhelming evidence: > > http://trac.webkit.org/changeset?reponame=&new=150559%40trunk%2FLayoutTests%2Fplatform%2Fmac%2FTestExpectations&old=150476%40trunk%2FLayoutTests%2Fplatform%2Fmac%2FTestExpectations > > - R. Niwa > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] obtaining webkit.org address?
It seems like you could get nearly both things if you just had people list both their @webkit.org and @company.com addresses (or at least a company affiliation, if not an actual address) in committers.py, right? -- Dirk On Mon, Apr 29, 2013 at 10:26 AM, Brian Weinstein wrote: > What Simon said is correct. > > We would prefer that contributors use their company affiliated email for > committing. However, if there is a need for a webkit.org email (some > people don't have access to their work email when not in the office, for > example), please email me with your request for an @webkit.org email with > the username you would like and where you would like it to be forwarded. > > A further note is, if you are given a @webkit.org email and have a > company affiliation, the company affiliated email should be used for > committing, the @webkit.org email is meant for mailing lists. > > Thanks! > Brian > > On Apr 29, 2013, at 10:17 AM, Simon Fraser wrote: > > I don't think WebKit has a strict policy on this. > > I would actually prefer that we phase out webkit.org email addresses. I > like to be able to determine what someone's affiliation is. > > Simon > > On Apr 28, 2013, at 9:10 PM, Glenn Adams wrote: > > > On Sun, Apr 28, 2013 at 8:46 PM, Glenn Adams wrote: > >> And if one prefers to use a webkit.org address, like you are doing? >> > > A little follow-up: when I got my SVN account and credentials earlier this > year as a committer, I expected to be given or at least asked if I wanted a > webkit.org address, to which I would have said yes. However, I wasn't > asked and the credentials went through with my company address. As my > company is basically just me, I would prefer to use a webkit.org address > in order to identify better with the community as such. So I'm just now > following up on this inquiry about how to get a "community" address. > > >> >> >> On Sun, Apr 28, 2013 at 11:55 AM, Antonio Gomes >> wrote: >> >>> Previously people used to get SVN accounts associated to a @webkit.org >>> alias. >>> Today it seems like it is preferable to get a company email as alias, and >>> credential to write-access SVN. >>> >>> >>> On Sunday, April 28, 2013, Glenn Adams wrote: >>> How does a committer/reviewer obtain a webkit.org address? I notice that the majority of existing committers and reviewers have either a webkit.org or a chromium.org address listed in contributors.json. I think it an important part of being part of the WK community to be able to identify oneself as being in that community, and having a usable webkit.org address seems a good way to effect that. G. >>> >> > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 4:00 PM, Darin Adler wrote: > On Apr 24, 2013, at 3:52 PM, Dirk Pranke wrote: > > > You mean it's easy to write a test that is bigger than 800x600, and > perhaps we should generate some sort of warning for this? > > It’s easy to write a test where part of what we’re trying to test is > outside 800x600, right. And it would be good if this mistake was easy to > notice. > > Yeah, seems like we could do either or both of modify DRT/WTR to draw a big border at 800x600 and log errors to stdout/stderr if the page wanted to be overflow that. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 3:49 PM, Darin Adler wrote: > On Apr 24, 2013, at 3:45 PM, Dirk Pranke wrote: > > > in the absence of tests that *need* bigger viewports, I'd side w/ David > and say that we should fix tests to fit within 800x600 so that they are > portable to other browsers and meet the W3C's criteria. > > Makes sense. > > I believe the 800x600 viewport makes it easy to accidentally write > incorrect ref tests that look good when you read the HTML in the test file > and the expected file, but are no good. It would be nice to find a simple > way to spot flawed ref tests easily so even an inexperienced beginner will > notice such mistakes. > I'm not sure I'm understanding you here. You mean it's easy to write a test that is bigger than 800x600, and perhaps we should generate some sort of warning for this? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 3:35 PM, Elliott Sprehn wrote: > > On Wed, Apr 24, 2013 at 3:31 PM, Dirk Pranke wrote: > >> >> >> >> On Wed, Apr 24, 2013 at 3:29 PM, Elliott Sprehn wrote: >> >>> On Wed, Apr 24, 2013 at 11:11 AM, Dirk Pranke wrote: >>> >>>> >>>>> ... >>>>> >>>>> >>>> I think this raises two related questions, both of which would be >>>> helped by concrete examples: >>>> >>>> 1) What sort of test are you writing where 800x600 isn't big enough to >>>> test what you need to test? >>>> >>>> 2) What sort of test are you writing where there's needs to be >>>> something on the page not included in the test >>>> *and* will render differently in different browsers? >>>> >>>> I can posit the existence of both sorts of things, but I'm not sure >>>> when they arise in practice? >>>> >>>> >>>> >>> I've seen tests before that have lots of form controls where a bunch >>> were outside the viewport. >>> >> >> Sure, you *can* write such tests. Did you *have to*? >> >> > Certainly not, I just know such tests already exist. > > Right, so in the absence of tests that *need* bigger viewports, I'd side w/ David and say that we should fix tests to fit within 800x600 so that they are portable to other browsers and meet the W3C's criteria. Looking at the entire page would be a bad idea, as it would let us tolerate non-conformant tests. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 3:29 PM, Elliott Sprehn wrote: > On Wed, Apr 24, 2013 at 11:11 AM, Dirk Pranke wrote: > >> >>> ... >>> >>> >> I think this raises two related questions, both of which would be helped >> by concrete examples: >> >> 1) What sort of test are you writing where 800x600 isn't big enough to >> test what you need to test? >> >> 2) What sort of test are you writing where there's needs to be something >> on the page not included in the test >> *and* will render differently in different browsers? >> >> I can posit the existence of both sorts of things, but I'm not sure when >> they arise in practice? >> >> >> > I've seen tests before that have lots of form controls where a bunch were > outside the viewport. > Sure, you *can* write such tests. Did you *have to*? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 10:30 AM, L. David Baron wrote: > On Wednesday 2013-04-24 09:55 -0700, Darin Adler wrote: > > On Apr 23, 2013, at 11:36 AM, Benjamin Poulain > wrote: > > > > > The current mode works fine for some ref-tests but is really annoying > for others. I would love to have a way to compare the entire page instead > of the viewport. > > > > > > Maybe it could just be an API on testrunner? Like: > > > testRunner.dumpEntirePagePixels()? > > > > What’s the downside of just doing this for all reftests? Would it > > make them slower? Would it cause failures? I’d love it if we could > > just do it without it having to be opt-in. > > Tests that rely on this wouldn't be sharable with other browsers or > usable in W3C test suites, at least based on current practice. > > (I think we had some sort of agreement within some set of people > involved with W3C reftests to use 800x600. Mozilla is currently > using 800x1000, though I'd like to switch to the smaller size.) > > I think this raises two related questions, both of which would be helped by concrete examples: 1) What sort of test are you writing where 800x600 isn't big enough to test what you need to test? 2) What sort of test are you writing where there's needs to be something on the page not included in the test *and* will render differently in different browsers? I can posit the existence of both sorts of things, but I'm not sure when they arise in practice? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Apr 16, 2013 at 3:58 PM, Brent Fulgham wrote: > Hi Daniel, > > I'm afraid I don't quite understand the nature of the change you are > proposing: > > 1. Is it sufficient to supply the full path to the include files (e.g., > change "Foo.h" to "WebCore/html/Foo.h") to achieve these gains? > 2. ... or ... is it sufficient to copy all header files to a common > include target path (e.g., "$(BuildDir)/include") to achieve these gains? > > I understand that in either case, we would want to remove the various > compiler include path directives so that there are fewer places for it to > search. > > I believe he's proposing the former -- (1) -- along with the necessary changes to all the code generation scripts to do the right thing. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Layout Tests and the W3C
Hi all, Those of you who are subscribed to blink-dev@ will see that I just sent out a note entitled "Blink, Testing and the W3C", describing stuff I'm working on to get the W3C tests running regularly as part of the layout tests. The W3C has gotten to the point where they have many thousands of ref tests, a lot of which are newer versions of old test suites that we imported once upon a time (meaning that we can replace a lot of pixel tests with ref tests). That work leverages a bunch of work Rebecca Hauck has been working on that actually started pre-Blink-fork, and in fact currently only exists as a patch posted to bugs.webkit.org ( https://bugs.webkit.org/show_bug.cgi?id=111513 ). I would like to get something resembling the same process working for both WebKit and Blink, since we use basically the same tools. There will probably be some differences, of course, e.g., in how we mirror repos and track issues, but I would hope we'd all agree the more tests we can share (both between WebKit and Blink and with other vendors like FF and IE) the happier everyone will be, right? We haven't really worked out the processes for how this'll work yet; Rebecca and I were starting to work on it when the fork happened. Assuming there is still interest, I'll start sketching out how things might work for WebKit and report back. Also, if we actually start to make progress on this, I'd be happy to talk about it at the WebKit committer's meeting if there's interest there as well. Thoughts? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Parallel JavaScript: Why a separate ParallelArray types
On Fri, Apr 12, 2013 at 10:44 PM, Filip Pizlo wrote: > > On Apr 12, 2013, at 10:43 PM, Ryosuke Niwa wrote: > > Perhaps we can come up with some JS API for shared memory & lock and > propose it in TC39 or WebApps WG? > > > I think that would be wonderful. Anyone else interested? > > Feel free to write something up, but as I said before, I suspect any such proposal is DOA in either group. If you wanted to get something adopted here, I think you would have to build something, ship it, and then have it become so successful that the standards groups would be forced to reconsider their positions. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Parallel JavaScript: Why a separate ParallelArray types
On Fri, Apr 12, 2013 at 1:50 PM, Filip Pizlo wrote: > I'm curious: would you want to use ParallelArray, if you had the > flexibility of building a different abstraction? > > I find ParallelArray to be an awkward abstraction to begin with. I like > work queues and such. The whole idea that the only parallelism you get is > from arrays feels narrow-minded. > I think the needs of the HPC / GPU-accelerating crowd are quite different from the people looking for general concurrency solutions to the multi-core problem. For what it's worth, NRWT is (IMO) an interesting example of a highly-parallel program. In a sense, the problem is perfectly suited to map/reduce, and the performance of NRWT scales linearly well up to (at least) 48-core machines; the necessarily-serial part of the program is quite small. However, in order to turn it into a usable program (where you can actually tell what's going on and manage it well), you do need to actually monitor tasks and queues and provide problem-specific feedback to the user. I'm not sure what the message of that is, exactly ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
On Wed, Apr 10, 2013 at 1:17 PM, Filip Pizlo wrote: > I.e. if you believe that (A) is not a solvable problem, then this > constitutes an argument against ParallelArrays, and not against inferring > that a normal array should behave like a ParallelArray. > > I believe that there are a class of solvable problems for soundness, but it's hard if not impossible to do it generally; that was my main concern. There are certainly languages that make this problem easier, for example (Haskell, J). I agree that the JIT+runtime bailout strategy you discuss gives you more options here. > - Adding hints to the language results in programming models that nobody > uses. Have you ever used OpenMP? > > I'm familiar enough with it. I was referring to things more like "restrict", "const", and "pure" which give you (or the compiler) more guarantees about what it is safe to infer about the parameters and functions. I'm not sure these are the same sorts of things you don't like. > > Hope my answers help! :-) > > Yes, thank you for expounding. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Rewriting binding code generator, maybe?
On Thu, Apr 11, 2013 at 10:29 AM, Leandro Pereira wrote: > dpranke, > > > On 04/11/2013 02:12 PM, Dirk Pranke wrote: > > >> I'm not sure how useful a suggestion this is, but ANTLR has a pretty >> strong framework for separating parsing from code generation and seems >> like it would be an ideal fit for this, except that the tool itself is >> written in Java. Perhaps that is a show-stopper? I'm not actually aware >> of any similar parser-generator tools that have similar architectures >> but are in more WebKit/Blink-friendly languages, but maybe they exist? >> > > Pyparsing is a nice little library that requires no code generation > (grammar is written in pure Python) and is very simple to use. > > Their website lives at > http://pyparsing.wikispaces.**com/<http://pyparsing.wikispaces.com/> > . > > That's pretty much the opposite of what I was getting at (I was talking about ANTLR's use of StringTemplate to separate code generation from the building of the AST). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Rewriting binding code generator, maybe?
On Thu, Apr 11, 2013 at 9:51 AM, Darin Adler wrote: > On Apr 11, 2013, at 5:02 AM, Thiago Marcos P. Santos > wrote: > > > Would be great if both projects could share the same generators. > > The part that would be OK to share would be the parsing. > > The actual code generation needs to change all the time as the internal > architecture of WebKit and of the JavaScript engine and other things we’re > binding change, so we should not try to change that. > > I'm not sure how useful a suggestion this is, but ANTLR has a pretty strong framework for separating parsing from code generation and seems like it would be an ideal fit for this, except that the tool itself is written in Java. Perhaps that is a show-stopper? I'm not actually aware of any similar parser-generator tools that have similar architectures but are in more WebKit/Blink-friendly languages, but maybe they exist? It certainly would be nice not to have to roll your own framework here. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Rewriting binding code generator, maybe?
On Wed, Apr 10, 2013 at 4:59 PM, Timothy Hatcher wrote: > > On Apr 10, 2013, at 6:32 PM, Ryosuke Niwa wrote: > > Can we rewrite CodeGenerator*.pm in Ruby or Python? I feel that the > current code is very hard to understand and hack on. In particular we have > CodeGenerator.pm and CodeGeneratorJS.pm (CodeGeneratorV8.pm has been > removed), and we need to merge them anyway. > > > They can't be merged. We also have CodeGeneratorObjC.pm. (And internally > we have CodeGeneratorSafari.pm.) > > > I've seen many people expressing their inability to improve the binding > code because of its being written in Perl. > > > I've also seen people express their frustration for our tools currently > written in Ruby and Python. I personally find Perl fine for this task and > don't see a need to rewrite things just because. > > I'm sure Perl is a fine tool for the job, if you happen to know Perl, but Perl code is generally much less approachable than Ruby, and Ruby less so than Python. Of course many if not most of the Python zealots have gone elsewhere :). I believe in Blink-land there may be a reasonably concerted effort to move the stuff Chromium still uses to Python where possible, and obviously this is one such case (as Eric alludes to earlier), so it does seem like it would be nice to be able share the code and make the rewrite generic if it wasn't too much additional work, there was interest in doing so, and the rewrite was gonna happen anyway. Otherwise I'm quite sympathetic to the "ain't broke" argument here :) -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
Right, you need some way to ensure that the functions applied to the array
are "pure", the array doesn't mutate, etc. I thought perhaps your
ParallelArray() type would be ensuring this (e.g., by enforcing type
checking and freezing the array), but maybe I misunderstood you? Even then,
I'm not sure how easy ensuring the functions were pure would be.
Plus there's the problem of figuring how best to parallelize the
operations, which is also often non-trivial.
-- Dirk
On Wed, Apr 10, 2013 at 12:58 PM, Oliver Hunt wrote:
> I'm not sure what you mean here: a parallel array is simply an array of
> numbers; there is nothing "magic" about it.
>
> If we _really_ wanted to be blunt about it, we could simply say (in
> Array.map)
>
>
> if (length >= giant && function.isEasyPeasy() && everything in the array
> is a number)
>return doHardcoreMap();
>
> // carry on doing regular map
> ...
>
>
> Understand that the parallel APIs have to do all of this as well, the only
> thing they lose is the "is everything a number" check
>
> --Oliver
>
>
> On Apr 10, 2013, at 12:50 PM, Dirk Pranke wrote:
>
> Right, I get how ParallelArrays would work. I was asking Filip how you
> might infer that an array could be a ParallelArray.
>
> -- Dirk
>
> On Wed, Apr 10, 2013 at 12:45 PM, Oliver Hunt wrote:
>
>> The parallel arrays apis aren't a magic "make my code parallel" wand.
>>
>> They don't make general code parallel, they simply provide a bunch of
>> functions like map, etc where if you restrict your list of language
>> features to a specific subset, they'll vectorise it.
>>
>> For instance
>>
>> ParallelArray([1,2,3]).map(function(a,v) { return v * 2; })
>>
>> would be vectorised
>>
>> ParallelArray([1,2,3]).map(function(a,v) { return v.x * 2; })
>>
>> wouldn't be.
>>
>> This isn't "solve autovectorisation", this is "we've been given a
>> specific operation to perform on each element of an array, parallelise it
>> if possible", and what's possible is arbitrarily limited, and require a
>> distinct type that isn't an array.
>>
>> --Oliver
>>
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
Right, I get how ParallelArrays would work. I was asking Filip how you
might infer that an array could be a ParallelArray.
-- Dirk
On Wed, Apr 10, 2013 at 12:45 PM, Oliver Hunt wrote:
> The parallel arrays apis aren't a magic "make my code parallel" wand.
>
> They don't make general code parallel, they simply provide a bunch of
> functions like map, etc where if you restrict your list of language
> features to a specific subset, they'll vectorise it.
>
> For instance
>
> ParallelArray([1,2,3]).map(function(a,v) { return v * 2; })
>
> would be vectorised
>
> ParallelArray([1,2,3]).map(function(a,v) { return v.x * 2; })
>
> wouldn't be.
>
> This isn't "solve autovectorisation", this is "we've been given a specific
> operation to perform on each element of an array, parallelise it if
> possible", and what's possible is arbitrarily limited, and require a
> distinct type that isn't an array.
>
> --Oliver
>
> On Apr 10, 2013, at 12:37 PM, Dirk Pranke wrote:
>
>
>
>
> On Wed, Apr 10, 2013 at 7:37 AM, Filip Pizlo wrote:
>
>>
>>
>> On Apr 10, 2013, at 2:29 AM, "Pozdnyakov, Mikhail" <
>> mikhail.pozdnya...@intel.com> wrote:
>>
>> >
>> >
>> >> Why not infer ParallelArrays automatically?
>> > Sorry, did not get it. Could you please elaborate?
>>
>> Normal JS arrays. You use them with pure computation. Profiler notices
>> this. And then the same things that you have in the ParallelArrays proposal
>> now work except you don't need a new type.
>>
>>
> Compiler writers have been chasing this problem for 50 years, with (AFAIK)
> very limited success for languages that don't give you compile-time
> annotations to help you along. I'm not aware of any work on JIT compilers
> doing this. Why do you think we could do better? (Honest question, I'm not
> trying to be snarky, nor am I an expert).
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
On Wed, Apr 10, 2013 at 7:37 AM, Filip Pizlo wrote: > > > On Apr 10, 2013, at 2:29 AM, "Pozdnyakov, Mikhail" < > mikhail.pozdnya...@intel.com> wrote: > > > > > > >> Why not infer ParallelArrays automatically? > > Sorry, did not get it. Could you please elaborate? > > Normal JS arrays. You use them with pure computation. Profiler notices > this. And then the same things that you have in the ParallelArrays proposal > now work except you don't need a new type. > > Compiler writers have been chasing this problem for 50 years, with (AFAIK) very limited success for languages that don't give you compile-time annotations to help you along. I'm not aware of any work on JIT compilers doing this. Why do you think we could do better? (Honest question, I'm not trying to be snarky, nor am I an expert). ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Tools/BuildSlaveSupport/.../TestFailures/run-unittests.html
TestFailures, as I understand it, provides a summarize view of current tree failures clustered by the changes that might have caused the failures, kind of like garden-o-matic's initial page. I believe I broke it at one point accidentally, asked if anyone used it, and Ossy said that he did (there might've been others). To my shame, I never got around to fixing my breakage, and it looks kinda still broken: http://build.webkit.org/TestFailures/ Seems like it would be good to fix it :). I think aroben built it and it has been neglected since he stopped working on WebKit full time. -- Dirk On Tue, Apr 9, 2013 at 10:08 PM, Glenn Adams wrote: > Does any bot (or person) run this? Regularly? I just noticed today it > reports a number of failures (using r148074). > > Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.37+ (KHTML, > like Gecko) Version/6.0.3 Safari/536.28.10 > > Tests completed in 705 milliseconds. > 513 tests of 545 passed, 32 failed. > > > > May I assume there would be no objection if I attempt to fix those > failures? > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Cleaning House
On Thu, Apr 4, 2013 at 12:50 PM, Geoffrey Garen wrote: > To clarify: > > (1) The EWS bots are still running. > > (2) The mac and mac-wk2 EWS bots are running tests, and passing. > > (3) The cr-linux bots are running tests, and failing. > > If we're OK with item (3), we can go ahead with cleaning house, and break > the cr-* EWS bots entirely, while we work on making the mac and mac-wk2 EWS > bots faster. > > I've posted a patch to turn off the tests on the cr-* EWS bots (though we'll still try to compile). https://bugs.webkit.org/show_bug.cgi?id=113959. Obviously, we can simply turn the bots off whenever they're no longer useful as well. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Cleaning House
On Thu, Apr 4, 2013 at 12:30 AM, Geoffrey Garen wrote: > Hi folks. > > Since we no longer need to support the Chromium port, let's take the > opportunity to streamline. Hopefully, this will make development easier and > more coherent for everyone. Adam and Eric offered to do some of this > cleanup, but I think it's healthier for people who will continue to be a > part of WebKit to decide what gets cleaned up, and execute on the plan. > > Below is a high-level view of some improvements we're planning over the > coming weeks. > > Concepts we plan to remove: > Layering violations in WebCore/platform, where a Page* or Frame* > is passed to a function > Supplementable and Supplement > #if USE(GOOGLEURL) > #if USE(V8) > #if !USE(JSC) > #if PLATFORM(CHROMIUM) > Skia > DOMFileSystem > WebLayer and its scrolling implementation > Features #defines that haven't gained traction > > Specific files we plan to remove: > .gyp build files > WebCore/bindings/v8 > WebCore/bindings/scripts/*v8* > LayoutTests/platform/chromium* > WebKit/chromium > WTF/wtf/chromium > WebCore/platform/chromium > WebCore/*Chromium* > Source/Platform/chromium > ManualTests/chromium/ > Tools/BuildSlaveSupport/chromium/ > Tools/DumpRenderTree/chromium/ > > Also: > Adopt libc++ > > Please let me know if you have other suggestions for improvements, or if > you see something in the list that shouldn't be there. > > FWIW, mrobinson has been working on a GYP build for the GTK port, so I wouldn't delete all of the .gyp files (at least not w/o them weighing in on it). I thought there was some interest at Apple in also using GYP, but perhaps things have (or will) change given that interop w/ Chromium is no longer the big plus it would've been. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Mar 26, 2013 at 1:42 PM, Alexis Menard wrote: > On Tue, Mar 26, 2013 at 5:40 PM, Dirk Pranke wrote: > > On Tue, Mar 26, 2013 at 1:29 PM, Benjamin Poulain > > wrote: > >> > >> On Tue, Mar 26, 2013 at 1:20 PM, Dirk Pranke > wrote > >>> > >>> If we have consensus that we should just switch to paths relative to > >>> Source (or maybe a couple different options), that would be (IMO) a > big win. > >>> It sounds like Daniel & co. have already done the big bang conversion. > >> > >> > >> I think using full path would be a serious hit regarding hackability. > >> > >> I would rather spend some time tweaking my compiler to cache each > >> directory content than waste time finding where is every single header I > >> need to include. > >> > > > > Interesting. I have the exact opposite experience, having to paw around > to > > figure out where "Font.h" actually lives rather than just seeing > > "WebCore/platform/graphics/Font.h". > > But a modern IDE can easily solve that for you. You just click on the > header and it opens it for you. Xcode, QtCreator handles it very well. > > That said if you don't use an IDE then I understand your point :). > > I think to me the fact that the paths are spelled out is both one less action I have to do to get my question answered (I don't have to click), and a helpful reminder of the general layout of the code (and which directories I might need to care about in relation to a particular file). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Mar 26, 2013 at 1:29 PM, Benjamin Poulain wrote: > On Tue, Mar 26, 2013 at 1:20 PM, Dirk Pranke wrote > >> If we have consensus that we should just switch to paths relative to >> Source (or maybe a couple different options), that would be (IMO) a big >> win. It sounds like Daniel & co. have already done the big bang conversion. >> > > I think using full path would be a serious hit regarding hackability. > > I would rather spend some time tweaking my compiler to cache each > directory content than waste time finding where is every single header I > need to include. > > Interesting. I have the exact opposite experience, having to paw around to figure out where "Font.h" actually lives rather than just seeing "WebCore/platform/graphics/Font.h". At any rate, to be clear, I would be in favor of that change but I'm not expecting it to happen :). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Mar 26, 2013 at 11:47 AM, Ryosuke Niwa wrote: > On Tue, Mar 26, 2013 at 11:42 AM, Jarred Nicholls wrote: > >> On Tue, Mar 26, 2013 at 2:37 PM, Ryosuke Niwa wrote: >> >>> On Tue, Mar 26, 2013 at 11:21 AM, Daniel Bratell wrote: >>> Is this something that has been talked about in the past, and would you be interested in replacing the long list of directories to search for every include with paths (relative some good base) directly in the include directives? >>> >>> Using explicit paths to include files has been talked about in the past; >>> e.g. >>> https://lists.webkit.org/pipermail/webkit-dev/2011-November/018632.html >>> >>> The most convincing counter argument I can remember is that it'll make >>> refactoring harder because you'll have to update all #include's when you >>> move headers. >>> >> >> A few tailored scripts make that an easier job. We're all engineers >> after all. >> > > We have been talking about this for years, and we even have bugs: > https://bugs.webkit.org/show_bug.cgi?id=61772 > Given that, I'm somewhat skeptical if we ever get around to it in > foreseeable future. > > Scripts that update include paths from Source/WebCore/foo.h to Source/Platform/foo.h are substantially easier than something to modify XCode projects robustly ;) If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel & co. have already done the big bang conversion. On Tue, Mar 26, 2013 at 11:58 AM, Geoffrey Garen wrote: > > In WebKit include directives are without path, and instead the compiler > is given a very long list of directories to search through. That process > takes a lot of time in Windows. It must take some time in OSX and in Linux > too but probably less. > > Can we make a first-order improvement just by making sure that paths are > searched in order of likelihood? > That actually sounds harder and more fragile to me than switch to full paths across the board. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New script to import W3C CSS tests
On Thu, Mar 7, 2013 at 3:04 PM, Tony Gentilcore wrote: >> I also got a little inspiration from the >> import-w3c-performance-wg-tests that already exists. I followed a few of >> their steps, but had to add a few layers to handle the added complexity of >> the CSS test suites. > > Is there any way we can merge the two scripts so there is only one > import/exporter than handles both performance and CSS test suites? > That is a good question, and something I'll be looking at as part of this :). More generally, I'm trying to look into importing and running tests from the w3c across the board as one of my next projects. Rebecca beat me to it! -- Dirk > -Tony > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New script to import W3C CSS tests
On Thu, Mar 7, 2013 at 8:46 AM, Rebecca Hauck wrote: > Hi Dirk & Everyone, > > Dirk, your question about this in IRC the other day was timely indeed. > > I've recently written a batch of tests for CSS Regions that I intend to > submit to both the W3C and Webkit (and have planned more to come). Rather > than push two slightly different versions of the same tests to each repo, > and to address how these tests are to be maintained over time, I took a stab > automating the import and just submitted a patch [1]. > > I know this conversation was started a while ago as I attended the WK > Contributor's meeting last spring where my colleagues Jacob Goldstein and > Alan Stearns led the discussion. I was sort of new onto the scene then and > was pretty much a lurker in the meeting, so you probably don't remember me. > I recall that much of the debate was about handling failing tests, approved > vs. submitted, duplicates, etc. Since all I've done is automate the > mechanics of the import itself, I have not solved a lot of those issues, but > the conversion of tests certainly had its own set of things to consider. I > figured I would put this one piece in place, get feedback on it, and > hopefully resurrect the discussion on fitting this into the bigger picture. > A great goal would be to have this script run on an automated schedule and > to extend it to deal with test results, expected failures, etc. > > All of the dirty details are in the ChangeLog of the patch and in the script > headers, but I basically tried to cover #8 from last spring's meeting notes > [2]. I think the script covers all of those bullets, with that last bullet > as "sort of." I also got a little inspiration from the > import-w3c-performance-wg-tests that already exists. I followed a few of > their steps, but had to add a few layers to handle the added complexity of > the CSS test suites. > > Dirk, when you have a chance, take a look. Review it with gusto as you say. > And anyone else who's interested, your feedback is welcome. I fairly > certain this will go a few rounds before it's settled. :) > Thanks, Rebecca! It will be great if we can start regularly importing and running the W3C's tests. (I'll review the patch and post any other comments on the bug). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Tue, Feb 26, 2013 at 1:03 PM, Ryosuke Niwa wrote: > On Tue, Feb 26, 2013 at 12:47 PM, Dirk Pranke wrote: >> >> On Tue, Feb 26, 2013 at 2:11 AM, Ryosuke Niwa wrote: >> > On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson >> > wrote: >> >> >> >> On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa >> >> wrote: >> >>> >> >>> It should be fairly straight forward to create a tool that analyzes >> >>> files >> >>> changed in each commit and deduce which tests' expected results have >> >>> been >> >>> changed. The tool can then fetch results from each port' bot for those >> >>> tests >> >>> and automatically land them. It can then comment on the bug >> >>> automatically >> >>> about these rebaseline commits. There is no need to add & remove >> >>> entries >> >>> from TestExpectation files. >> >> Wait, what? >> >> For some reason neither I nor the mailing list archives got your >> initial message, nor Silvia or Tom's responses, nor your responses >> (at least as of the time of me writing this), so I feel like I've >> missed a radical shift in this thread, and maybe I missed some of the >> context. > > > https://lists.webkit.org/pipermail/webkit-dev/2013-February/023967.html > This link doesn't point to any of those messages, but perhaps it's not that important. >> You're proposing that we automatically land updated baselines without >> review and then somehow update bugs, have people go back and look at >> the updated bugs to see if the baseline changes represent actual >> regressions or just expected changes? > > > Right. Given that the commit already contains information as to which tests > have been rebaselined, a script should be able to fetch new baselines for > those affected tests on each platform and land them or upload as patches as > needed. > It's possible that we could fetch and cluster new baselines based on what changed in the initial commit. I would be concerned that there could be a fair amount of noise in either direction (tests that changed on the initial platform didn't on others, and others did), and you'd also have to figure out how to cluster changes since most builds on the bots contain multiple changes. But, you could probably use some of garden-o-matic's results to help here. That said, I'm not sure this workflow would actually improve things much over garden-o-matic. I am quite a bit more reluctant to automatically land any such changes; it seems like it would be hard if not impossible to tell (programmatically) whether a baseline changed as expected or if it represented a regression. If we were to work on new tooling, I would be much more in favor of pushing this up to an EWS-time step like Ossy suggests. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Tue, Feb 26, 2013 at 2:11 AM, Ryosuke Niwa wrote: > On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson wrote: >> >> On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa wrote: >>> >>> It should be fairly straight forward to create a tool that analyzes files >>> changed in each commit and deduce which tests' expected results have been >>> changed. The tool can then fetch results from each port' bot for those tests >>> and automatically land them. It can then comment on the bug automatically >>> about these rebaseline commits. There is no need to add & remove entries >>> from TestExpectation files. >> Wait, what? For some reason neither I nor the mailing list archives got your initial message, nor Silvia or Tom's responses, nor your responses (at least as of the time of me writing this), so I feel like I've missed a radical shift in this thread, and maybe I missed some of the context. You're proposing that we automatically land updated baselines without review and then somehow update bugs, have people go back and look at the updated bugs to see if the baseline changes represent actual regressions or just expected changes? Can you walk me through how this would work in more detail? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 2:05 PM, Ryosuke Niwa wrote: > On Mon, Feb 25, 2013 at 1:39 PM, Dirk Pranke wrote: >> Are you suggesting we should land a "failling" baseline in the meantime? > > > No. I'm suggesting patch authors perform their due diligence and either ask > port maintainers to rebaseline or rebaseline tests themselves. > I think either you misunderstood my question, or I am misunderstanding your answer. I'm not asking "who", I'm asking "what" ... If we know some tests are failing, and when we fix a bug the tests will start passing again (but that patch might not land for quite some time), what should we (anyone) do in the meantime? Leave the tree red, land "incorrect" -expected baselines so that we can catch changes in behavior, or add lines to TestExpectations? Many of the lines you cited fell into the last category. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 1:33 PM, Ryosuke Niwa wrote: > On Mon, Feb 25, 2013 at 1:29 PM, Dirk Pranke wrote: >> >> On Mon, Feb 25, 2013 at 1:22 PM, Ryosuke Niwa wrote: >> > On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke >> > wrote: >> > >> >> > I remember some discussion of a [rebaseline] keyword in >> >> > TestExpectations, >> >> > but I'm not sure that ever made it in? >> >> >> >> The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and >> >> unclaimed [2]. It shouldn't be too hard for someone to try it if >> >> they're looking for a reason to explore the NRWT code :) >> > >> > >> > I object to adding such a thing. People add and forget about these >> > entries >> > way too often: >> > >> >> The rationale for adding the keyword was precisely this ... it would >> make it easier to programmatically identify tests needing updated >> baselines and alert people :). >> >> Do you have an alternative you'd suggest for keeping people from >> forgetting (particularly in the case where we have an expected failure >> waiting on a bug fix)? > > Just turn bots red. People maintaining bots will notice and rebaseline them > as needed. > If there's a significant time lapse between failing test and bug fix, it seems like a bad idea to leave the tree red. Are you suggesting we should land a "failling" baseline in the meantime? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 1:22 PM, Ryosuke Niwa wrote: > On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke wrote: >> >> On Mon, Feb 25, 2013 at 12:38 PM, Eric Seidel wrote: >> > I've noticed as of late several different approaches being used when >> > adding/changing LayoutTests which need rebaselining on other platforms. >> > >> > Obviously we cannot expect developers to test/rebaseline on all >> > platforms >> > before landing given our current infrastructure. >> > >> > But what should we expect them to do? >> > >> > Currently some folks add failing expectations to other ports >> > TestExpectations. Some add [skip]. Some even add them to the (new) >> > global >> > TestExpectations file. >> > >> > What's the proper course of action? >> > >> > I checked: >> > http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green >> > >> > http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches >> > http://trac.webkit.org/wiki/TestExpectations >> > and didn't see the "I expect this to fail/need rebaseline on other >> > ports" >> > case discussed. >> > >> >> Memory says that the last time this was raised [1], the consensus was >> to land the change, turn the tree red, notify as many gardeners / port >> maintainers as possible, and rebaseline as quickly as possible. I.e., >> don't add entries to TestExpectations. Of course, such a policy >> wouldn't play nicely w/ the EWS bots, and I think this discussion >> predated everyone switching over to TestExpectations (and certainly >> predated the generic expectations file). >> >> I don't find the above entirely satisfactory, but I also don't have >> any great alternatives to endorse given the existing tooling. >> >> Also, if the test fails generically (all ports), it probably shouldn't >> be landed. I'm not sure why you couldn't at least update the port >> you're developing on; if you don't, how do you know your fix is >> working at all? (Of course, there could be a case where a >> high-priority fix might break some low-priority tests). > > > Adding a new feature behind a build flag disabled everywhere by default. > >> > I remember some discussion of a [rebaseline] keyword in >> > TestExpectations, >> > but I'm not sure that ever made it in? >> >> The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and >> unclaimed [2]. It shouldn't be too hard for someone to try it if >> they're looking for a reason to explore the NRWT code :) > > > I object to adding such a thing. People add and forget about these entries > way too often: > The rationale for adding the keyword was precisely this ... it would make it easier to programmatically identify tests needing updated baselines and alert people :). Do you have an alternative you'd suggest for keeping people from forgetting (particularly in the case where we have an expected failure waiting on a bug fix)? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 12:38 PM, Eric Seidel wrote: > I've noticed as of late several different approaches being used when > adding/changing LayoutTests which need rebaselining on other platforms. > > Obviously we cannot expect developers to test/rebaseline on all platforms > before landing given our current infrastructure. > > But what should we expect them to do? > > Currently some folks add failing expectations to other ports > TestExpectations. Some add [skip]. Some even add them to the (new) global > TestExpectations file. > > What's the proper course of action? > > I checked: > http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green > http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches > http://trac.webkit.org/wiki/TestExpectations > and didn't see the "I expect this to fail/need rebaseline on other ports" > case discussed. > Memory says that the last time this was raised [1], the consensus was to land the change, turn the tree red, notify as many gardeners / port maintainers as possible, and rebaseline as quickly as possible. I.e., don't add entries to TestExpectations. Of course, such a policy wouldn't play nicely w/ the EWS bots, and I think this discussion predated everyone switching over to TestExpectations (and certainly predated the generic expectations file). I don't find the above entirely satisfactory, but I also don't have any great alternatives to endorse given the existing tooling. Also, if the test fails generically (all ports), it probably shouldn't be landed. I'm not sure why you couldn't at least update the port you're developing on; if you don't, how do you know your fix is working at all? (Of course, there could be a case where a high-priority fix might break some low-priority tests). > I remember some discussion of a [rebaseline] keyword in TestExpectations, > but I'm not sure that ever made it in? The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and unclaimed [2]. It shouldn't be too hard for someone to try it if they're looking for a reason to explore the NRWT code :) -- Dirk [1] https://lists.webkit.org/pipermail/webkit-dev/2012-April/020250.html [2] https://bugs.webkit.org/show_bug.cgi?id=100415 ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Getting involved
Hi Jason, On http://trac.webkit.org/wiki there are a bunch of pages under "Layout Tests" and "Performance Tests" that describe much of the infrastructure. These days most of it is written in Python. I don't know what projects Benjamin has in mind so I don't know if that will match what he was referring to, but hopefully this will be helpful. -- Dirk On Mon, Feb 18, 2013 at 3:44 PM, Jason Anderssen wrote: > Hello Benjamin, > > Thank you for a starting point and would like to explore further with this. > Please point me in the right direction to how I can help in the cleaning of > the WebKit testing infrastructure please. > This will also allow me to understand how all you guys work together and > understand the procedures etc.. > > Cheers > Jason > > > From: Benjamin Poulain > Date: Tue, 19 Feb 2013 08:52:17 +1000 > > To: Jason Anderssen > Cc: "webkit-dev@lists.webkit.org" > Subject: Re: [webkit-dev] Getting involved > > Hi Jason, > > On Mon, Feb 18, 2013 at 2:45 PM, Jason Anderssen > wrote: >> >> With a background in development (both windows and MAC) I was wanting to >> get involved in an open source project that I believe is going to make the >> world a greater place. So I have decided to get involved with the WebKit >> project. >> I would like to start of by performing the boring work of reformatting >> code to suite the guidelines as stipulated on the webkit.org website (and >> double check that they are still up to date), and would like recommendations >> to which parts of the project could do with the re-formatting first up. ( I >> feel this will get me to know the framework, and structure of the code base >> well, before I can start inputting my coding and help. ) >> >> Any feedback to where to start would be very much appreciated. >> >> Also I have the code all built, however if I ./run-safari I get >> >> Can't find built framework at >> "/Users/jason/research/webkit/WebKitBuild/Release/JavaScriptCore.framework/Versions/A/JavaScriptCore." >> >> I will work out the problem, but in case someone knows exactly what I have >> done wrong, it may save me some time. > > > First welcome to WebKit :) > > I am sure there is a better use of your time than fixing the coding style. > > A side project of mine is improving and cleaning some of WebKit testing > infrastructure. There are some fairly easy tasks related to that if you are > interested. > I am sure others will have ideas of easy things to do to get started. > > For your build/run issues, it will be easier to ask on IRC #webkit than on > the mailing list. > > Benjamin > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain wrote:
> On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke wrote:
>>
>> > Those changes are not harmless. There are people monitoring tests
>> > results
>> > full time in order to keep WebKit in good shape. No other part of WebKit
>> > require continuous attention.
>>
>> I'm sorry, but either I don't understand Dongsung's suggestion, or I
>> don't understand your criticism. What does this sentence paragraph
>> mean? Are you suggesting someone needs to look at this type of test
>> full time in a way that we don't have to look at the other tests?
>
>
> My lack of sleep is likely to blame here :)
>
> All I am saying is tests have a hight cost when they start failing, we
> should be careful how we design them so we can easily triage bad tests,
> flaky tests, and real errors.
>
No argument here :).
>
>> >> Case 1: CSS Filters & Shaders
>> >> I wanted this test functionality when I commented
>> >> http://webkit.org/b/97859#c19
>> >> If I want to make gaussian blur test, I prefer using 'getPixel' test as
>> >> follows,
>> >
>> >
>> > Why wasn't a ref-test a better solution in this particular case?
>> >
>>
>> I'm curious, what would you imagine the ref test contains?
>
>
> If I am not mistaken, the composition operations are parallel with the ones
> of SVG and Canvas (aren't they?).
> I would have attempted comparing the 3 implementations as it seems to me the
> pixels values should be the same.
>
> That was a genuine question about that case :)
>
>>
>>
>> >> Case 2: Fixed Position Element
>> >> [...]
>> >> function repeatedlyCalledDuringScrolling() {
>> >> ASSERT(getPixel(15, 9) == white);
>> >> ASSERT(getPixel(15, 10) == green);
>> >> ASSERT(getPixel(9, 15) == white);
>> >> ASSERT(getPixel(10, 15) == green);
>> >>
>> >> }
>> >
>> >
>> > I think this shows what I said about correctness and readability:
>> > -Asserting the correctness of the test and the result becomes close to
>> > impossible for the reader. One has to review the full code to have a
>> > chance
>> > of understanding an error.
>> > -You cannot cover non trivial cases (images, text, form elements, etc).
>> > -And it is inefficient. You have to render each frame on the UIProcess,
>> > move
>> > it to the WebProcess, and box it for JavaScript to process (with pixel
>> > format conversions depending on your graphics system)
>> >
>> > Of the ideas raised, I think this is one of my least favorite for
>> > testing
>> > fixed positioning.
>> >
>>
>> Isn't his suggestion the equivalent of what we do today in text-only
>> tests? i.e., printing "pass" or "fail" and making you have to look at
>> the test itself to see what's being tested?
>>
>> If the correctness of the rendering depends on those 4 specific pixels
>> having those four specific values, how exactly are you going to verify
>> that by looking at it?
>>
>> Again, I think I'm just not understanding you here?
>
>
> When looking at a test test, you follow the flow to know what it is supposed
> to do and where it breaks.
>
> How are you supposed to know, _by reading the code_, that the color at
> position 15, 9 should be white?
>
is "the code" the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.
And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.
Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.
As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 11:07 PM, Benjamin Poulain wrote:
> On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang
> wrote:
>>
>> I like this idea. I cannot find any harm if we have this functionality.
>
>
> Those changes are not harmless. There are people monitoring tests results
> full time in order to keep WebKit in good shape. No other part of WebKit
> require continuous attention.
>
I'm sorry, but either I don't understand Dongsung's suggestion, or I
don't understand your criticism. What does this sentence paragraph
mean? Are you suggesting someone needs to look at this type of test
full time in a way that we don't have to look at the other tests?
>>
>> Case 1: CSS Filters & Shaders
>> I wanted this test functionality when I commented
>> http://webkit.org/b/97859#c19
>> If I want to make gaussian blur test, I prefer using 'getPixel' test as
>> follows,
>
>
> Why wasn't a ref-test a better solution in this particular case?
>
I'm curious, what would you imagine the ref test contains?
>> Case 2: Fixed Position Element
>> [...]
>> function repeatedlyCalledDuringScrolling() {
>> ASSERT(getPixel(15, 9) == white);
>> ASSERT(getPixel(15, 10) == green);
>> ASSERT(getPixel(9, 15) == white);
>> ASSERT(getPixel(10, 15) == green);
>>
>> }
>
>
> I think this shows what I said about correctness and readability:
> -Asserting the correctness of the test and the result becomes close to
> impossible for the reader. One has to review the full code to have a chance
> of understanding an error.
> -You cannot cover non trivial cases (images, text, form elements, etc).
> -And it is inefficient. You have to render each frame on the UIProcess, move
> it to the WebProcess, and box it for JavaScript to process (with pixel
> format conversions depending on your graphics system)
>
> Of the ideas raised, I think this is one of my least favorite for testing
> fixed positioning.
>
Isn't his suggestion the equivalent of what we do today in text-only
tests? i.e., printing "pass" or "fail" and making you have to look at
the test itself to see what's being tested?
If the correctness of the rendering depends on those 4 specific pixels
having those four specific values, how exactly are you going to verify
that by looking at it?
Again, I think I'm just not understanding you here?
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 7:44 PM, Benjamin Poulain wrote: > > I am sorry, I should have given more context. > > There is visibly a growing discontent in the community about the cost > imposed from small ports. Just two weeks ago, there were 2 threads > discussing the cost of "peripheral ports". > I am convinced a part of this is technical. The project has not changed its > policies while the number of ports was growing. While duplicated code and > interfaces was okay when there were only 3 ports, it has become a pain when > we have 7+ ports to updates ... > In his email "WebKit Wishes", Eric said "It can’t be the job of the core > maintainers to care about all the peripheral ports which contribute very > little core code." I think it was an entirely reasonable question to ask if the wx port was being maintained, but I'm surprised by how this thread has evolved. There is a lot of discussion going on about the cost of so many ports, but not much about the benefits. Speaking personally, even before I joined Google, I was drawn to WebKit partially because it was used on such a wide variety of projects and in so many different ways. I was fortunate to be able to get a job that allows me to contribute to it, and I have found the work that I've done to help maintain the "peripheral" ports, while not pain free, quite rewarding (although I would be quite happy if it was less costly, of course). My point is that I think that lots of ports are part of what makes WebKit the goodness it is. Maybe I'm alone here, or at best part of a minority, but I wanted us to not lose sight of this idea. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 4:30 PM, Kevin Ollivier wrote: > > On Feb 11, 2013, at 3:33 PM, Dirk Pranke wrote: > >> On Mon, Feb 11, 2013 at 3:26 PM, Kevin Ollivier >> wrote: >>> >>> On Feb 11, 2013, at 2:01 PM, Martin Robinson wrote: >>> >>>> On Mon, Feb 11, 2013 at 1:41 PM, Kevin Ollivier >>>> wrote: >>>> >>>>> Actually, it will, if anything, increase the workload. Because I use waf, >>>>> I am able to use Python to auto-generate the list of sources to build. In >>>>> other words, I tell it to build all sources in a defined list of base >>>>> source directories, along with all sources in baseDirectory/portName >>>>> subdirs (where in this case I set portName to wx) and >>>>> FileName.cpp (e.g. FileNameWX.cpp if the port is wx), along >>>>> with some additional similar rules for USE defines, like CF. Then if >>>>> there are exceptions to these rules, I just filter them out of the >>>>> results or explicitly add files when I need to, say if I need to compile >>>>> a single Mac port source file. Since the WebKit tree is well-structured, >>>>> this approach works quite well, with me having to define exceptions >>>>> fairly rarely. The main issue I run into seems to be derived sources >>>>> files that are no longer built / used but are still being generated. The >>>>> performance hit of this is about 1 added second to my build, though on >>>>> slower mac hin >>> es it might be a couple seconds. For me, it's negligible given the benefit >>> I get from it. >>>> >>>> If I understand correctly, gyp is also capable of this kind of >>>> wildcard inclusion and exclusion. This will be the tool that allows >>>> the gyp build to be shared among many ports with the same source >>>> lists. >>> >>> Yes, but that is not what I see when I check, say, >>> Source/WebCore/WebCore.gypi. Which, BTW, has had around 500 revisions in it >>> over the past 6 months in comparison to the ~10 lines of changes to source >>> files in my patch-in-progress. So while gyp itself might have that feature >>> in it, for whatever reason, the WebKit projects do not utilize that feature >>> right now, so in practice, a switch to gyp means a switch away from >>> rule-based compilation. >>> >> >> There's two different things going on here: ease of maintenance, and >> reproducibility. Using wildcards in the (meta-)build file itself can >> make things easier but make it harder to get reproducible builds since >> you might accidentally include files in the tree you didn't want. >> >> When you're on a minority port, having wildcards certainly makes your >> life easier because you don't have to track every change we make >> explicitly, and it's probably easier for you to ensure you're doing >> things correctly in your own checkouts, so you don't have to worry >> about reproducibility as much. > > If the filesystem always follows clearly defined rules, then I'd argue that > reproducibility would not be an issue, though. WebKit currently doesn't > always, but the vast majority of times it does. The cases where it doesn't > are few and far between, and I don't think fixes would be all that difficult. > You can always create specific locations, like perhaps a Shared directory, > for any special cases that should explicitly be added by more than one, but a > subset of, ports, and can't be properly encapsulated into a feature or use > define. (For example, the WhateverNone.cpp are one of the special exceptions > I have to account for, and they could go in there.) > > I think the concern is really that this approach seems 'implicit' rather than > 'explicit'. With clear filesystem rules, though, it's not actually implicit. > A file going into one of the directories listed as a primary build dir means > it is to be built by all ports. So long as the project defines said rules, > the act of putting it there is explicit, and expresses intent. This also, > BTW, addresses the problem of old stuff being left in the tree. To remove it > from the build, you have to remove it, so you can just leave it sitting in > the tree for someone to stumble across later. (I'm pretty sure a number of > the exceptions I define in waf are actually examples of this.) I'm not sure we're talking about the same sort of concerns. The case I'm talking about when
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Sun, Feb 10, 2013 at 1:30 AM, Ryosuke Niwa wrote: > On Sat, Feb 9, 2013 at 5:35 PM, Dirk Pranke wrote: >> >> Perhaps -- and this is a tangent to this thread -- we could also >> investigate ways of writing tests that did render pngs but told NRWT >> to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or >> conveyed which parts of the image were important ... > > > What's the point of that? > Oops. I think I deleted the part of my reply explaining that :). What I meant to say is that the downside of text-only/programmatic tests is that you don't actually get a rendered page to look at. The downside of pixel tests is that you can get a rendered page that shows you what's wrong, but the rendered page often contains a bunch of "extraneous" things like explanatory text that can cause unnecessary diffs. It might be nice if we had better ways to filter out the parts that caused the unnecessary diffs. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 3:26 PM, Kevin Ollivier wrote: > > On Feb 11, 2013, at 2:01 PM, Martin Robinson wrote: > >> On Mon, Feb 11, 2013 at 1:41 PM, Kevin Ollivier >> wrote: >> >>> Actually, it will, if anything, increase the workload. Because I use waf, I >>> am able to use Python to auto-generate the list of sources to build. In >>> other words, I tell it to build all sources in a defined list of base >>> source directories, along with all sources in baseDirectory/portName >>> subdirs (where in this case I set portName to wx) and >>> FileName.cpp (e.g. FileNameWX.cpp if the port is wx), along with >>> some additional similar rules for USE defines, like CF. Then if there are >>> exceptions to these rules, I just filter them out of the results or >>> explicitly add files when I need to, say if I need to compile a single Mac >>> port source file. Since the WebKit tree is well-structured, this approach >>> works quite well, with me having to define exceptions fairly rarely. The >>> main issue I run into seems to be derived sources files that are no longer >>> built / used but are still being generated. The performance hit of this is >>> about 1 added second to my build, though on slower machi n > es it might be a couple seconds. For me, it's negligible given the benefit I > get from it. >> >> If I understand correctly, gyp is also capable of this kind of >> wildcard inclusion and exclusion. This will be the tool that allows >> the gyp build to be shared among many ports with the same source >> lists. > > Yes, but that is not what I see when I check, say, > Source/WebCore/WebCore.gypi. Which, BTW, has had around 500 revisions in it > over the past 6 months in comparison to the ~10 lines of changes to source > files in my patch-in-progress. So while gyp itself might have that feature in > it, for whatever reason, the WebKit projects do not utilize that feature > right now, so in practice, a switch to gyp means a switch away from > rule-based compilation. > There's two different things going on here: ease of maintenance, and reproducibility. Using wildcards in the (meta-)build file itself can make things easier but make it harder to get reproducible builds since you might accidentally include files in the tree you didn't want. When you're on a minority port, having wildcards certainly makes your life easier because you don't have to track every change we make explicitly, and it's probably easier for you to ensure you're doing things correctly in your own checkouts, so you don't have to worry about reproducibility as much. On the other hand, if you were to convert to using the same gypi files we were using, then you wouldn't have to track every change we make, either; you'd get them for free. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 1:41 PM, Kevin Ollivier wrote: > Hi Martin, > > On Feb 11, 2013, at 12:47 PM, Martin Robinson wrote: > >> On Mon, Feb 11, 2013 at 11:27 AM, Kevin Ollivier >> wrote: >> >>> The project is not dead per-se, but it's true that it's not very active >>> these days. Recently, I haven't had a lot of time to devote to the port. The >>> other big factor has been that I've been the only active contributor to the >>> port pretty much since its inception, which has made patch submission pretty >>> difficult due to the challenge of finding a reviewer with knowledge of the >>> wx code. As a result, the wx activity on trunk will look pretty sparse since >>> I'm making most of my changes on the wx port's git repo on gitorious.org >>> rather than submitting patches for trunk. When it comes to trunk, I >>> basically just periodically submit build fixes for any build breakages in >>> order to merge and update my git mirror. >> >> Regardless of whether the Wx port stays in the tree, I think you >> should consider changing your build system. I believe the only port >> that uses waf is Wx, so using cmake or gyp will make your work a lot >> easier. > > Actually, it will, if anything, increase the workload. Because I use waf, I > am able to use Python to auto-generate the list of sources to build. In other > words, I tell it to build all sources in a defined list of base source > directories, along with all sources in baseDirectory/portName subdirs (where > in this case I set portName to wx) and FileName.cpp (e.g. > FileNameWX.cpp if the port is wx), along with some additional similar rules > for USE defines, like CF. Then if there are exceptions to these rules, I just > filter them out of the results or explicitly add files when I need to, say if > I need to compile a single Mac port source file. Since the WebKit tree is > well-structured, this approach works quite well, with me having to define > exceptions fairly rarely. The main issue I run into seems to be derived > sources files that are no longer built / used but are still being generated. > The performance hit of this is about 1 added second to my build, though on > slower machine s > it might be a couple seconds. For me, it's negligible given the benefit I > get from it. > > As an example, I'm still at the linking phase right now on Mac (dealing with > the removal of the libWebKitLibrarySnowLeopard.a and a couple other things), > but my patch so far to catch up with 6 months of changes in trunk is right > now at 19KB, and about 30 lines or so of that is actually the build system > changes. The number of exceptions I needed to add to the build over that time > is 4, and I added about as many build directories to the list as well. I > suspect if we were to accumulate the changes to cmake and gyp files over > those 6 months, it will probably be well above 30-40 lines. Sure, I wouldn't > have written all those lines myself, but sharing work with other developers > is still more work for any individual developer than having the computer do > the work for you. :) > > Everyone tells me how great cmake and gyp is, but I'm not sure any of them > have taken any time out to actually investigate whether or not waf is as good > as or better than their preferred tool. (It's not as good as auto-generating > project files, but it's also in Python so integrating it with gyp would > probably be a simple task.) I've often wanted to take some time out to get > other ports building with it, as it probably would take a day or two at most, > but I lack both the time and expertise in other ports to spend much time on > that. Much of the code in there already is port-agnostic, with most > wx-specific bits in a port_name == "wx" check, but of course there's the > issue of the various port-specific build configurations and such for each > port. > I have to admit, I thought waf was more like scons and less like cmake and gyp, i.e., I thought it was a portable build system rather than a meta-build system (and had never looked at it in detail accordingly). I am taking another look at it now :) -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain wrote: > On Sat, Feb 9, 2013 at 1:24 AM, wrote: >> >> I did not say they cannot be tested with the two methods suggested :) It's >> more about a preference to move some of those decisions from the test >> infrastructure to the test itself, but potentially those bugs could be >> tested in either way. >> Examples for bugs I've encountered/reviewed and can use better in-motion >> testing (note that those are specific to Qt/EFL, but I'm sure there are tons >> of bugs like this that come up for Apple/Google as well) >> http://trac.webkit.org/changeset/140825 >> http://trac.webkit.org/changeset/142112 >> http://trac.webkit.org/changeset/134953 >> https://bugs.webkit.org/show_bug.cgi?id=109179 >> >> Controlling the clock and programmatically sampling the end result would >> definitely make those more testable, but of course any progress in this area >> would be beneficial and my preference to a canvas-based API is more of an >> opinion. > > > To explain my concerns: > Sometime, I look at a failing test, and think: "what the f**k is this > supposed to test?". Then I have to dig a ton of logs, and finally read the > change to understand a the single JS statement in the whole script that make > the test useful. > > This is the situation I am afraid with a solution where pixels would be > evaluated from JavaScript. You can test, but you lack visibility why > something is correct or incorrect. Text tests, ref-tests and pixel tests > have a great readability, you can quickly evaluate their correctness. This > is important in my opinion, I don't think we want more opaque output like > the RenderTree dump. > > I am not against your plan if the readability of the tests can be good. I'd > be happy to review patches toward testing dynamic changes in webpages. > A counter-argument, of course, that there are a lot of pixel tests that would be a lot better as text-only tests that were verifying certain aspects of how the page rendered programmatically. Perhaps -- and this is a tangent to this thread -- we could also investigate ways of writing tests that did render pngs but told NRWT to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or conveyed which parts of the image were important ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa wrote: > On Fri, Feb 8, 2013 at 5:35 AM, wrote: >> >> we don't currently have a solution in webkit's test infrastructure for >> testing animations "on the fly", or in general for testing multiple image >> results in the same test. >> (If this was discussed before and I'm unaware of that discussion, please >> stop me here...) >> >> This is because ImageDiff works on the single-test level, and you can't >> explicitly make image comparisons in the same test at different points in >> time. >> This has before caused us several regressions that were not caught by >> layout tests. >> >> I'd like to propose a solution, and would welcome some feedback on whether >> it's a good one... >> The idea is that you would be able to programatically retrieve the current >> snapshot into a canvas ImageData, and then compare the pixel results with >> JavaScript in the LayoutTest. Something like: > > > I had similar thoughts but my counter "proposal" is to let DRT/WTR generate > multiple actual results either in the form of multiple layers in PNG or > multiple PNG images. The advantage of this latter approach is that we can > have multiple reference files as well. e.g. if a green box is to be moved > from (0,0) to (50,0) and to (100, 0), we could create three reference files > that correspond to each state. > > But perhaps there is a good reason you didn't choose this approach. Could > you elaborate on the reason you picked this particular API? > I also appreciate the problem -- and would be interesting in solving it -- but I'm a bit concerned that either proposal is at least somewhat nondeterministic (or flaky), and I'm not sure how you'd be able to control which frames of the animation you were getting well enough. But I'm not much of expert here so I'm not sure how often it matters. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Gated trunk, experiences from OpenStack
On Tue, Feb 5, 2013 at 3:34 PM, Tim Ansell wrote: > On 6 February 2013 07:17, Dirk Pranke wrote: >> >> On Tue, Feb 5, 2013 at 9:46 AM, Martin Robinson >> wrote: >> > On Tue, Feb 5, 2013 at 9:28 AM, Adam Barth wrote: >> >> Do you know how they got rid of flakiness in their tests? We've spent >> >> a bunch of effort fixing flaky tests (and in marking the remaining >> >> flaky tests as flaky), but there's still a long tail of flakiness. I >> >> wonder if that sort of thing might be different for OpenStack if they >> >> have a different approach to testing than we do. > > > From what I can see they have a pretty similar goal to us. I personally > don't know where our test flakyness comes from, so can't really comment on > how we could fix it. > >> >> > >> > Another useful thing is to know the number of tests in OpenStack. >> > WebKit has more tests than any other project I've worked on. >> > >> >> There are two other related aspects that make our tests flaky: >> >> 1) They're very high level integration tests (mostly), which, as they >> cover large swaths of code in each test, are much more susceptible to >> flakiness than method-level unit tests. > > > While OpenStack doesn't have anywhere near the number of integration tests > WebKit does, it does have large integration tests. Infact, one of their > tests brings up a whole cloud stack and checks that you can operate the > cluster. > >> >> 2) They weren't generally written to be run in parallel, and thus we >> often have to be concerned with system-level resource contention. > > > Neither where OpenStack's originally. They made heavy use of tool called > testr ( http://pypi.python.org/pypi/testrepository ) which has a mode to > automatically find when two tests are interfering with each other. testr > also has a bunch of other useful features, like only re-running tests which > are currently failing and keeping a database of test runs and allowing stat > collection. > Ah, the testr isolation bisection does look interesting. I have done a little work along those lines but haven't gotten very far. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
I have looked at YAML off and on over the years, and I'm not sure that it would be much of an improvement in this case. I do believe that dropping the strict python syntax could make some things easier to read. I don't have a fully-baked proposal in mind, and I don't know what the perf hit would be. I will also note that -- like in many languages -- it's easy to write hard-to-read files. GYP has never had much of a style guide, and it probably could greatly benefit from one. It is important to make a fair apples-to-apples comparison for whatever we end up looking at. I suspect that given any tool or language, you can find some things that are easy to express and some that are harder. Even the existing chromium gyp files are not necessarily a model of clean code or good style. Also, one thing that's important to realize about GYP's syntax is that it is very set-based; the order things appear in the files is much less important than in many other languages (especially in how they handle conditionals, i.e., if/then/else). This has many advantages, but does also have some disadvantages. -- Dirk On Tue, Feb 5, 2013 at 12:16 PM, Eric Seidel wrote: > I'm curious if YAML was ever considered? I have very limited > experience with YAML, except for Google App Engine config files. > > It's very python parse-able? :) > > On Tue, Feb 5, 2013 at 11:55 AM, Mark Mentovai wrote: >> You’re not supposed to use arbitrary Python, it’s highly discouraged. We >> have a linter that keeps you from doing things you’re not supposed to do >> (like this), but it slows things down, so it’s not part of the “standard” >> GYP run that developers normally use. It can run as a pre-commit script or >> test on the bot or something else. Used as we’re using it, GYP basically is >> simple JSON, except the rules about commas are looser and # introduces >> comments. >> >> >> On Tue, Feb 5, 2013 at 2:47 PM, Ryosuke Niwa wrote: >>> >>> On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai wrote: The parser (and the grammar) works the way it does because it’s just Python >>> >>> >>> This works great for people who like Python syntax but not for someone >>> like myself who dislikes Python syntax. >>> >>> I also find it particularly annoying that people can use whatever Python >>> constructs they want to use in GYP. It dramatically reduces language >>> portability because you need to support quite a few Python constructs and >>> quirks in order to correctly parse GYP. >>> >>> I personally would have much preferred for it be a simple JSON file. >>> >>> - R. Niwa >>> >> >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> https://lists.webkit.org/mailman/listinfo/webkit-dev >> > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Gated trunk, experiences from OpenStack
On Tue, Feb 5, 2013 at 9:46 AM, Martin Robinson wrote: > On Tue, Feb 5, 2013 at 9:28 AM, Adam Barth wrote: >> Do you know how they got rid of flakiness in their tests? We've spent >> a bunch of effort fixing flaky tests (and in marking the remaining >> flaky tests as flaky), but there's still a long tail of flakiness. I >> wonder if that sort of thing might be different for OpenStack if they >> have a different approach to testing than we do. > > Another useful thing is to know the number of tests in OpenStack. > WebKit has more tests than any other project I've worked on. > There are two other related aspects that make our tests flaky: 1) They're very high level integration tests (mostly), which, as they cover large swaths of code in each test, are much more susceptible to flakiness than method-level unit tests. 2) They weren't generally written to be run in parallel, and thus we often have to be concerned with system-level resource contention. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Using ninja (was Re:Common build system (was Re: WebKit Wishes))
The short answer is, you can't. The fastest path is probably to get a working set of gyp files for the apple mac wk2. I'm going to start working on this just to see how far of we are (Adam's work from a year or two ago had JSC and WebCore building, but wasn't too functional beyond that; even so, it's probably a good starting point). -- Dirk On Sun, Feb 3, 2013 at 1:22 PM, Dean Jackson wrote: > OK, this sounds fantastic. And I've noticed how much faster Chromium > incrementally builds using ninja when I've done that. > > So, ignoring the discussion of a single build system for a moment, how > can I, as a developer using the OS X + WK2 port, living mostly > in Xcode for editing and debugging, use ninja? I need the idiot's > guide :) > > (Note: I am an idiot, but not so much an idiot to realise that the > answer involves lots of work and probably updating some old GYP > files that you and Adam were testing with, etc etc. I'm just selfishly > thinking that cutting even 30s off each incremental rebuild would make > me so much happier that I'd be willing to put up with other > inconveniences.) > > Dean > > On 03/02/2013, at 4:54 PM, Eric Seidel wrote: > >> +1 >> >> Ninja is beyond-words amazing. http://martine.github.com/ninja/ For >> better or worse, it is not designed to use human-editable build files, >> but rather to be used by a meta build system, like GYP or CMake. So >> using ninja is really an orthogonal discussion to the "single build >> system" discussion for WebKit. :) >> >> Were the WebKit project to convert to using a single meta-build >> system, ninja would become an option many users might choose. I'm >> told most Chromium hackers have GYP set to output ninja files these >> days, with the exception of some folks who still want the MSVC build >> environment. For WebKit ports already using CMake, they should >> definitely try ninja today! >> >> >> Anyway, my wish was not about arguing for a specific build solution. >> I'm instead noting that for the project to continue to move quickly, >> we need to stop needing to edit 8 build systems for every file >> move/addition. Whether that's GYP or CMake or something else, I don't >> really care. Adam and I tried GYP-for-WebKIt a while back. But any >> of these solutions will require buy-in from Apple, as they will have >> to do the largest amount of work converting to use something other >> than XCode. >> >> >> On Sat, Feb 2, 2013 at 8:20 PM, Nico Weber wrote: >>> On Sat, Feb 2, 2013 at 4:58 PM, Adam Barth wrote: Ninja has extremely fast incremental builds and can be generated by GYP. Here are some stats from a year ago: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC Ninja has gotten even faster since then. If you're interested in trying it out, you can play around with incremental builds of the Chromium port on Mac or Linux. >>> >>> You can also look at the build output from the chromium bots. >>> >>> Empty build in 1s: >>> http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66807/steps/compile-webkit/logs/stdio >>> Build with a few files changed in 15s: >>> http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66800/steps/compile-webkit/logs/stdio >>> >>> …and this is on fairly slow bots. On my SSD-equipped laptop, I can do >>> incremental rebuilds of all of chrome after touching one (cpp or mm) >>> file in 2-6s. >>> > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Sun, Feb 3, 2013 at 9:20 PM, Maciej Stachowiak wrote: > > On Feb 3, 2013, at 8:18 PM, Mike Lawther wrote: > > Hi Maciej! > > On 31 January 2013 11:15, Maciej Stachowiak wrote: >> >> It's far simplest for us if: >> (a) There is an Xcode project (or a Makefile) that builds the Mac port >> checked in to source control. >> (b) The generated project invokes only tools that are part of the default >> Mac OS X install. > > > For b), does 'default' include an Xcode install? From my memory of setting > up a Mac dev box Xcode was needed to compile. > > > The limitation in (a) is all about how the build fleet is managed, and yes, > the machines in question have the Mac OS X user install as well as developer > tools installed, and it is ok to rely on that. > > To clarify: the things that are not kosher are submitting a binary tool as > part of the module, downloading things as part of the build, or relying on > something to be installed out of band before building. It is ok for the > build to rely on scripts, so long as the relevant interpreter is part of the > default install. Any such script would have to be in each directory under > Source/ since these build separately without access to the source of the > others. > > I should mention that there's a lot of interest right now at Apple in the > possibility of switching to Gyp. We think it's likely closer to what we need > than CMake, and we suspect there is probably somewhat better probability of > modifying it if necessary. If we find someone at Apple who is able to work > on such a project in the short term (not completely impossible), would any > Google folks be interested in collaborating? Ideally we could use assistance > from someone who (a) understands how the Chromium WebKit Gyp-based build > works; and (b) is willing and able to change Gyp if necessary. > Mark is the gyp-guru, but I would be happy to help out as well. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Exporting symbols (was Re: Common build system (was Re: WebKit Wishes))
On Thu, Jan 31, 2013 at 3:12 PM, Alexey Proskuryakov wrote: > > 31.01.2013, в 14:57, Dirk Pranke написал(а): > >>>>>> One thing to keep an eye on with WEBCORE_EXPORT is that it can increase >>>>>> the number of exports for each port, because it would export symbols that >>>>>> are only needed for other ports. >>>>> >>>> >>>> I'm not sure I understand the concern here. Is the idea that some >>>> ports might build WebCore as a DLL (and need exports) and others >>>> wouldn't? Or the idea that the API that is needed to be exported might >>>> vary from port to port in such a way that too many symbols for WebCore >>>> might be exported on some ports? >>> >>> The latter. >>> >>> I suspect that it's somewhat less of an issue for JavaScriptCore, because >>> its exports are mostly used from cross-platform code. >>> >> >> I see. Some concrete examples would be helpful here (to me, at least). >> What components and ports are we talking about, and what sorts of >> symbols? > > I don't have concrete examples, and I don't know how big an effect on > performance increasing the number of exports would have. It's something to > keep an eye on, not a reason to avoid trying. > I'm not parsing your reply properly -- avoid trying what? A common EXPORT macro that is set as appropriate by each port? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Exporting symbols (was Re: Common build system (was Re: WebKit Wishes))
On Thu, Jan 31, 2013 at 2:52 PM, Alexey Proskuryakov wrote: > > 31.01.2013, в 14:45, Dirk Pranke написал(а): > >>>> One thing to keep an eye on with WEBCORE_EXPORT is that it can increase >>>> the number of exports for each port, because it would export symbols that >>>> are only needed for other ports. >>> >> >> I'm not sure I understand the concern here. Is the idea that some >> ports might build WebCore as a DLL (and need exports) and others >> wouldn't? Or the idea that the API that is needed to be exported might >> vary from port to port in such a way that too many symbols for WebCore >> might be exported on some ports? > > The latter. > > I suspect that it's somewhat less of an issue for JavaScriptCore, because its > exports are mostly used from cross-platform code. > I see. Some concrete examples would be helpful here (to me, at least). What components and ports are we talking about, and what sorts of symbols? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Exporting symbols (was Re: Common build system (was Re: WebKit Wishes))
On Thu, Jan 31, 2013 at 9:48 AM, Ryosuke Niwa wrote: > On Thu, Jan 31, 2013 at 9:43 AM, Alexey Proskuryakov wrote: >> >> >> 31.01.2013, в 0:17, Ryosuke Niwa написал(а): >> >>> I did it for WTF and JSC with kevino@. >>> https://bugs.webkit.org/show_bug.cgi?id=72855 >>> I hope I had had time to extend the work to WebCore, which was my >>> original goal :-/ >>> >>> I did it through an automation but the tool was so slow and hacky that it >>> was not capable for large WebCore codebase. >>> >>> Seeing that the number of exported symbol is about 2700, manual work with >>> regular expression will also work >>> if it is OK to do the work incrementally... and if you are patient enough >>> :-) >> >> >> 2700 is a lot of symbols to export! Perhaps we can reduce the number of >> exported symbols? For example, we can move more code from WebKit into >> WebCore to share more code between ports so that the API surface WebCore has >> to expose will be smaller and more well defined. >> >> >> One thing to keep an eye on with WEBCORE_EXPORT is that it can increase >> the number of exports for each port, because it would export symbols that >> are only needed for other ports. > I'm not sure I understand the concern here. Is the idea that some ports might build WebCore as a DLL (and need exports) and others wouldn't? Or the idea that the API that is needed to be exported might vary from port to port in such a way that too many symbols for WebCore might be exported on some ports? > > Can we make the macro starter and specify ports under which a given symbol > needs to be exported? e.g. WEBCORE_EXPORT(MAC, QT) > I'm not sure how this helps, but I think that's because of my previous comment. I could see something like this maybe being needed in my later scenario (where MAC might need a symbol exported from WebCore that Qt needed to keep from being exported)? Is that a hard requirement, or an optimization? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 9:14 AM, Patrick Gansterer wrote: > > Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: > > > On Jan 31, 2013, at 1:56 AM, Patrick Gansterer wrote: > > Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: > > On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger > wrote: >> >> Another option is to add a webkit-patch command for modifying the build >> files. That way, the syntax doesn't need to be overly human friendly. There >> was also some attempt to write a tool to add files automatically: >> https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a >> tool becomes easier if it would only modify one source of truth and >> generates all other artifacts such as Xcode projects from it. > > > I don't want build file's syntax to be so human unfriendly that I need a > tool for it. > > Often times, these syntax problems can be improved dramatically by simple > changes. e.g. we had a similar discussion about TestExpectation syntax, and > I'm much happier with the new syntax even though the new syntax is > functionally equivalent to the old one, and two syntaxes are very similar. > > On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe wrote: >> >> I’ve experimented with this in the past and you’re right that it shouldn’t >> be particularly difficult to do. However, I suspect that the task would be >> similar in scope to defining an improved syntax for gyp. And if the syntax >> is the primary sticking point with gyp then it’d seem preferable to tackle >> initially. > > > Yeah. In fact, we can just come up with whatever syntax we like and convert > it to the existing gyp format if the syntax was the biggest issue. > > > Do we want to define the whole build system (including information how to > invoke the generators) with the new system, or is a "simple" list for the > input files sufficient? IMHO adding a new generator build step happens very > rarely. So maybe we can spit the "input file list" (mainly *.cpp and *.idl) > into new files. > Then GYP and CMake can read them and generate the build system out of them > directly (like they to already today) instead of listing the files in the > *.gpyi and *.cmake. This might work for other systems like qmake too. > For XCode we can maybe have a "template XCode project" and generate the > "work XCode project" with a script. This script then only need to fill in > the files from the "input file list" into the "template XCode project". > Defining the feature flags can be done like Maciej suggested with "Port.h" > files. > > > I think it would be better to adapt an existing meta build system to our > needs than to make one from scratch, unless we find that completely > impractical. > > In particular, gyp and cmake both know how to handle generated sources, and > while it may not be super common to make a new type of generated source, > it's bad for hackability of the project of doing so is super hard. We get a > lot of hackability benefits from using various kinds of generated sources, > many first introduced in the days when we had a lot fewer build systems. So > in my mind, they are already ahead of a hypothetical "simple" system. > > > Do you want to kick the requirements of the smaller ports from trunk or do > you think that e.g. a qmake generate for GYP makes sense? > AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, > where CMake/GYP is not an option. > It could certainly make sense for us to add Autotools and Qmake generators to GYP; I'm less certain if a CMake generator makes much sense, but I haven't thought about it as much. I'm not super familiar with any of these three tools, so I could be dead wrong. > I completely agree that creating a new meta meta build system isn't a good > idea, but sharing the common parts (which reduce the daily productivity) > might be a step in the right direction. Using simple text files which > contain the list of files (like the gpyi files already do today) isn't a new > build system. It only offers the existing meta build systems (CMake, GYP, > autotools, qmake) to use a common base. > > The remaining build systems can be ported to one of these systems or be > adopted to use this file lists too. > > -- Patrick I suspect that we would quickly find that we would want some sort of support for conditionals and/or file inclusion in our "simple text files", at which point you basically get a meta-meta-build system :). I don't actually think such a thing is that bad of an idea, but it's all in the details. I would like to find a solution where all of the ports were able to retain integration with their "native" build environments one way or another. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 12:10 PM, Patrick Gansterer wrote: > > Am 31.01.2013 um 21:07 schrieb Dirk Pranke: > >> On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima >> wrote: >>> On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: >>>> On Jan 30, 2013, at 3:24 PM, Dirk Pranke wrote: >>>>> On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo wrote: >>>>>> Thanks for sharing this. >>>>>> >>>>>> On Jan 30, 2013, at 1:28 PM, Eric Seidel wrote: >>>>>> >>>>>> I wish we only had one build system (it were easy to add/remove/move >>>>>> files). >>>>>> >>>>>> I believe changes like http://trac.webkit.org/changeset/74849 are an >>>>>> unhealthy sign for the project. Adam is not the only person who has >>>>>> chosen >>>>>> to empty files instead of removing them. The pain of updating 8 build >>>>>> system is so great, we jump through hoops to avoid it. Which means it >>>>>> took >>>>>> us months to move JavaScriptCore/wtf to WTF, and will take us years to >>>>>> kill >>>>>> WebCore/platform. >>>>>> >>>>>> >>>>>> +1 >>>>>> >>>>>> This is a hard problem. It is a problem worth solving. >>>>>> >>>>>> Do you have more thoughts on this, particularly since you know quite well >>>>>> how both Xcode and gyp work? >>>>>> >>>>>> I suspect this is one of those things that it would be hard to achieve >>>>>> consensus on since there are so many stakeholders. But it may be >>>>>> fruitful >>>>>> to have a "what if" discussion about what this might look like. >>>>> >>>>> I think we can solve this problem if we agree that we want to. I think >>>>> we haven't in the past mostly because we couldn't reach a consensus >>>>> that it was worth solving enough to really try. >>>>> >>>>> I would love to see this fixed and would be glad to work on it. I >>>>> think we should at least pursue this far enough to fully understand >>>>> what our options are and what the costs and tradeoffs might be; does >>>>> anyone disagree, and is anyone else willing to help pitch in? >>>>> >>>>> I think there are several possible ways we could solve this. One would >>>>> be to switch to a common meta-build system. My understanding is that >>>>> Apple's internal production build processes impose certain constraints >>>>> here that I don't fully understand, but I know we've discussed the >>>>> possibility of checking in generated project files as a workaround. >>>>> Maybe there are other options as well to those constraints? I would >>>>> love to discuss this further w/ someone from Apple ... >>>> >>>> It's far simplest for us if: >>>> (a) There is an Xcode project (or a Makefile) that builds the Mac port >>>> checked in to source control. (b) The generated project invokes only tools >>>> that are part of the default Mac OS X install. >>>> >>>> It may not be completely impossible to violate these requirements but it >>>> will require a lot of bureaucracy. >>>>> (Also, just to get this out of the way, I don't think gyp needs to be >>>>> the solution). >>>>> >>>>> Another alternative would be to write a script that did support at >>>>> least the common use cases (add/move/delete files). There have been >>>>> attempts in the past, but they have foundered on at least some >>>>> perceived skepticism over how well this would work w/ XCode. That >>>>> said, I don't think we've really pushed this to see. At some point >>>>> this script might turn into a meta-meta-build system, which might be >>>>> silly but also be the shortest path to the finish line. >>>>> >>>>> I suggest if there is interest in this we can start a new thread to >>>>> discuss further ... >>>> >>>> My preference would be to use a common meta-build system with a comfortably >>>> human-readable and human-editable syntax, and checked in generated project >>>> files for those ports t
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima wrote: > On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: >> On Jan 30, 2013, at 3:24 PM, Dirk Pranke wrote: >> > On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo wrote: >> >> Thanks for sharing this. >> >> >> >> On Jan 30, 2013, at 1:28 PM, Eric Seidel wrote: >> >> >> >> I wish we only had one build system (it were easy to add/remove/move >> >> files). >> >> >> >> I believe changes like http://trac.webkit.org/changeset/74849 are an >> >> unhealthy sign for the project. Adam is not the only person who has >> >> chosen >> >> to empty files instead of removing them. The pain of updating 8 build >> >> system is so great, we jump through hoops to avoid it. Which means it >> >> took >> >> us months to move JavaScriptCore/wtf to WTF, and will take us years to >> >> kill >> >> WebCore/platform. >> >> >> >> >> >> +1 >> >> >> >> This is a hard problem. It is a problem worth solving. >> >> >> >> Do you have more thoughts on this, particularly since you know quite well >> >> how both Xcode and gyp work? >> >> >> >> I suspect this is one of those things that it would be hard to achieve >> >> consensus on since there are so many stakeholders. But it may be >> >> fruitful >> >> to have a "what if" discussion about what this might look like. >> > >> > I think we can solve this problem if we agree that we want to. I think >> > we haven't in the past mostly because we couldn't reach a consensus >> > that it was worth solving enough to really try. >> > >> > I would love to see this fixed and would be glad to work on it. I >> > think we should at least pursue this far enough to fully understand >> > what our options are and what the costs and tradeoffs might be; does >> > anyone disagree, and is anyone else willing to help pitch in? >> > >> > I think there are several possible ways we could solve this. One would >> > be to switch to a common meta-build system. My understanding is that >> > Apple's internal production build processes impose certain constraints >> > here that I don't fully understand, but I know we've discussed the >> > possibility of checking in generated project files as a workaround. >> > Maybe there are other options as well to those constraints? I would >> > love to discuss this further w/ someone from Apple ... >> >> It's far simplest for us if: >> (a) There is an Xcode project (or a Makefile) that builds the Mac port >> checked in to source control. (b) The generated project invokes only tools >> that are part of the default Mac OS X install. >> >> It may not be completely impossible to violate these requirements but it >> will require a lot of bureaucracy. >> > (Also, just to get this out of the way, I don't think gyp needs to be >> > the solution). >> > >> > Another alternative would be to write a script that did support at >> > least the common use cases (add/move/delete files). There have been >> > attempts in the past, but they have foundered on at least some >> > perceived skepticism over how well this would work w/ XCode. That >> > said, I don't think we've really pushed this to see. At some point >> > this script might turn into a meta-meta-build system, which might be >> > silly but also be the shortest path to the finish line. >> > >> > I suggest if there is interest in this we can start a new thread to >> > discuss further ... >> >> My preference would be to use a common meta-build system with a comfortably >> human-readable and human-editable syntax, and checked in generated project >> files for those ports that need them. >> >> I think a key to making this work is to get Chromium and the Apple Mac port >> onto a common build system, which will probably require both Google and >> Apple ponying up at least one person to work on this project for a >> reasonable period of time. >> >> I think the plausible meta-build-systems to use would be CMake and Gyp, but >> in both cases it may be necessary to modify them to work well for everyone. >> >> I'd also like to add that I think a key related issue to common build system >> is common feature configuration. The many different ways ports con
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Wed, Jan 30, 2013 at 4:15 PM, Maciej Stachowiak wrote: > > On Jan 30, 2013, at 3:24 PM, Dirk Pranke wrote: > >> On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo wrote: >>> Thanks for sharing this. >>> >>> On Jan 30, 2013, at 1:28 PM, Eric Seidel wrote: >>> >>> I wish we only had one build system (it were easy to add/remove/move files). >>> >>> I believe changes like http://trac.webkit.org/changeset/74849 are an >>> unhealthy sign for the project. Adam is not the only person who has chosen >>> to empty files instead of removing them. The pain of updating 8 build >>> system is so great, we jump through hoops to avoid it. Which means it took >>> us months to move JavaScriptCore/wtf to WTF, and will take us years to kill >>> WebCore/platform. >>> >>> >>> +1 >>> >>> This is a hard problem. It is a problem worth solving. >>> >>> Do you have more thoughts on this, particularly since you know quite well >>> how both Xcode and gyp work? >>> >>> I suspect this is one of those things that it would be hard to achieve >>> consensus on since there are so many stakeholders. But it may be fruitful >>> to have a "what if" discussion about what this might look like. >>> >> >> I think we can solve this problem if we agree that we want to. I think >> we haven't in the past mostly because we couldn't reach a consensus >> that it was worth solving enough to really try. >> >> I would love to see this fixed and would be glad to work on it. I >> think we should at least pursue this far enough to fully understand >> what our options are and what the costs and tradeoffs might be; does >> anyone disagree, and is anyone else willing to help pitch in? >> >> I think there are several possible ways we could solve this. One would >> be to switch to a common meta-build system. My understanding is that >> Apple's internal production build processes impose certain constraints >> here that I don't fully understand, but I know we've discussed the >> possibility of checking in generated project files as a workaround. >> Maybe there are other options as well to those constraints? I would >> love to discuss this further w/ someone from Apple ... > > It's far simplest for us if: > (a) There is an Xcode project (or a Makefile) that builds the Mac port > checked in to source control. > (b) The generated project invokes only tools that are part of the default Mac > OS X install. > > It may not be completely impossible to violate these requirements but it will > require a lot of bureaucracy. > >> (Also, just to get this out of the way, I don't think gyp needs to be >> the solution). >> >> Another alternative would be to write a script that did support at >> least the common use cases (add/move/delete files). There have been >> attempts in the past, but they have foundered on at least some >> perceived skepticism over how well this would work w/ XCode. That >> said, I don't think we've really pushed this to see. At some point >> this script might turn into a meta-meta-build system, which might be >> silly but also be the shortest path to the finish line. >> >> I suggest if there is interest in this we can start a new thread to >> discuss further ... > > My preference would be to use a common meta-build system with a comfortably > human-readable and human-editable syntax, and checked in generated project > files for those ports that need them. > > I think a key to making this work is to get Chromium and the Apple Mac port > onto a common build system, which will probably require both Google and Apple > ponying up at least one person to work on this project for a reasonable > period of time. > > I think the plausible meta-build-systems to use would be CMake and Gyp, but > in both cases it may be necessary to modify them to work well for everyone. > Premake might also be an option, though I wouldn't necessarily vote for it. Gyp's syntax is ... awkward ... but apart from that might just work for checking in generated apple mac xcode projects. We should try it (shouldn't be too hard, since abarth had it working at one point). I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. However, we've since switched to ninja for most things so I'm less sure how important this is n
Re: [webkit-dev] WebKit Wishes
On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo wrote: > Thanks for sharing this. > > On Jan 30, 2013, at 1:28 PM, Eric Seidel wrote: > > I wish we only had one build system (it were easy to add/remove/move files). > > I believe changes like http://trac.webkit.org/changeset/74849 are an > unhealthy sign for the project. Adam is not the only person who has chosen > to empty files instead of removing them. The pain of updating 8 build > system is so great, we jump through hoops to avoid it. Which means it took > us months to move JavaScriptCore/wtf to WTF, and will take us years to kill > WebCore/platform. > > > +1 > > This is a hard problem. It is a problem worth solving. > > Do you have more thoughts on this, particularly since you know quite well > how both Xcode and gyp work? > > I suspect this is one of those things that it would be hard to achieve > consensus on since there are so many stakeholders. But it may be fruitful > to have a "what if" discussion about what this might look like. > I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] build.webkit.org down
This is the output of the test bots, right? I'm guessing this is due to (a) us running pixel tests and (b) not checking in the expected failing baselines. If I ever get back to working on supporting -failing.png, this could go down *a lot*. Alternatively, we could just decide to check in the existing failures as -expected and call it good enough ... -- Dirk On Fri, Jan 25, 2013 at 9:35 AM, Eric Seidel wrote: > I'm surprised that chromium mac is 3x the size of Apple Mac... debug > even! Chromium build output should be much smaller... at least as of > late. > > On Fri, Jan 25, 2013 at 9:30 AM, William Siegrist wrote: >> Here are the largest results sets on the master in GB. We currently store >> 6.8TB of total results, going back 14 months, and 1.1TB of archives going >> back 1 week. >> >> 1500Apple MountainLion (Leaks) >> 1018Chromium Mac Release (Tests) >> 857 Chromium Linux Release (Tests) >> 532 Chromium Win Release (Tests) >> 371 Apple MountainLion Debug WK2 (Tests) >> 324 Apple Lion Debug WK2 (Tests) >> 299 Chromium Linux Release (Grid Layout) >> 173 GTK Linux 64-bit Release >> 171 Chromium Android Release (Tests) >> 160 EFL Linux 64-bit Debug WK2 >> 158 Apple Lion (Leaks) >> 145 EFL Linux 64-bit Release >> 113 EFL Linux 64-bit Debug >> 105 GTK Linux 32-bit Release >> 94 GTK Linux 64-bit Debug >> 94 GTK Linux 64-bit Release WK2 (Tests) >> 85 EFL Linux 64-bit Release WK2 >> 80 Apple Lion Release WK1 (Tests) >> 77 Apple Lion Release WK2 (Tests) >> 60 Apple Lion Debug WK1 (Tests) >> 60 Apple MountainLion Debug WK1 (Tests) >> 59 Apple MountainLion Release WK1 (Tests) >> 53 Qt Linux Release >> >> Archives are already pruning after 1 week. Next week, I'm going to start >> pruning results older than 14 months since we don't have much more space. If >> anyone thinks you need more than 14 months of results, let me know soon and >> we can see what our options are. >> >> -Bill >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo/webkit-dev > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Please avoid rolling out patches speculatively and reland them ASAP if you had to
The build cop / gardener / sheriff / whatever may not have local or easy access to a bot that reproduces the problem ... rolling it out might be the only feasible way to test in that case. On Tue, Dec 11, 2012 at 2:20 PM, Oliver Hunt wrote: > > On Dec 11, 2012, at 2:17 PM, Peter Kasting wrote: > > On Tue, Dec 11, 2012 at 2:14 PM, Oliver Hunt wrote: >> >> I don't understand why anyone is _speculatively_ rolling out patches. >> >> You should only be rolling it out if you _know_ the patch is bad. > > > Sometimes something bad happens to the tree, the sheriff doesn't know which > patch is responsible, and the change authors are not present to ask for > help. In a case like this the sheriff has to either do speculative rollouts > or leave the tree broken. > > Ideally, of course, change authors are around when something like this > happens. But maybe the bustage doesn't happen until much later, due to some > subtle/latent issue, or maybe the change author is in fact irresponsible. > > > Or the sheriff could actually see if rolling out a patch locally fixes the > problem. I'm not sure why they're considering "not testing" to be a valid > behaviour for someone who is ostensibly meant to be keeping things going in > the face of people who aren't testing. > > > > PK > > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Proposal: Make Ninja the default build-system for build-webkit --chromium
On Mon, Dec 10, 2012 at 1:30 AM, Jochen Eisinger wrote: > Will the buildbots use ninja or the "native" build tools? > > My only concern is that we're catching problems with e.g. MSVS only after we > roll the WebKit deps in chromium and one of the MSVS bots starts failing. > Eric is only suggesting changing update-webkit and build-webkit, which means that only the webkit.org bots would be affected. As long as the chromium.org canaries are still using chromium checkouts (and the native build systems), we'll still have coverage. Of course, your point is still valid for other scenarios where we don't have coverage of what we use on the official builds, but as Nico pointed out in a separate thread, so far this hasn't been a frequent problem. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] r- your own patches [was: Re: RenderArena: Teaching an old dog new tricks]
On Fri, Nov 16, 2012 at 7:42 AM, Julien Chaffraix wrote: >> Seconded. I also think only the one who submitted the patch can clear >> the r? flag. Others should NOT do that, please, even you are a >> reviewer. You can r- the patch if you believe it is bad. > > I disagree with that. You seem to think that patches falls into either > good or bad. However the reality is more complex and there are levels > of goodness and badness. I use r- for patches that I really think are > not in the right direction or shouldn't be landed: it is a statement > in this direction. Clearing the flag is for patches that are close > enough but still not up to our standards and that I want to kick off > the review queue. > > This is my reasoning on that and other people likely have different > views. However I don't think it's unreasonable to clear the flag > instead of r-'ing in some contexts. > I would be very confused if anyone but me cleared the r? on one of my bugs, unless it was obvious that the patch was old and buried and I had just forgotten about it. If I think a patch has a few flaws but is on the right track (and I just have questions), I will often leave the r? as-is and expect the contributor to figure it out (meaning that someone else might then be okay to r+ it. I will usually only r- a patch if I think the patch should not be landed as-is and I don't want someone else to r+ it. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 5:06 PM, Eric Seidel wrote: > > > > > Does seem pretty simple. > > > > > > is even shorter. :) > > I support getting rid of pixel tests. I suspect that some very dumb > scripts could turn large chunks of these existing pixel-tests into > ref-tests. I doubt that those would be the interesting ones though > (where platforms have divergent results). > I've been spending a fair amount of time working on this, actually. I think it's harder than you might think. I'm happy to talk further about it. >From what I can tell, we get most of divergence between platforms from the fact that we render text differently everywhere and we tend to render controls differently everywhere. Most of the time the differences are unrelated to what's actually being tested, unfortunately :(. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 4:59 PM, Darin Adler wrote: > On Nov 13, 2012, at 4:56 PM, Dirk Pranke wrote: > >> Wouldn't the fact that there are a large set of tests with the same result >> be an argument *for* doing the iframe thing? > > The simple hand-coded green square in upper left corner should be simple, > perhaps even simpler than the iframe thing. > >> What is the advantage to having 50 copies of a hand-coded "green square in >> upper left corner" reference test? > > Tests standing alone and being independent, easy to move around, revise, and > understand individually rather than as part of a suite. > Got it. It seems like referencing a well-known result makes things easier to understand, not harder, once you see it at least once, but I imagine it certainly depends on the complexity of the result. E.g., "PASSED" is better than "". Past about four lines it seems like the iframe would win. > I don’t have a strong objection to your iframe technique, but I’d start > simpler and do it only if it’s really needed. > I will also note that there are a large number of tests where we seem to have duplicate results, e.g., dom/html/level1 and dom/xhtml/level1 where the results are basically the same between the two suites, and having the xhtml results just be iframe'd versions of the html one seems like it would make sense. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 4:33 PM, Darin Adler wrote: > On Nov 13, 2012, at 4:01 PM, Dirk Pranke wrote: > >> It turns out that we have a reasonably large number of tests that produce >> the exact same pixel results. On chromium-mac on 10.8, for example, there >> are 2048 tests that share a result with some other test. 50 of them, for >> example, draw a green square in the upper left corner of the page (e.g., for >> svg/custom/root-element.html). > > It seems to me that when we find a pattern like this, we should create a > hand-coded reference test result. The fact that so many tests produce the > same pixels means that each reference can be used to move a lot of tests from > the pixel test to the reference test category. > > I don’t think we need to do that thing you said, as long as there > are large sets of tests with the same result, since if the payoff for each > reference is big enough, it should be affordable to hand-write the reference > file. > I don't think I'm understanding you. Wouldn't the fact that there are a large set of tests with the same result be an argument *for* doing the iframe thing? What is the advantage to having 50 copies of a hand-coded "green square in upper left corner" reference test? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] sharing more test references
Hi all, Currently, we have ~8000 pixel tests in the tree, and ~800 reference tests. I would like to make that first number a lot smaller. It turns out that we have a reasonably large number of tests that produce the exact same pixel results. On chromium-mac on 10.8, for example, there are 2048 tests that share a result with some other test. 50 of them, for example, draw a green square in the upper left corner of the page (e.g., for svg/custom/root-element.html). The way ref tests are currently implemented, there is no direct way to say "use test X as a reference for test Y" (although this is in the W3C specs for ref tests, which suggests using tags in the test html). Previous conversations on this topic have concluded with us thinking that we don't want to give up the convention of having an "-expected" file next to each test, and I still think this is a good idea. However, we can probably have our cake and eat it too in many cases by simply doing something like creating an -expected.html file that contains solely "". So, I would like to : 1) document that as an accepted pattern somewhere 2) start building up a directory of shared references, e.g., "LayoutTests/references/green-square.html" 3) start converting existing pixel tests to use these. Anyone think this won't, work, otherwise object, or have better ideas? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 1:04 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 12:09 PM, Glenn Adams wrote: >> >> On Tue, Nov 13, 2012 at 1:02 PM, Tony Chang wrote: >>> >>> I don't think we should support port specific ref test results. That >>> kind of misses the point of using a ref test in the first place. I mean, >>> you may as well check in port specific pixel results which are easier to >>> review for correctness. >>> >>> It may be the case that a ref test is not appropriate for what you're >>> trying to test. >> >> >> In the case of line break behavior, using reftests seem better than pixel >> tests, since there is less need for port-specific expectations. If I can >> come up with a text based approach (perhaps using range boundary rects), >> then I'll do so, but in the mean time, reftests seem a better option, >> especially for defining correctness based expectations (instead of merely >> regression based expectations). But we are straying from the original topic >> of this thread. > > > Did you try using the ahem font to see if you can get the same line break > behavior across ports? The ahem font doesn't always work for pixel tests > (anti-aliasing issues? maybe line height issues?), but I think the width of > each character should be the same. Yeah, there were lots of pixel test failures in mountain lion w/ ahem due to changes in aliasing. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] do you want "WK1" and "WK2" keywords in the TestExpectations files?
On Tue, Nov 13, 2012 at 12:29 PM, Dirk Pranke wrote: > On Tue, Nov 13, 2012 at 12:13 PM, Darin Adler wrote: >> I’d prefer an directory-based overlay/inheritance approach to sharing >> WebKit1- vs. WebKit2-specific expectations over in-file keywords. I’d like >> to share more than just a central expectations file, so we could share >> expected results or even WebKit1 or WebKit2-specific tests. >> > > We pretty much have this today (with platform/wk2 and > platform/mac-wk2). You're saying you'd prefer to add platform/wk1, > platform/mac-wk1, platform/mac-lion-wk1, and platform/mac-lion-wk2 > if/where necessary (and no keywords), right? > One specific example to motivate this ... imagine a test that we want to skip everywhere except current (and future) mac wk2. This would require a Skip in platform/mac/TestExpectations, a Pass in platform/mac-wk2/TestExpectations, and then a Skip in (a newly created, since we don't currently have this) platform/mac-lion-wk2/TestExpectations. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] do you want "WK1" and "WK2" keywords in the TestExpectations files?
On Tue, Nov 13, 2012 at 12:13 PM, Darin Adler wrote: > I’d prefer an directory-based overlay/inheritance approach to sharing > WebKit1- vs. WebKit2-specific expectations over in-file keywords. I’d like to > share more than just a central expectations file, so we could share expected > results or even WebKit1 or WebKit2-specific tests. > We pretty much have this today (with platform/wk2 and platform/mac-wk2). You're saying you'd prefer to add platform/wk1, platform/mac-wk1, platform/mac-lion-wk1, and platform/mac-lion-wk2 if/where necessary (and no keywords), right? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:16 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 12:04 PM, Darin Adler wrote: >> >> On Nov 13, 2012, at 12:02 PM, Tony Chang wrote: >> >> > I don't think we should support port specific ref test results. That >> > kind of misses the point of using a ref test in the first place. I mean, >> > you >> > may as well check in port specific pixel results which are easier to review >> > for correctness. >> >> I don’t agree that pixel results are easier to review for correctness. > > > Here is a ref test result from ietestcenter: > http://trac.webkit.org/browser/trunk/LayoutTests/ietestcenter/css3/flexbox/flexbox-flex-002-expected.htm > > Looking at that HTML file, it's not immediately obvious that the result is > correct. If I had a png file, it would easy to see if there's red showing > or not. > Actually reviewing the rendered pixels makes some things easier, but it can be difficult to figure out which rendered pixels matter and which don't. Even in the case of "if you see any red", it can be easy to miss one red pixel on an edge (and of course, there's issues around red-green colorblindness). Plus, you have to consider both the cost of reviewing the initial "correct output" and then the cost of reviewing changes to that output. in theory, we'll get far fewer changes to the correct output that need to be reviewed. For example, if we change how text is rendered (cough *mountain lion* cough) we'll need a new PNG but no change to the ref test. So, that's one less review you have to do. That said, I don't think there's a blanket rule here; I suspect some things are easier to review as pixel tests and some aren't. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:02 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke wrote: >> >> On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams wrote: >> > >> > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke >> > wrote: >> >> >> >> We don't currently support port-specific reftests (or at least, not >> >> very well), and we certainly should be trying to minimize where they >> >> occur. >> > >> > >> > Hmm, I actually used port specific reftest expectation files in a recent >> > patch [1] (since rolled out), and it appeared to work (as a way to >> > effectively rebaseline those expectations). So something seems to be >> > working. >> > >> > [1] http://trac.webkit.org/changeset/133529 >> > >> >> I expect it'll sort of work, but it won't be robust; you may hit weird >> behavior and/or bugs. We really haven't beaten on this aspect of >> things, and I don't know yet how much we want to. > > > I don't think we should support port specific ref test results. That kind > of misses the point of using a ref test in the first place. I mean, you may > as well check in port specific pixel results which are easier to review for > correctness. > > It may be the case that a ref test is not appropriate for what you're trying > to test. I think that there are probably cases where we will have differences in results because of (legal and entirely correct or permissible) differences in the implementations and in this case a reftest can still be an improvement over maintaining N platform-specific pixel versions. The obvious example is when we haven't implemented features yet (or have bugs). The W3C's spec for handling reftests also gives you a way to say "a test passes if any of these N references may match", which is fairly consistent with the idea that platform specific references are okay in some cases. As to whether pixel-tests are easier to review for correctness than reftests or not, I think it depends on the test. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] do you want "WK1" and "WK2" keywords in the TestExpectations files?
On Tue, Nov 13, 2012 at 11:59 AM, Raphael Kubo da Costa wrote: > Dirk Pranke writes: > >> So, it seems like WK1 and WK2 keywords might be useful. However, I >> don't really want to add more divergence between ports, so it would be >> nice to have everyone agree to use it if we were to add it. >> >> What do you all think? Would you like this feature, and would you all >> use it ? > > At least on the EFL side I think things are good the way they are: we > have platform/efl for common stuff and platform/efl-wk1 and > platform/efl-wk2 for WK1- or WK2-specific stuff (not only > TestExpectations files but also test results). If we got rid of those > and put everything together in platform/efl, I think we'd end up with a > very big TestExpectations file and don't know what we'd do with the > occasional different results for WK1 and WK2. > >> However, this is a little awkward and gets worse if you also need to >> support different expectations for multiple different configs (e.g., >> mac-lion vs mac-snowleopard vs mac-mountainlion). > > It wouldn't really solve this problem, right? > Well, it might help with the combinatorial explosion of -wk2 files, so you wouldn't need a mac-lion-wk2/TestExpectations. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] do you want "WK1" and "WK2" keywords in the TestExpectations files?
Hi all, I occasionally get asked if the TestExpectations syntax supports a way to distinguish different results for WebKit1 and WebKit2 via keywords. We currently don't do this, and different ports have worked around this in slightly different ways by using dedicated wk2-specific TestExpectations files and sometimes wk1-specific TestExpectations files. However, this is a little awkward and gets worse if you also need to support different expectations for multiple different configs (e.g., mac-lion vs mac-snowleopard vs mac-mountainlion). So, it seems like WK1 and WK2 keywords might be useful. However, I don't really want to add more divergence between ports, so it would be nice to have everyone agree to use it if we were to add it. What do you all think? Would you like this feature, and would you all use it ? (Since I don't regularly switch between WK1 and WK2 I don't have a strong feeling here beyond what I've written above). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
Right. On Tue, Nov 13, 2012 at 11:36 AM, Darin Adler wrote: > If we do add base test expectations shared by all platforms, please don’t put > the file into LayoutTests/platform/generic; just put it at the top level of > LayoutTests. > > -- Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams wrote: > > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke wrote: >> >> We don't currently support port-specific reftests (or at least, not >> very well), and we certainly should be trying to minimize where they >> occur. > > > Hmm, I actually used port specific reftest expectation files in a recent > patch [1] (since rolled out), and it appeared to work (as a way to > effectively rebaseline those expectations). So something seems to be > working. > > [1] http://trac.webkit.org/changeset/133529 > I expect it'll sort of work, but it won't be robust; you may hit weird behavior and/or bugs. We really haven't beaten on this aspect of things, and I don't know yet how much we want to. >> >> At any rate, we encourage people these days to check in expected >> failures rather than suppressing them using the TestExpectations >> files. > > > The problem is essentially a chicken and egg problem. I don't know what the > per-port failures will be ahead of time, but I do know the set of "correct" > expectations. Since I am (independently) unable to build/test all ports run > by build bots, I would like to commit the set of tests plus known good > expectations as a preliminary step (with a generic skip all tests for all > ports), and then subsequently commit the feature itself, and then > subsequently override the generic skip on a port specific basis, effectively > re-enabling the tests on a port by port basis as I refine the feature patch > (as needed) to handle port specific behavioral differences. > I think this is a reasonable approach. I would be interested to hear if others had alternatives they preferred. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Perhaps we should discuss why you need them (in a separate thread with a separate subject line)? It sounds like this largely has to do with what features are enabled and/or supported? Perhaps said tests should just be skipped in the meantime? At any rate, we encourage people these days to check in expected failures rather than suppressing them using the TestExpectations files. I am hoping this week to finally get back to working on the "-failing" feature so you can at least distinguish expectations that are known to be failing/incorrect more easily. -- Dirk On Tue, Nov 13, 2012 at 11:02 AM, Glenn Adams wrote: > That would seem to work only if the test(s) fail the same way on all ports. > In the case that I'm working from, I'm using reftests, where I know the > correct expectations, but the actual behavior will (does) differ on > different ports (when the corresponding feature is committed). > > I would like to be able to (independently) commit new reftests *and* their > known good expectation counterparts (that should apply on all ports), and > then subsequently commit an independent patch that implements the expected > behavior (on some but not all ports), and the comment further follow-on > patches that fix the failing ports (possibly by writing new expectations for > those specific ports). > > On Tue, Nov 13, 2012 at 11:52 AM, Ryosuke Niwa wrote: >> >> It is customary to add a failing test expectation (i.e. *-expected.txt >> file that contains the said failure) in such cases. >> >> On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams wrote: >>> >>> Just checking, but I don't see a way to add test expectations that apply >>> generically (to all ports). >>> >>> It would be nice to have something like >>> LayoutTests/platform/generic/TestExpectations to which one could add new >>> tests that are known to fail everywhere (e.g., because the code that >>> implements a feature that is tested by those tests is not yet committed), >>> but which will (at some point in the near future) not fail (when the code >>> that is to be tested is committed). >>> >>> At present, it seems that if one wishes to do this, then it is necessary >>> to add entries to the each base port expectations (i.e., chromium, mac, win, >>> etc), which is rather annoying. >>> >>> If there is no objection to adding such a "generic" platform expectations >>> file, then I will undertake to do so. > > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
That said, I think adding a generic TestExpectations file is a good idea; it would allow us to replace the "-disabled" convention for some tests and allow us to skip tests that were temporarily crashing or timing out everywhere (which *-expected or *-failing wouldn't help with). It should be easy enough to add. -- Dirk On Tue, Nov 13, 2012 at 10:52 AM, Ryosuke Niwa wrote: > It is customary to add a failing test expectation (i.e. *-expected.txt file > that contains the said failure) in such cases. > > On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams wrote: >> >> Just checking, but I don't see a way to add test expectations that apply >> generically (to all ports). >> >> It would be nice to have something like >> LayoutTests/platform/generic/TestExpectations to which one could add new >> tests that are known to fail everywhere (e.g., because the code that >> implements a feature that is tested by those tests is not yet committed), >> but which will (at some point in the near future) not fail (when the code >> that is to be tested is committed). >> >> At present, it seems that if one wishes to do this, then it is necessary >> to add entries to the each base port expectations (i.e., chromium, mac, win, >> etc), which is rather annoying. >> >> If there is no objection to adding such a "generic" platform expectations >> file, then I will undertake to do so. >> >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo/webkit-dev >> > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] commit-queue bot missing a dependency? ("ImportError: No module named logilab.astng.utils")
Also, I just landed a change (r134445) that should in theory auto-fix this. Let me know if anyone is still seeing issues. -- Dirk On Tue, Nov 13, 2012 at 10:53 AM, Dirk Pranke wrote: > % rm -fr Tools/Scripts/webkitpy/thirdparty/autoinstalled/.pylint.url \ > > Tools/Scripts/wekbitpy/thirdparty/autoinstalled/.pylint-0.25.1.url > \ > Tools/Scripts/webkitpy/thridparty/autoinstalled/pylint > > This is roughly the equivalent of a clobber for the autoinstalled > packages (well, one of them anyway). > > I think things got partially installed a weel ago when I had a bug in > the configuration and at the moment it doesn't automatically recover. > I'll look into seeing if there's a more robust way to recover things. > > -- Dirk > > On Tue, Nov 13, 2012 at 10:44 AM, Peter Beverloo wrote: >> I'm seeing the same on the Android bot. What is the right way to fix this? >> >> Cheers, >> Peter >> >> On 13 Nov 2012 18:31, "Dirk Pranke" wrote: >>> >>> Argh, yeah. Unfortunately, I'm not sure what triggers this problem. >>> It's easy enough to fix as soon as I can find someone w/ access to the >>> CQ. >>> >>> -- Dirk >>> >>> On Tue, Nov 13, 2012 at 7:50 AM, Adam Barth wrote: >>> > Sounds related to a recent changed made by dpranke. >>> > >>> > Adam >>> > >>> > >>> > On Tue, Nov 13, 2012 at 7:33 AM, Jussi Kukkonen >>> > wrote: >>> >> >>> >> commit-queue bot seems to be missing a pylint dependency? This is what >>> >> I got >>> >> for my rebaseline in https://bugs.webkit.org/show_bug.cgi?id=102059: >>> >> --- >>> >> Failed to run >>> >> "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', >>> >> '--status-host=queues.webkit.org', '-..." exit_code: 2 >>> >> >>> >> Last 500 characters of output: >>> >> yle/checkers/python.py", line 31, in >>> >> from webkitpy.thirdparty.autoinstalled.pylint import lint >>> >> File >>> >> >>> >> "/mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/lint.py", >>> >> line 31, in >>> >> from pylint.checkers import utils >>> >> File >>> >> >>> >> "/mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/checkers/__init__.py", >>> >> line 44, in >>> >> from logilab.astng.utils import ASTWalker >>> >> ImportError: No module named logilab.astng.utils >>> >> >>> >> Full output: http://queues.webkit.org/results/14823414 >>> >> --- >>> >> >>> >> - jussi >>> >> >>> >> ___ >>> >> webkit-dev mailing list >>> >> webkit-dev@lists.webkit.org >>> >> http://lists.webkit.org/mailman/listinfo/webkit-dev >>> >> >>> ___ >>> webkit-dev mailing list >>> webkit-dev@lists.webkit.org >>> http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] commit-queue bot missing a dependency? ("ImportError: No module named logilab.astng.utils")
% rm -fr Tools/Scripts/webkitpy/thirdparty/autoinstalled/.pylint.url \ Tools/Scripts/wekbitpy/thirdparty/autoinstalled/.pylint-0.25.1.url \ Tools/Scripts/webkitpy/thridparty/autoinstalled/pylint This is roughly the equivalent of a clobber for the autoinstalled packages (well, one of them anyway). I think things got partially installed a weel ago when I had a bug in the configuration and at the moment it doesn't automatically recover. I'll look into seeing if there's a more robust way to recover things. -- Dirk On Tue, Nov 13, 2012 at 10:44 AM, Peter Beverloo wrote: > I'm seeing the same on the Android bot. What is the right way to fix this? > > Cheers, > Peter > > On 13 Nov 2012 18:31, "Dirk Pranke" wrote: >> >> Argh, yeah. Unfortunately, I'm not sure what triggers this problem. >> It's easy enough to fix as soon as I can find someone w/ access to the >> CQ. >> >> -- Dirk >> >> On Tue, Nov 13, 2012 at 7:50 AM, Adam Barth wrote: >> > Sounds related to a recent changed made by dpranke. >> > >> > Adam >> > >> > >> > On Tue, Nov 13, 2012 at 7:33 AM, Jussi Kukkonen >> > wrote: >> >> >> >> commit-queue bot seems to be missing a pylint dependency? This is what >> >> I got >> >> for my rebaseline in https://bugs.webkit.org/show_bug.cgi?id=102059: >> >> --- >> >> Failed to run >> >> "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', >> >> '--status-host=queues.webkit.org', '-..." exit_code: 2 >> >> >> >> Last 500 characters of output: >> >> yle/checkers/python.py", line 31, in >> >> from webkitpy.thirdparty.autoinstalled.pylint import lint >> >> File >> >> >> >> "/mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/lint.py", >> >> line 31, in >> >> from pylint.checkers import utils >> >> File >> >> >> >> "/mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/checkers/__init__.py", >> >> line 44, in >> >> from logilab.astng.utils import ASTWalker >> >> ImportError: No module named logilab.astng.utils >> >> >> >> Full output: http://queues.webkit.org/results/14823414 >> >> --- >> >> >> >> - jussi >> >> >> >> ___ >> >> webkit-dev mailing list >> >> webkit-dev@lists.webkit.org >> >> http://lists.webkit.org/mailman/listinfo/webkit-dev >> >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] commit-queue bot missing a dependency? ("ImportError: No module named logilab.astng.utils")
Argh, yeah. Unfortunately, I'm not sure what triggers this problem. It's easy enough to fix as soon as I can find someone w/ access to the CQ. -- Dirk On Tue, Nov 13, 2012 at 7:50 AM, Adam Barth wrote: > Sounds related to a recent changed made by dpranke. > > Adam > > > On Tue, Nov 13, 2012 at 7:33 AM, Jussi Kukkonen > wrote: >> >> commit-queue bot seems to be missing a pylint dependency? This is what I got >> for my rebaseline in https://bugs.webkit.org/show_bug.cgi?id=102059: >> --- >> Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', >> '--status-host=queues.webkit.org', '-..." exit_code: 2 >> >> Last 500 characters of output: >> yle/checkers/python.py", line 31, in >> from webkitpy.thirdparty.autoinstalled.pylint import lint >> File >> "/mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/lint.py", >> line 31, in >> from pylint.checkers import utils >> File >> "/mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/checkers/__init__.py", >> line 44, in >> from logilab.astng.utils import ASTWalker >> ImportError: No module named logilab.astng.utils >> >> Full output: http://queues.webkit.org/results/14823414 >> --- >> >> - jussi >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo/webkit-dev >> ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] The --test-list option behavior in NRWT
On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek wrote: > Hi WebKitties, > > I'd like to see in what scale the --test-list option in NRWT is used and > whether anyone would object a change in how its usage behaves. > > Currently the test gathering takes into account any paths that were passed > in through the command line and any paths listed in the file to which a > possible --test-file option points[1]. These paths are then expanded if > necessary (due to platform-specific tests and virtual test suites) and all > the tests to which these paths apply are gathered[2]. All the gathered tests > are then either sorted into a natural order or randomized (if the > --randomize-order option is used)[3]. > > I'd like to propose a change in the behavior when using the --test-list > option. When used, the tests would be gathered only from the paths listed in > the file to which the option points. Also, the gathered tests would be > sorted to follow the order of the paths in the test list file. For instance, > consider the following two entries being placed in the test list file: > > c/d/e.html > a/b/ > > The current behavior would gather all the tests to which these two paths > apply and sort them in this order (on the premise that no other paths are > passed through the command line arguments): > > a/b/1.html > a/b/2.html > c/d/e.html > > The new behavior would place the c/d/e.html test at the top of the list but > sort the other two tests in order, like this: > > c/d/e.html > a/b/1.html > a/b/2.html > > The idea behind this is to be able to specify the exact order in which the > tests should be run. I'm currently playing around with a tool that randomly > chooses a certain number of tests, runs them and bisects down the first > regression that occurs (if any) to the smallest possible ordered list of > tests that reproduces that regression. The proposed change makes this a > simple task from the test listing perspective and I'd argue that exact test > ordering is the main point of an option such as --test-list. > > Currently I'm using an untested workaround of adding a new option that > specifies the tests should be reordered back to the original order that the > test list file provided, but I'd like to move on with implementing the > change in behavior if everyone feels it's OK and won't tamper with his/her > workflow. > > -Z > > > [1] > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46 > [2] > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576 > [3] > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309 > Hi, I have used the option from time to time, and I have also wanted from time to time the ability to run tests in a specific order. This change would be fine by me. If people want to ensure that tests run in the old order, they can always sort the test list. I would also be fine changing the code so that order is preserved for tests passed in on the command line as well. (e.g. so "run-webkit-tests foo.html bar.html" runs them in that order). I think it would be good to keep the behavior consistent across the two methods of specifying names. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Does anyone still use the TestFailures app?
On Mon, Nov 5, 2012 at 12:47 PM, Osztrogonac Csaba wrote: > Hi, > > We use it and http://build.webkit.sed.hu/TestFailures/ for gardening, > it is the first step we usually do if determining who broke what about > the waterfall isn't trivial. > > On http://build.webkit.sed.hu/TestFailures/ we use a very old copy > of test failures and it still works fine. > > Dirk Pranke írta: > >> http://build.webkit.org/TestFailures/ >> >> I think Adam Roben was working on this a year or so ago. It appears to >> be broken at the moment (it's likely that I broke it, in fact), but >> before I spend much time fixing it I thought I'd check. > > > It works for me more or less, but I got strange link names: > http/tests/security/cross-origin-plugin-private-browsing-toggled.html: > [object DocumentFragment] > > Maybe one of the garden-o-matic patches broke it somehow. > > >> I've never actually used it myself, so I'm not sure what all it was >> supposed to do; it looks like it overlaps in functionality some with >> the flakiness dashboard, but was probably written before the flakiness >> dashboard worked with the build.webkit.org bots and everyone was >> converted to using NRWT. > > >> If anyone is still using it (or would if it was actually working) in >> preference to the flakiness dashboard, can you let me know why? > > We still use it, because it is very simple, it works almost always, > it isn't hakced day by day and its output is very very simple. We > get the result - which revision broke a given test, which are the > related bug reports - with _one_ click on the name of the slave. > > It is more complicated to do same thing on flakiness dashboard: > - select webkit.org from group > - select a given slave > - select "tests with wrong expectations" > - (unselect flaky) > - find manually the last good revision for test by test > but it is _impossible_ if the breakage is too old > > >> Ideally I'd like to get rid of it and roll any good features it had >> into the flakiness dashboard, but I'm happy to fix it and/or keep it >> around if it does other things I'm not aware of or if people are still >> using it. > > > Please don't remove this good and simple tool, we use it day by day. > Thanks, Ossy! I guess I'll figure out how to fix it and go from there :). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Does anyone still use the TestFailures app?
http://build.webkit.org/TestFailures/ I think Adam Roben was working on this a year or so ago. It appears to be broken at the moment (it's likely that I broke it, in fact), but before I spend much time fixing it I thought I'd check. I've never actually used it myself, so I'm not sure what all it was supposed to do; it looks like it overlaps in functionality some with the flakiness dashboard, but was probably written before the flakiness dashboard worked with the build.webkit.org bots and everyone was converted to using NRWT. If anyone is still using it (or would if it was actually working) in preference to the flakiness dashboard, can you let me know why? Ideally I'd like to get rid of it and roll any good features it had into the flakiness dashboard, but I'm happy to fix it and/or keep it around if it does other things I'm not aware of or if people are still using it. Cheers, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] does anyone still use 'webkit-patch rebaseline-server'?
It is/was intended to be useful for quickly reviewing and rebaselining a bunch of failures in a local (on-disk) checkout. It's been largely unmaintained and ignored for quite some time in favor of garden-o-matic. I have recently started to land some patches that will make garden-o-matic work locally as well as with the bots, and it will ultimately replace rebaseline-server; I'm attempting to consolidate all of our different change-reviewing UIs so that they are more consistent and share more code. The "local garden-o-matic" should roughly work now, but the UI isn't well-tuned for this use case and there's some things left do to. If you have any features you would like to add to the garden-o-matic UI (or want to make sure are transferred over from rebaseline-server), now would be a good time to mention them. And, if no one speaks up to say that rebaseline-server is still being used, I will assume I can delete it when I'm ready to do so. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Some stderr output missing when using run-webkit-tests
That would still be a bug, and still new to me :) -- Dirk On Mon, Oct 29, 2012 at 4:04 PM, Dana Jansens wrote: > On Mon, Oct 29, 2012 at 6:59 PM, Dirk Pranke wrote: >> If that's the case, it's a bug, and new to me. > > The output was present on the results page, but it would only include > the first, maybe, 60 lines or so. > > - Dana > >> >> -- Dirk >> >> On Mon, Oct 29, 2012 at 3:42 PM, Terry Anderson >> wrote: >>> I was actually noticing that some of the stderr output was missing from a >>> failing test, not a passing one. >>> >>> Terry >>> >>> >>> On Sun, Oct 28, 2012 at 8:42 PM, Dirk Pranke wrote: >>>> >>>> As Balazs said, we don't save the stderr output from tests that pass. >>>> So, you don't have to crash, but your tests have to at least fail. It >>>> wouldn't be hard to change that somehow ... >>>> >>>> -- Dirk >>>> >>>> On Sun, Oct 28, 2012 at 4:29 PM, Terry Anderson >>>> wrote: >>>> > Hi webkit-dev, >>>> > >>>> > When I include fprintf(stderr, ...) statements in WebKit code that I >>>> > expect >>>> > to be executed when running a set of layout tests, the summary page of >>>> > run-webkit-tests will sometimes only show a subset of these statements. >>>> > However, when I add a CRASH() somewhere in the code, the "missing" >>>> > stderr >>>> > output will appear on the summary page. Has anyone else experienced this >>>> > issue? Is there a way to force run-webkit-tests to display all stderr >>>> > output >>>> > without needing to force a crash at a particular point in the code? >>>> > >>>> > Terry >>>> > >>>> > ___ >>>> > webkit-dev mailing list >>>> > webkit-dev@lists.webkit.org >>>> > http://lists.webkit.org/mailman/listinfo/webkit-dev >>>> > >>> >>> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Some stderr output missing when using run-webkit-tests
If that's the case, it's a bug, and new to me. -- Dirk On Mon, Oct 29, 2012 at 3:42 PM, Terry Anderson wrote: > I was actually noticing that some of the stderr output was missing from a > failing test, not a passing one. > > Terry > > > On Sun, Oct 28, 2012 at 8:42 PM, Dirk Pranke wrote: >> >> As Balazs said, we don't save the stderr output from tests that pass. >> So, you don't have to crash, but your tests have to at least fail. It >> wouldn't be hard to change that somehow ... >> >> -- Dirk >> >> On Sun, Oct 28, 2012 at 4:29 PM, Terry Anderson >> wrote: >> > Hi webkit-dev, >> > >> > When I include fprintf(stderr, ...) statements in WebKit code that I >> > expect >> > to be executed when running a set of layout tests, the summary page of >> > run-webkit-tests will sometimes only show a subset of these statements. >> > However, when I add a CRASH() somewhere in the code, the "missing" >> > stderr >> > output will appear on the summary page. Has anyone else experienced this >> > issue? Is there a way to force run-webkit-tests to display all stderr >> > output >> > without needing to force a crash at a particular point in the code? >> > >> > Terry >> > >> > ___ >> > webkit-dev mailing list >> > webkit-dev@lists.webkit.org >> > http://lists.webkit.org/mailman/listinfo/webkit-dev >> > > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Mon, Oct 29, 2012 at 4:17 AM, Maciej Stachowiak wrote: > > On Oct 28, 2012, at 3:30 PM, Antti Koivisto wrote: > >> We could clear the cache between tests but run each test twice in a row. >> Second run will then happen with deterministically pre-populated cache. That >> would both make things more predictable and improve our test coverage for >> cached cases. Unfortunately it would also slow down testing significantly, >> though less than 2x. > > I actually really like this idea. Doing it this way would effectively run > each test both completely uncached, and fully cached, which would be better > test coverage than our current approach. Can we get an estimate on what this > would cost if applied to our whole test suite? Could we do it for just a > subset of the tests? > > (BTW I think this is better than the "virtual test suite" approach suggested > by Dirk; running the test with all its resources cached from having loaded it > immediately before is more reliable and better test coverage than running it > later as part of some sequence that doesn't clear the cache.) > > Does anyone strongly object to this approach? It seems way better to me than > other options discussed on this thread. > I would like to understand the proposal and thinking behind it a little better. First, I'm not quite sure what the intent is here ... are you thinking that maybe this would help ensure tests get more isolated environments and yet we still exercise the cache all the time? Second, today you could get close by saying NRWT --repeat-each=2 --batch-size=2; that would run each test twice in succession and then restart DRT. Are you suggesting that we'd change something such that we'd clear the cache rather than restarting DRT? Third, it seems like it would probably introduce a *lot* of extra redundant test-running. I would imagine the very large majority of our tests are not sensitive to the cache contents at all and so running them twice doesn't buy you anything. An advantage of the virtual test suite approach is that you can specify which subsets of tests you'd like to run; perhaps you could combine that with running them twice? Lastly, it seems like this is focusing on the wrong thing; wouldn't it be better to try and write specific tests to provoke different configurations and code paths through the cache? Are you suggesting we'd run all the tests twice as an interim measure until we had better specific test suites? -- Dirk > Regards, > Maciej > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Mon, Oct 29, 2012 at 5:48 AM, Maciej Stachowiak wrote: > > On Oct 28, 2012, at 10:09 PM, Dirk Pranke wrote: > >> >> On Sun, Oct 28, 2012 at 6:32 AM, Maciej Stachowiak wrote: >>> >>> I think the nature of loader and cache code is that it's very hard to make >>> tests which always fail deterministically when regressions are introduced, >>> as opposed to randomly. The reason for this is that bugs in these areas are >>> often timing-dependent. I think it's likely this tendency to fail randomly >>> will be the case whether or not the tests are trying to explicitly test the >>> cache or are just incidentally doing so in the course of other things. >>> >> >> I am not familiar with the loader and caching code in webkit, but I >> know enough about similar problem spaces to be puzzled by why it's >> impossible to write tests that can adequately test the code. > > Has anyone claimed that? I think "impossible to write tests that can > adequately test the code" is not a position that anyone in this thread has > taken, certainly not me above. > > My claim is only that many classes of loader and cache bugs, when first > introduced, are likely to cause nondeterministic test failures. And further, > this is likely to be the case even if tests are written to target that > subsystem. That's not the same as saying adequate tests are impossible. I'm sorry, I didn't mean "impossible" literally. Please strike that, as it sounds like it has just made a confusing situation worse. But, you did claim that it would be "very hard to make tests that always fail deterministically", and I don't see why that's true? Testing things that are timing-dependent only require that you be able to control or simulate time. It may be that this is hard to do with layout tests, but it's pretty straightforward with unit tests that allow you to control the layers above and below the cache. > It just means to have good testing of some areas of the code, we need a good > way of dealing with nondeterministic failures. This is backwards. If you *don't* have good testing, more of your failures are likely to show up sporadically, which leads you to want to build tools for them. Randomized testing is a helpful tool to use *alongside* focused testing to ensure coverage, but should not be used as a replacement. >> >>> What I personally would most wish for is good tools to catch when a test >>> starts failing nondeterministically, and to identify the revision where the >>> failures began. The reason we hate random failures is that they are hard to >>> track down and diagnose. But some types of bugs are unlikely to manifest in >>> a purely deterministic way. It would be good if we had a reliable and >>> useful way to catch those types of bugs. >> >> This is a fine idea -- and I'm always happy to talk about ways we can >> improve our test tooling, please feel free to start a separate thread >> on these issues -- but I don't want to lose sight of the main issue >> here. > > I think the problem I identified -- that it's overly hard to track down and > diagnose regressions that cause tests to fail only part of the time -- is > more important and more fundamental than any of the three problems that you > cite below. Our test infrastructure ultimately exists to help us notice and > promptly fix regressions, and for some types of regressions, namely those > that do not manifest 100% of the time, it is not working so well. The > problems you mention are all secondary consequences of that fundamental > problem, in my opinion. First of all, this isn't an either/or situation. We should be capable of addressing all of these issues in parallel. Second, I don't see how the existence of bugs in the code, the lack of test isolation, or the lack of good test coverage for certain layers of the code follow from not having good tools to triage intermittent failures? That seems like putting the cart before the horse. Third, are you familiar with the flakiness dashboard? http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20webkit.org&builder=Apple%20Lion%20Debug%20WK1%20(Tests) Does it not do exactly what you're describing? Are there things that you would like added? If it would be helpful for us to have a meeting or something to help explain how this works, I'm sure we could set one up. > > - Maciej > >> >> It sounds like we've identified three existing problems - please >> correct me if I'm misstating them: >> >> 1. There appears to be a bug in the caching code that is causing
Re: [webkit-dev] Some stderr output missing when using run-webkit-tests
As Balazs said, we don't save the stderr output from tests that pass. So, you don't have to crash, but your tests have to at least fail. It wouldn't be hard to change that somehow ... -- Dirk On Sun, Oct 28, 2012 at 4:29 PM, Terry Anderson wrote: > Hi webkit-dev, > > When I include fprintf(stderr, ...) statements in WebKit code that I expect > to be executed when running a set of layout tests, the summary page of > run-webkit-tests will sometimes only show a subset of these statements. > However, when I add a CRASH() somewhere in the code, the "missing" stderr > output will appear on the summary page. Has anyone else experienced this > issue? Is there a way to force run-webkit-tests to display all stderr output > without needing to force a crash at a particular point in the code? > > Terry > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Sun, Oct 28, 2012 at 2:47 PM, Ryosuke Niwa wrote: > On Sun, Oct 28, 2012 at 2:09 PM, Dirk Pranke wrote: >> >> > On Oct 26, 2012, at 11:11 PM, Ryosuke Niwa wrote: >> >> I’m sure Antti, Alexey, and others who have worked on the loader and >> >> other parts of WebKit are happy to write those tests or list the kind of >> >> things they want to test. Heck, I don’t mind writing those tests if >> >> someone >> >> could make a list. >> >> >> >> I totally sympathize with the sentiment to reduce the test flakiness >> >> but loader and cache code have historically been under-tested, and we’ve >> >> had >> >> a number of bugs detected only by running non-loader tests consecutively. >> >> >> >> On the contrary, we’ve had this DRT behavior for ages. Is there any >> >> reason we can’t wait for another couple of weeks or months until we add >> >> more >> >> loader & cache tests before making the behavior change? >> > >> >> Please correct me if I'm misinformed, but it's been three months since >> this issue was first raised, and it doesn't sound like they've been >> writing those tests or are happy to do so, and despite people asking >> on this thread, they haven't been listing the kinds of tests they >> think they need. > > > I don't think anyone else had suggested adding tests as an option or set a > deadline until I suggested yesterday (or when I did in my original reply to > the thread). In fact, since Ami posted his reply on October 26th 1:20AM > (PST), many contributors from non-PST timezones haven't even had a chance to > read his post during normal business hours. > > Given that I'd think it's totally unreasonable to land the patch as is > without giving people reasonable amount of time (~one week) to respond to > this thread. > Both you and Eric U suggesting adding new tests for this in the original thread on 8/9; in fact, this whole issue got a fair amount of discussion then, and this round hasn't really added anything new. I'm happy to wait a little longer if others want to come up with some other suggestions; I apologize if my previous response sounded like I was throwing down a gauntlet or otherwise not open to ideas; that was definitely not my intent. Rather, I was attempting to say that unless someone else has other ideas, the right path forward seems fairly clear to me and that I intended to proceed down it. Does that seem more reasonable to you? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
> On Oct 26, 2012, at 11:11 PM, Ryosuke Niwa wrote: >> I’m sure Antti, Alexey, and others who have worked on the loader and other >> parts of WebKit are happy to write those tests or list the kind of things >> they want to test. Heck, I don’t mind writing those tests if someone could >> make a list. >> >> I totally sympathize with the sentiment to reduce the test flakiness but >> loader and cache code have historically been under-tested, and we’ve had a >> number of bugs detected only by running non-loader tests consecutively. >> >> On the contrary, we’ve had this DRT behavior for ages. Is there any reason >> we can’t wait for another couple of weeks or months until we add more loader >> & cache tests before making the behavior change? > Please correct me if I'm misinformed, but it's been three months since this issue was first raised, and it doesn't sound like they've been writing those tests or are happy to do so, and despite people asking on this thread, they haven't been listing the kinds of tests they think they need. Have we actually made any progress here, or was the issue dropped until Ami raised it again? It seems like the latter to me ... again, please correct me if this is being actively worked on, because that would change the whole tenor of this debate. On Sun, Oct 28, 2012 at 6:32 AM, Maciej Stachowiak wrote: > > I think the nature of loader and cache code is that it's very hard to make > tests which always fail deterministically when regressions are introduced, as > opposed to randomly. The reason for this is that bugs in these areas are > often timing-dependent. I think it's likely this tendency to fail randomly > will be the case whether or not the tests are trying to explicitly test the > cache or are just incidentally doing so in the course of other things. > I am not familiar with the loader and caching code in webkit, but I know enough about similar problem spaces to be puzzled by why it's impossible to write tests that can adequately test the code. Is the caching disk-based, and maybe running tests in parallel screwing with things? If so, then maybe the fact that we now run tests in parallel is why this is a problem now and hasn't been before? Or maybe the fact that a given process doesn't always see the same tests in the same order is the problem? > Unfortunately, it's very tempting when a test is failing randomly to blame > the test rather than to investigate whether there is an actual regression > affecting it. And sometimes it really is the test's fault. But sometimes it > is a genuine bug in the code. > > On the other hand, nondetermisitic test failures make it harder to use test > infrastructure in general. > > These are difficult things to reconcile. The original philosophy of WebKit > tests is to test end-to-end under relatively realistic conditions, but at the > same time unpredictability makes it hard to stay at zero regressions. > Exactly. Personally, the cost of unpredictability in the test infrastructure is so much higher than the value we're getting (implicitly) that this is a no-brainer to me. There are some tradeoffs (like running tests in parallel) that are worth it, but this isn't one of them. I am happy to explain further my thinking and standards if there's interest. Hopefully that partially answers Alexey's questions about where we should draw the line in trying to make our tests deterministic and hermetic: do everything you reasonably can. We're not picking on caching here. > I think making different ports do testing under different conditions makes it > more likely that some contributors will introduce regressions without > noticing, leaving it for others to clean up. So it's regrettable if we go > that way because we are unable to reach consensus. I agree that it is bad to have different ports behaving differently, and I would like to avoid that as well. I don't want any port suffering from flaky tests, but I also don't think it's reasonable to have one group force that on everyone else indefinitely, either. I am also fine with having some way to test systems more non-deterministically in a way to expose more bugs, but that needs to be clearly separated from the other testing we do; it is an unfair cost to impose on the rest of the system otherwise and should be tolerated only if we have no other choice. We have other choices. > Creating some special opt-in --antti mode would be even worse, as it's almost > certain that failures would creep into a mode that nobody runs. > This comment (and Antti's suggestion, below) makes me think that you didn't understand my "virtual test suite" suggestion; that's not surprising, since Apple doesn't actual use this feature of NRWT yet. A virtual test suite is a way of saying (re-)run the tests under directory X with command-line flags Y and Z, and put the results in a new directory. For example, Chromium runs all of the tests in fast/canvas twice, once "normally" using the regular software
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Fri, Oct 26, 2012 at 3:02 PM, Ryosuke Niwa wrote: > On Fri, Oct 26, 2012 at 2:57 PM, Dirk Pranke wrote: >> >> Perhaps a slight variant of this is that we can agree to make the >> changes on the Chromium port to clear the cache (much like the Qt and >> EFL ports already do), and you can continue to not clear the cache on >> the Apple Mac port until you feel comfortable that you've added >> additional tests? > > > That sounds much more reasonable than adding a flag. > I will note that if we were to add a flag, then we could set up a "virtual test suite" to re-run some chunk of tests with the flag automatically, and that way the overhead of running both configurations is near zero (though of course the test run time is longer). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Fri, Oct 26, 2012 at 12:43 PM, Alexey Proskuryakov wrote: > > 26.10.2012, в 11:04, Antti Koivisto написал(а): > >> The reality is that this "test coverage" today shows up as flakiness and >> so is ignored anyway, meaning we don't actually have useful coverage here. >> Even when flakiness is investigated, the "fix" is to cache-bust using unique >> URL params, which just means we "lose" the coverage you describe for that >> test, anyway. > > > I think that this is the real issue here. Test flakiness is very important > to investigate, this often leads to discovery of bad bugs, including > security ones. The phrase "flaky test" often misplaces the blame. > > When making cache related changes I have frequently found bugs from my > patches because some seemingly random test started failing and I > investigated. Without the test coverage some of those bugs would probably > now be in the tree. > I agree strongly with both of these sentiments. My experience, though, is that most people are disinclined to actually spend the time figuring out why a test is flaky; in addition, it can be very difficult to even figure out if a test has just started becoming flaky or if your change introduced flakiness. As a result, people tend to just suppress or skip over flaky tests. > > I agree with Antti. Finding regressions is what tests are for, and it would > be difficult to make enough explicit tests to compensate for such loss of > coverage. It would certainly be very unfortunate to lose test coverage > without even an attempt to compensate for that. Because of what I've written above, having flaky tests is causing us to lose coverage today. So, I suspect that with this change we'll be able to unsuppress a number of failures and re-gain lost coverage happening now. Whether this offsets Antti's concern, I am not informed enough to know. Moreover, in my experience, flaky tests cause far more pain than they are worth, and as a result it is much more important to get tests that run consistently every time than it is to keep running tests that cause intermittent failures. I believe this is a generally accepted industry / QA principle (i.e., I don't think I'm in a minority here). A corollary of this is that a change that fixes or removes test flakiness is valued highly, even if it causes the underlying problems to stop manifesting themselves. Of course, we have to balance the desire to find bugs against other sources of productivity gain and loss as well. For example, there is no question that running the layout tests in parallel increases test flakiness as well, and yet we think that that is an acceptable tradeoff generally (although perhaps not everyone agrees with this choice). Given all this, it seems like Elliot's suggestion is a near-ideal compromise. You can have your cake and eat it ... we get less flakiness by default, and if you want to test for more flakiness / additional code paths, you can still do so. Perhaps a slight variant of this is that we can agree to make the changes on the Chromium port to clear the cache (much like the Qt and EFL ports already do), and you can continue to not clear the cache on the Apple Mac port until you feel comfortable that you've added additional tests? WDYT? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
