[webkit-dev] Baselines ignoring metrics
Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Enable encoding detector on layout(regression) test
On Sep 18, 2012, at 6:25 PM, Kangil Han kangil@samsung.com wrote: However, enabling encoding detector by javaScript manipulation seems not feasible since encoding detector works on reading input stream level. You could probably make an encoding detection test using an iframe with a data URL in it. Have you tried that? -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Enable encoding detector on layout(regression) test
On Sep 19, 2012, at 9:22 AM, Darin Adler da...@apple.com wrote: On Sep 18, 2012, at 6:25 PM, Kangil Han kangil@samsung.com wrote: However, enabling encoding detector by javaScript manipulation seems not feasible since encoding detector works on reading input stream level. You could probably make an encoding detection test using an iframe with a data URL in it. Have you tried that? Or perhaps an iframe loading a resource that’s just a separate file in the testing directory. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
As ossy has suggested in the bug, we should just convert those tests that don't require metrics equivalency to dumpAsText tests. On Sep 19, 2012 2:53 AM, Allan Sandfeld Jensen k...@carewolf.com wrote: Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? I thought the initial motivation for --ignore-metrics was for new ports to at least confirm that the render trees had the right shape even if they didn't have the right metrics. It's unclear to me if this has ever been useful for people, though. On a related note (which I thought I mentioned in some ignore-metrics-related bug, but apparently at least not this one), I've been working on various render-tree-parsing-related features, and I've been thinking that it might be nice to have a smarter diff that could distinguish between fields that actually rendered differently due to different intrinsic metrics vs. fields that rendered differently because of different derived metrics (e.g., two text fields are different sizes because the text renders differently vs. two containing blocks that are also different because they contain the different text fields; if both container and containing block are off by one pixel, on the the contained block's diff is really interesting). But, I don't think (but am not yet 100% sure) that the existing rendertree format contains the information needed to do this accurately. -- Dirk On Wed, Sep 19, 2012 at 1:08 PM, Ryosuke Niwa rn...@webkit.org wrote: As ossy has suggested in the bug, we should just convert those tests that don't require metrics equivalency to dumpAsText tests. On Sep 19, 2012 2:53 AM, Allan Sandfeld Jensen k...@carewolf.com wrote: Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] platform-specific differences in test case inputs
So this thread kinda died. Anyone have any suggestions? Where is JSC's SerializedScriptValue consistency tested? Alec On Tue, Sep 18, 2012 at 1:22 PM, Alec Flett alecfl...@chromium.org wrote: Sorry I totally left out the I expose this through Internals - and adam has explained the rationale On Tue, Sep 18, 2012 at 12:46 PM, Oliver Hunt oli...@apple.com wrote: What exactly are you trying to test here? The internal serialisation format isn't exposed anywhere (nor should it be). By definition newer formats won't be backwards compatible, or are you trying to ensure that newer deserialisers will be able to continue to deserialise old content? exactly. Where does JSC test this? I'm happy to follow whatever pattern is already established. Alec ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? They could be. The tests I had in mind and made a conversion for are however examples of tests from external test suites. We have many tests of this nature, and converting them all the a webkit specific framework would be a lot of work, no one would actually do, and would mean we could not share and update the tests as easily with other members of W3C. This is exactly why I asked for what kind of guidelines we should make for when we should use baselines without metrics. For instance we could decide new tests cases made for WebKit bugs should alway use dumpAsText, and restrict this feature to only test cases from an external source, or only older tests. Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] platform-specific differences in test case inputs
It isn't, it's never been a concern as our serialisation format is simple enough to be hard to break. That said tests would be fine, i'm just not sure how you would do them nicely. --Oliver On Sep 19, 2012, at 1:50 PM, Alec Flett alecfl...@chromium.org wrote: So this thread kinda died. Anyone have any suggestions? Where is JSC's SerializedScriptValue consistency tested? Alec On Tue, Sep 18, 2012 at 1:22 PM, Alec Flett alecfl...@chromium.org wrote: Sorry I totally left out the I expose this through Internals - and adam has explained the rationale On Tue, Sep 18, 2012 at 12:46 PM, Oliver Hunt oli...@apple.com wrote: What exactly are you trying to test here? The internal serialisation format isn't exposed anywhere (nor should it be). By definition newer formats won't be backwards compatible, or are you trying to ensure that newer deserialisers will be able to continue to deserialise old content? exactly. Where does JSC test this? I'm happy to follow whatever pattern is already established. Alec ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen k...@carewolf.com wrote: On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? They could be. The tests I had in mind and made a conversion for are however examples of tests from external test suites. We have many tests of this nature, and converting them all the a webkit specific framework would be a lot of work, no one would actually do, and would mean we could not share and update the tests as easily with other members of W3C. This is exactly why I asked for what kind of guidelines we should make for when we should use baselines without metrics. For instance we could decide new tests cases made for WebKit bugs should alway use dumpAsText, and restrict this feature to only test cases from an external source, or only older tests. I see. My understanding is that the W3C is supposed to be moving in the direction of more reftests everywhere, so hopefully this becomes less of an issue. What to do with the legacy tests that may not get updated is a different issue, of course. My personal hope / theory / claim is that we should make the diff smarter about using the information we have to detect real issues, rather than throwing away information and then potentially missing issues. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] platform-specific differences in test case inputs
My suggestion is to write a unit test in Chromium's webkit_unit_test framework. Adam On Wed, Sep 19, 2012 at 1:50 PM, Alec Flett alecfl...@chromium.org wrote: So this thread kinda died. Anyone have any suggestions? Where is JSC's SerializedScriptValue consistency tested? Alec On Tue, Sep 18, 2012 at 1:22 PM, Alec Flett alecfl...@chromium.org wrote: Sorry I totally left out the I expose this through Internals - and adam has explained the rationale On Tue, Sep 18, 2012 at 12:46 PM, Oliver Hunt oli...@apple.com wrote: What exactly are you trying to test here? The internal serialisation format isn't exposed anywhere (nor should it be). By definition newer formats won't be backwards compatible, or are you trying to ensure that newer deserialisers will be able to continue to deserialise old content? exactly. Where does JSC test this? I'm happy to follow whatever pattern is already established. Alec ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Hash tables and unique string identifiers in JavaScriptCore
On Wed, Sep 19, 2012 at 1:32 AM, Geoffrey Garen gga...@apple.com wrote: Thanks! So is PropertyMapHashTable for properties that have been defined by the user, or is it not that simple? Yes. Apologies. Basically, does the implementation of object property access in the JIT codebase also use strings which have been made unique identifiers in the same way as in the runtime stack? (i.e. can they be assumed equal iff they have the same address). Yes -- typically, though, if the JIT needs to do a hash lookup, it will pass an Identifier to a C++ helper function. Thanks! Do you mind pointing me to where this happens in the code? (passing an Identifer to a C++ helper?) Geoff Stephen ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wed, Sep 19, 2012 at 2:00 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen k...@carewolf.com wrote: On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? They could be. The tests I had in mind and made a conversion for are however examples of tests from external test suites. We have many tests of this nature, and converting them all the a webkit specific framework would be a lot of work, no one would actually do, and would mean we could not share and update the tests as easily with other members of W3C. This is exactly why I asked for what kind of guidelines we should make for when we should use baselines without metrics. For instance we could decide new tests cases made for WebKit bugs should alway use dumpAsText, and restrict this feature to only test cases from an external source, or only older tests. I see. My understanding is that the W3C is supposed to be moving in the direction of more reftests everywhere, so hopefully this becomes less of an issue. What to do with the legacy tests that may not get updated is a different issue, of course. My personal hope / theory / claim is that we should make the diff smarter about using the information we have to detect real issues, rather than throwing away information and then potentially missing issues. Sorry, I realize this answer isn't as helpful as it could be. Personally, I'm not a fan of adding new (external) test suites that aren't text-only tests or reftests, and so I'm a bit negative here. However, I think the value depends on the test suite in question and what it's testing, so it's hard for me to give a blanket statement. If you can point to specific tests you'd like to try this for I can comment in more detail, but I'm not really okay with a blanket approval for using --ignore-metrics whenever we feel like it yet. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wednesday 19 September 2012, Dirk Pranke wrote: If you can point to specific tests you'd like to try this for I can comment in more detail, but I'm not really okay with a blanket approval for using --ignore-metrics whenever we feel like it yet. In the referenced bug I added an example patch converting all the tests under fast/selectors. They are mostly from the css selectors suite plus a few extra. Since they all use text, almost none of the results are shared between platforms, but they could all be with this patch. `Allan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] the new TestExpectations syntax is landing soon
These changes are now starting to land ... as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal keywords in the TestExpectations syntax ... you should use FAIL instead. I will be landing the support for the new syntax as quickly as I can to minimize the transition period. Apologies for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ] Andis documented in full at https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded . ( The [ and ] characters are delimiters, not EBNF optional markers, although those sections are in fact optional :) ). Note that the new syntax means that Skipped files are a syntactic subset of TestExpectations files, and I plan to convert all of the Skipped files to TestExpectations files via copy and paste shortly after the new syntax is landed, and then drop support for Skipped files (I will update ORWT to use the new files and treat any entry as a Skip). The plan for landing these changes is: 1) Add support for parsing the new lines and converting them back into the old format (internally) so that both syntaxes are supported 2) Convert all the existing files over 3) Make sure things aren't broken :) 4) Drop support for the old syntax I plan for this to all happen quickly, in less than a day. This means that if you have patches posted that modify those files they may become stale and need to be updated. Changes from the old syntax: 1. We use URLs (a specific whitelisted set; let me know if you want to add to it) instead of BUGWK12345 etc. 2. We use bug(dpranke) instead of BUGDPRANKE 3. We use CamelCase instead of SHOUTING 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and AUDIO - these failures will be indistinguishable in the new world, meaning that you can't distinguish between text only and both image and text. Since only Chromium runs pixel tests by default, this shouldn't be a big deal. 5. We use ImageOnlyFailure to represent what used to be IMAGE 6. We use [ and ] for delimiters instead of : and = 7. We use # instead of // as a comment 7. WontFix will now imply Skip, i.e., tests marked WontFix will automatically be Skipped 8. WontFix and Skip will not require (or even allow) any other expectations, i.e., you can't say [ WontFix Crash ]. If you want to indicate that the test will crash if you actually run it, use a comment. 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand side to the right. The only keywords on the left restrict which configurations the lines apply to. I will send out follow-up emails as this stuff lands. Please let me know if you have any questions. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Hash tables and unique string identifiers in JavaScriptCore
Thanks! Do you mind pointing me to where this happens in the code? (passing
an Identifer to a C++ helper?)
Here's an example function called by the JIT, from DFGOperations.cpp:
void DFG_OPERATION operationPutByIdNonStrict(ExecState* exec, EncodedJSValue
encodedValue, JSCell* base, Identifier* propertyName)
{
JSGlobalData* globalData = exec-globalData();
NativeCallFrameTracer tracer(globalData, exec);
PutPropertySlot slot(false);
base-methodTable()-put(base, exec, *propertyName,
JSValue::decode(encodedValue), slot);
}
Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] the TestExpectations syntax has changed
Assuming my changes stick, all of the TestExpectations files in the repo have been converted to the new syntax. The old syntax is still supported as well, but please don't use it :). http://trac.webkit.org/wiki/TestExpectations At the moment I'm chasing down some minor issues but I haven't noticed anything massively broken. If you find issues please let me know ASAP (preferably instead of just reverting things ;). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
