Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Ryosuke Niwa]

On Mon, Apr 30, 2012 at 3:29 AM, Sergio Villar Senin wrote:

> En 27/04/12 01:19, Ryosuke Niwa escribiu:
> > See my posts about archive.org-based performance tests. We can use the
> > same infrastructure to make sure WebKit doesn't crash :)
>
> Yeah that's great! I just forgot to replay to that thread :)
>
> > In fact... running top 100 websites as performance tests might just do
> > the trick because we'll catch any crashes (can't catch assertion
> > failures though :( ).
>
> Having performance tests will indeed catch the "obvious" crashes, but I
> was not specially concerned about them because our Layout tests would
> likely catch them first. I was mainly interested in assertions. Why not
> just use debug builds with the same cool infrastructure you're
> developing and call it top-100/assertion/whatever tests?
>

Possible but we'll likely need a separate bot for that. If you're
interested in this idea, please file a bug.

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Sergio Villar Senin]

En 27/04/12 01:19, Ryosuke Niwa escribiu:
> See my posts about archive.org-based performance tests. We can use the
> same infrastructure to make sure WebKit doesn't crash :)

Yeah that's great! I just forgot to replay to that thread :)

> In fact... running top 100 websites as performance tests might just do
> the trick because we'll catch any crashes (can't catch assertion
> failures though :( ).

Having performance tests will indeed catch the "obvious" crashes, but I
was not specially concerned about them because our Layout tests would
likely catch them first. I was mainly interested in assertions. Why not
just use debug builds with the same cool infrastructure you're
developing and call it top-100/assertion/whatever tests?

BR
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Ryosuke Niwa]

See my posts about archive.org-based performance tests. We can use the same
infrastructure to make sure WebKit doesn't crash :)

In fact... running top 100 websites as performance tests might just do the
trick because we'll catch any crashes (can't catch assertion failures
though :( ).

On Sat, Jan 21, 2012 at 2:12 AM, Sergio Villar Senin wrote:

> Hi,
>
> I've been thinking about this for some time now, but only a recent bug
> I'm constantly hitting these days
> (https://bugs.webkit.org/show_bug.cgi?id=76574) triggered this email
> [1]. What I would like to propose here is to have a battery of tests
> that would check that the most visited sites (let's say the top 100 for
> example) are correctly loaded by WebKit. By correctly loaded, I don't
> mean layout or ref tests, they'll just check that the page poad ends
> without any assertion.
>
> I know that trunk is for raw development but having >25k layout tests
> passing is nothing if a so popular site as Wikipedia triggers an
> assertion while being loaded. The obvious answer is "ok so we need more
> tests", we all agree on that, but having like "real-word" permanent
> tests would not harm I guess (and probably help defining more layout
> tests).
>
> I haven't took a detailed look at them, but maybe these main sites
> browsing tests could be part of the perf tests rniwa and others recently
> setup.
>
> What do you think?
>
> BR
>
> [1] note that I am not blaming anyone in particular, we all add bugs,
> just that this one finally flipped the switch :)
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Joe Mason]

People run test builds against live servers by hand all the time.  As long as 
we don't do it too often, I don't see a problem.  Especially as this list would 
be taken from a list of "top 100" web sites, which presumably would each be big 
enough to handle the traffic.

> -Original Message-
> From: webkit-dev-boun...@lists.webkit.org [mailto:webkit-dev-
> boun...@lists.webkit.org] On Behalf Of Pablo Flouret
> Sent: Monday, January 23, 2012 2:45 AM
> To: Sergio Villar Senin
> Cc: webkit-dev@lists.webkit.org
> Subject: Re: [webkit-dev] Top 100 sites browsing tests proprosal
> 
> On Sun, Jan 22, 2012 at 11:37 PM, Sergio Villar Senin
>  wrote:
> > En 22/01/12 20:24, Thiago Marcos P. Santos escribiu:
> >> On Sun, Jan 22, 2012 at 1:06 PM, Pablo Flouret  >
> >> Maybe instead of storing a local snapshot (assuming you need static
> >> content for the tests), you can use a online snapshot of a top 100
> >> website from http://web.archive.org for validation. Not sure if by
> doing
> >> that you can face the same copyright issues, and... IANAL. :)
> >
> > Maybe I didn't explain myself correctly, but my idea was not to store
> > snapshots in the WebKit SVN, it will not be a developer's tool like
> the
> > Layout tests or ref tests but something that will act similar to the
> > EWS. You will not be supposed to run those tests manually, bots will.
> >
> > That's the reason why I was not understanding the copyright concerns.
> I
> > know that they won't work offline, but again their purpouse will not
> be
> > to detect regressions ASAP as the layout tests, but to complement
> them.
> 
> But are you planning to run test builds against live websites? That's
> not very polite on people's servers (i certainly wouldn't like anyone
> stress-testing my own servers for their benefit). And like some other
> people said, tests that rely on network randomness are not very
> reliable indicators of anything.
> 
> --
> pablo flouret
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

-
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Joe Mason]

I don't think this needs to be done by every developer on every checkin.  It 
could be done by the buildbots daily, and whenever a crash is found, somebody 
would need to find the cause and turn it into a real regression test for later. 
 The fact that the web sites change over time isn't important because the cause 
of the failure would get a separate test.

(It would still be a good idea to save the site locally so that the exact 
version can be replayed after a crash, but this would be a temporary copy which 
is thrown away as soon as the test succeeds.)

From: webkit-dev-boun...@lists.webkit.org 
[mailto:webkit-dev-boun...@lists.webkit.org] On Behalf Of Peter Kasting
Sent: Sunday, January 22, 2012 3:01 PM
To: Balazs Kelemen
Cc: webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Top 100 sites browsing tests proprosal

On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen 
mailto:kbal...@webkit.org>> wrote:
As the goal is to test "real world" use case I think it can be even better to 
simply load the sites from network.
I see only two disadvantage of that but neither of them are blocker:
   1. The sites changing over time. However, as this would be a smoke test (and 
not a regtest or a perf test) I don't think it's a big problem
   2. Cannot test offline. Well, I don't think taking part in the development 
of WebKit is possible without being online anyway :)
Do you see other disadvantages?

Loading over a network connection is slower.  Especially if you expect 
developers to run these tests locally, this could be a serious time sink.

Loading real-world sites over a real network connection adds numerous possible 
flakiness/failure sources.  I would never use something like this as part of an 
automated system.

Arguably, it's not kind to site authors to constantly reload their real sites 
as part of an automated testing program.

>From experience using offline copies of popular sites for testing in both 
>Firefox and Chrome, I strongly suggest you go the offline route.

PK

-
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Tony Gentilcore]

On Mon, Jan 23, 2012 at 2:14 AM, Aaron Boodman  wrote:
> On Sun, Jan 22, 2012 at 12:00 PM, Peter Kasting  wrote:
>> On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen  wrote:
>>>
>>> As the goal is to test "real world" use case I think it can be even better
>>> to simply load the sites from network.
>>> I see only two disadvantage of that but neither of them are blocker:
>>>    1. The sites changing over time. However, as this would be a smoke test
>>> (and not a regtest or a perf test) I don't think it's a big problem
>>>    2. Cannot test offline. Well, I don't think taking part in the
>>> development of WebKit is possible without being online anyway :)
>>> Do you see other disadvantages?
>>
>>
>> Loading over a network connection is slower.  Especially if you expect
>> developers to run these tests locally, this could be a serious time sink.
>>
>> Loading real-world sites over a real network connection adds numerous
>> possible flakiness/failure sources.  I would never use something like this
>> as part of an automated system.
>>
>> Arguably, it's not kind to site authors to constantly reload their real
>> sites as part of an automated testing program.
>>
>> From experience using offline copies of popular sites for testing in both
>> Firefox and Chrome, I strongly suggest you go the offline route.
>
> How would you capture and replay websites offline? This is non-trivial.

Very much non-trivial ;)

Depending on which route you end up deciding to take, you might also
want to consider http://code.google.com/p/web-page-replay/ which
scrapes and replays web pages without being tied to a specific
browser. It does some fuzzy request matching and injects a bit of JS
to ensure deterministic playback w.r.t the issues Aaron mentions here.

Without something like that, it is surprisingly difficult to replay
real-world web pages.

>
> In Chromium, we have --record-mode and --playback-mode flags for this
> use case. They put the cache, cookie jar, and some other components
> into a special mode where everything is aggressively captured and
> replayed without going to the net. In the case of Firefox, I believe
> someone at one time implemented a Firefox extension that rewrote pages
> to capture resources and use local references.
>
> Since most websites are not redistributable, it seems like you need
> some code like Chromium's or Firefox's in WebKit in order to make this
> proposal work with offline data.
>
> - a
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Zoltan Herczeg]

> But are you planning to run test builds against live websites? That's
> not very polite on people's servers (i certainly wouldn't like anyone
> stress-testing my own servers for their benefit). And like some other
> people said, tests that rely on network randomness are not very
> reliable indicators of anything.

I think he is planning to have a local copy of those websites, and its
public interface will only be "run tests" like the EWS system. As far as I
know that is possible way to avoid copyright issues.

Thanks for the explanation Sergio,
Zoltan


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Pablo Flouret]

On Sun, Jan 22, 2012 at 11:37 PM, Sergio Villar Senin
 wrote:
> En 22/01/12 20:24, Thiago Marcos P. Santos escribiu:
>> On Sun, Jan 22, 2012 at 1:06 PM, Pablo Flouret 
>> Maybe instead of storing a local snapshot (assuming you need static
>> content for the tests), you can use a online snapshot of a top 100
>> website from http://web.archive.org for validation. Not sure if by doing
>> that you can face the same copyright issues, and... IANAL. :)
>
> Maybe I didn't explain myself correctly, but my idea was not to store
> snapshots in the WebKit SVN, it will not be a developer's tool like the
> Layout tests or ref tests but something that will act similar to the
> EWS. You will not be supposed to run those tests manually, bots will.
>
> That's the reason why I was not understanding the copyright concerns. I
> know that they won't work offline, but again their purpouse will not be
> to detect regressions ASAP as the layout tests, but to complement them.

But are you planning to run test builds against live websites? That's
not very polite on people's servers (i certainly wouldn't like anyone
stress-testing my own servers for their benefit). And like some other
people said, tests that rely on network randomness are not very
reliable indicators of anything.

-- 
pablo flouret
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Sergio Villar Senin]

En 22/01/12 20:24, Thiago Marcos P. Santos escribiu:
> On Sun, Jan 22, 2012 at 1:06 PM, Pablo Flouret  Maybe instead of storing a local snapshot (assuming you need static
> content for the tests), you can use a online snapshot of a top 100
> website from http://web.archive.org for validation. Not sure if by doing
> that you can face the same copyright issues, and... IANAL. :)

Maybe I didn't explain myself correctly, but my idea was not to store
snapshots in the WebKit SVN, it will not be a developer's tool like the
Layout tests or ref tests but something that will act similar to the
EWS. You will not be supposed to run those tests manually, bots will.

That's the reason why I was not understanding the copyright concerns. I
know that they won't work offline, but again their purpouse will not be
to detect regressions ASAP as the layout tests, but to complement them.

BR
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Aaron Boodman]

On Sun, Jan 22, 2012 at 12:00 PM, Peter Kasting  wrote:
> On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen  wrote:
>>
>> As the goal is to test "real world" use case I think it can be even better
>> to simply load the sites from network.
>> I see only two disadvantage of that but neither of them are blocker:
>>    1. The sites changing over time. However, as this would be a smoke test
>> (and not a regtest or a perf test) I don't think it's a big problem
>>    2. Cannot test offline. Well, I don't think taking part in the
>> development of WebKit is possible without being online anyway :)
>> Do you see other disadvantages?
>
>
> Loading over a network connection is slower.  Especially if you expect
> developers to run these tests locally, this could be a serious time sink.
>
> Loading real-world sites over a real network connection adds numerous
> possible flakiness/failure sources.  I would never use something like this
> as part of an automated system.
>
> Arguably, it's not kind to site authors to constantly reload their real
> sites as part of an automated testing program.
>
> From experience using offline copies of popular sites for testing in both
> Firefox and Chrome, I strongly suggest you go the offline route.

How would you capture and replay websites offline? This is non-trivial.

In Chromium, we have --record-mode and --playback-mode flags for this
use case. They put the cache, cookie jar, and some other components
into a special mode where everything is aggressively captured and
replayed without going to the net. In the case of Firefox, I believe
someone at one time implemented a Firefox extension that rewrote pages
to capture resources and use local references.

Since most websites are not redistributable, it seems like you need
some code like Chromium's or Firefox's in WebKit in order to make this
proposal work with offline data.

- a
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Peter Kasting]

On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen  wrote:

> As the goal is to test "real world" use case I think it can be even better
> to simply load the sites from network.
> I see only two disadvantage of that but neither of them are blocker:
>1. The sites changing over time. However, as this would be a smoke test
> (and not a regtest or a perf test) I don't think it's a big problem
>2. Cannot test offline. Well, I don't think taking part in the
> development of WebKit is possible without being online anyway :)
> Do you see other disadvantages?
>

Loading over a network connection is slower.  Especially if you expect
developers to run these tests locally, this could be a serious time sink.

Loading real-world sites over a real network connection adds numerous
possible flakiness/failure sources.  I would never use something like this
as part of an automated system.

Arguably, it's not kind to site authors to constantly reload their real
sites as part of an automated testing program.

>From experience using offline copies of popular sites for testing in both
Firefox and Chrome, I strongly suggest you go the offline route.

PK
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Balazs Kelemen]

On 01/22/2012 08:24 PM, Thiago Marcos P. Santos wrote:
On Sun, Jan 22, 2012 at 1:06 PM, Pablo Flouret > wrote:


On Sun, Jan 22, 2012 at 2:52 AM, Sergio Villar Senin
mailto:svil...@igalia.com>> wrote:
> En 21/01/12 14:31, Zoltan Herczeg escribiu:
>> Just be careful with copyright issues.
>
> Could you elaborate on that?

I believe you'd need explicit permission from each website owner to
keep a (public-facing) copy of the pages you intend to use for
testing. I suppose if any local copies are stashed away somewhere and
never published it should be alright, but IANAL.


Maybe instead of storing a local snapshot (assuming you need static 
content for the tests), you can use a online snapshot of a top 100 
website from http://web.archive.org for validation. Not sure if by 
doing that you can face the same copyright issues, and... IANAL. :)


Cheers,

As the goal is to test "real world" use case I think it can be even 
better to simply load the sites from network.

I see only two disadvantage of that but neither of them are blocker:
   1. The sites changing over time. However, as this would be a smoke 
test (and not a regtest or a perf test) I don't think it's a big problem
   2. Cannot test offline. Well, I don't think taking part in the 
development of WebKit is possible without being online anyway :)

Do you see other disadvantages?

-kbalazs
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Thiago Marcos P. Santos]

On Sun, Jan 22, 2012 at 1:06 PM, Pablo Flouret  wrote:

> On Sun, Jan 22, 2012 at 2:52 AM, Sergio Villar Senin 
> wrote:
> > En 21/01/12 14:31, Zoltan Herczeg escribiu:
> >> Just be careful with copyright issues.
> >
> > Could you elaborate on that?
>
> I believe you'd need explicit permission from each website owner to
> keep a (public-facing) copy of the pages you intend to use for
> testing. I suppose if any local copies are stashed away somewhere and
> never published it should be alright, but IANAL.


Maybe instead of storing a local snapshot (assuming you need static content
for the tests), you can use a online snapshot of a top 100 website from
http://web.archive.org for validation. Not sure if by doing that you can
face the same copyright issues, and... IANAL. :)

Cheers,
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Adam Barth]

On Sun, Jan 22, 2012 at 3:06 AM, Pablo Flouret  wrote:
> On Sun, Jan 22, 2012 at 2:52 AM, Sergio Villar Senin  
> wrote:
>> En 21/01/12 14:31, Zoltan Herczeg escribiu:
>>> Just be careful with copyright issues.
>>
>> Could you elaborate on that?
>
> I believe you'd need explicit permission from each website owner to
> keep a (public-facing) copy of the pages you intend to use for
> testing. I suppose if any local copies are stashed away somewhere and
> never published it should be alright, but IANAL.

To elaborate a bit more: most web sites are copyrighted and not
licensed under an open source license.  Therefore, you won't be able
to check them into the WebKit SVN repo.  We get around this constraint
by using things like the HTML5 specification itself as test data for
the perf tests because the HTML5 spec has a very liberal license.

Obviously, IANAL.  If you're going to do things with copyrighted
works, you'll probably want to consult an actual lawyer to make sure
you don't get yourself into hot water.

Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Pablo Flouret]

On Sun, Jan 22, 2012 at 2:52 AM, Sergio Villar Senin  wrote:
> En 21/01/12 14:31, Zoltan Herczeg escribiu:
>> Just be careful with copyright issues.
>
> Could you elaborate on that?

I believe you'd need explicit permission from each website owner to
keep a (public-facing) copy of the pages you intend to use for
testing. I suppose if any local copies are stashed away somewhere and
never published it should be alright, but IANAL.

--
pablo flouret
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Sergio Villar Senin]

En 21/01/12 14:31, Zoltan Herczeg escribiu:
> Just be careful with copyright issues.

Could you elaborate on that?

BR


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Zoltan Herczeg]

Just be careful with copyright issues.

Regards,
Zoltan

> Go for it.
>
> Adam
>
>
> On Sat, Jan 21, 2012 at 2:12 AM, Sergio Villar Senin 
> wrote:
>> Hi,
>>
>> I've been thinking about this for some time now, but only a recent bug
>> I'm constantly hitting these days
>> (https://bugs.webkit.org/show_bug.cgi?id=76574) triggered this email
>> [1]. What I would like to propose here is to have a battery of tests
>> that would check that the most visited sites (let's say the top 100 for
>> example) are correctly loaded by WebKit. By correctly loaded, I don't
>> mean layout or ref tests, they'll just check that the page poad ends
>> without any assertion.
>>
>> I know that trunk is for raw development but having >25k layout tests
>> passing is nothing if a so popular site as Wikipedia triggers an
>> assertion while being loaded. The obvious answer is "ok so we need more
>> tests", we all agree on that, but having like "real-word" permanent
>> tests would not harm I guess (and probably help defining more layout
>> tests).
>>
>> I haven't took a detailed look at them, but maybe these main sites
>> browsing tests could be part of the perf tests rniwa and others recently
>> setup.
>>
>> What do you think?
>>
>> BR
>>
>> [1] note that I am not blaming anyone in particular, we all add bugs,
>> just that this one finally flipped the switch :)
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Top 100 sites browsing tests proprosal

[Adam Barth]

Go for it.

Adam


On Sat, Jan 21, 2012 at 2:12 AM, Sergio Villar Senin  wrote:
> Hi,
>
> I've been thinking about this for some time now, but only a recent bug
> I'm constantly hitting these days
> (https://bugs.webkit.org/show_bug.cgi?id=76574) triggered this email
> [1]. What I would like to propose here is to have a battery of tests
> that would check that the most visited sites (let's say the top 100 for
> example) are correctly loaded by WebKit. By correctly loaded, I don't
> mean layout or ref tests, they'll just check that the page poad ends
> without any assertion.
>
> I know that trunk is for raw development but having >25k layout tests
> passing is nothing if a so popular site as Wikipedia triggers an
> assertion while being loaded. The obvious answer is "ok so we need more
> tests", we all agree on that, but having like "real-word" permanent
> tests would not harm I guess (and probably help defining more layout tests).
>
> I haven't took a detailed look at them, but maybe these main sites
> browsing tests could be part of the perf tests rniwa and others recently
> setup.
>
> What do you think?
>
> BR
>
> [1] note that I am not blaming anyone in particular, we all add bugs,
> just that this one finally flipped the switch :)
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] Top 100 sites browsing tests proprosal

[Sergio Villar Senin]

Hi,

I've been thinking about this for some time now, but only a recent bug
I'm constantly hitting these days
(https://bugs.webkit.org/show_bug.cgi?id=76574) triggered this email
[1]. What I would like to propose here is to have a battery of tests
that would check that the most visited sites (let's say the top 100 for
example) are correctly loaded by WebKit. By correctly loaded, I don't
mean layout or ref tests, they'll just check that the page poad ends
without any assertion.

I know that trunk is for raw development but having >25k layout tests
passing is nothing if a so popular site as Wikipedia triggers an
assertion while being loaded. The obvious answer is "ok so we need more
tests", we all agree on that, but having like "real-word" permanent
tests would not harm I guess (and probably help defining more layout tests).

I haven't took a detailed look at them, but maybe these main sites
browsing tests could be part of the perf tests rniwa and others recently
setup.

What do you think?

BR

[1] note that I am not blaming anyone in particular, we all add bugs,
just that this one finally flipped the switch :)
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev