[webkit-dev] Goal of Fixing non-Unified Build Failures (Was Deployment of new EWS Non-Unified builder)

[Ryosuke Niwa via webkit-dev]

> There is no such thing as "not maintaining the non-unified build"; there 
> never has been. We have covered that this is an *inherent* problem in a 
> unified build mechanism and that this would be an issue even if Mac were the 
> only platform.

It appears to me that we’re lacking the clarify in defining what problem(s) 
we’re trying to solve here. As far as I can tell, fixing non-unified builds is 
a potential solution to two problems:
Adding a new cpp file to Sources.txt causes random other file to trigger a 
compiler error because the set of cpp files unified together have changed / 
shuffled across.
Build failure may occur in an exotic port or configuration which unify a 
different set of cpp files than the supported configurations.

(1) might be a problem worth solving but there are multiple solutions to this 
problem. For example, we could stop lexicologically sorting cpp files in 
Sources.txt, and always append new cpp files at the end of each directory 
instead. This will avoid shuffling cpp files that are unified together when 
adding new cpp files. Someone who feels like it could come in & sort files 
occasionally, or maybe we should just forget about this entirely and let the 
chaos ensue. Either solution seems less costly than forcing every contributor 
to maintain non-unified builds functional.

While I’m sympathetic to the view that (2) is a problem or that it would be 
nice for the WebKit project to address the concerns of such exotic 
configurations or ports, I just don’t see how this differs from all other build 
failures that occur from WebKit code changes such as refactoring of platform 
abstraction. I grant that build failures which are effect of cpp files getting 
unified differently across different ports and configurations might be tricky 
to diagnose.

Even if we were to agree that (2) is a problem worth addressing, again, there 
are multiple solutions to this problem. We could, for example, stabilize cpp 
files that are unified across configurations / ports so that if a set of cpp 
files get unified under one configuration, they will always get unified in all 
other configurations. In practice, this might mean moving every platform 
specific file into a sub directory and unifying them separately for example.

- R. Niwa

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Deployment of new EWS Non-Unified builder

[Alexey Proskuryakov via webkit-dev]

Ross, I didn't mean any disrespect.

I absolutely agree that this issue is not about the project supporting multiple 
platforms.

What I disagree with is the statement that omitting includes necessary for 
non-unified build is a mistake, or that it needs to be made visible. Fixing all 
of these is very time consuming, whether it's done pre- or post-commit. In my 
mind, it is a logical conclusion that this shouldn't be done at all, neither 
pre- nor post-commit. The downside is that patches are more likely to break the 
build, but overall it's less work, because only a small percentage of "missing" 
includes will ever cause problems. So, ignoring non-unified build saves work 
over time, and saves the 6 days per month that its maintenance has been costing.

Perhaps that's incorrect, and the cost situation is the opposite? So that 
ignoring non-unified builds is costlier overall? It is a real cost, in 
particular because it sometimes requires fixing issues in unfamiliar code, but 
that cost hasn't been quantified AFAICT.

In other words, the choices are:

With non-unified EWS:
- many patches get rejected for breaking it;
- it's easy for the patch author to add the includes.

Without non-unified EWS, or anyone fixing non-unified build manually:
- a smaller number of patches gets rejected for breaking existing EWS builders;
- it's sometimes harder to fix, because the errors could be in unfamiliar code 
(although how hard can it be to add an include, on average).

Without non-unified EWS, but someone fixes non-unified build manually:
- an even smaller number of patches gets rejected due to missing includes;
- it's a huge investment on the part of those who keep fixing the non-unified 
build.

- Alexey

7 сент. 2022 г., в 6:20 PM, Kirsling, Ross via webkit-dev 
mailto:webkit-dev@lists.webkit.org> > написал(а):

This conversation has had some diverging threads (which is what makes mailing 
list-based communication so difficult), but I'm disappointed that this should 
mean that the crux of the conversation was lost.

There is no such thing as "not maintaining the non-unified build"; there never 
has been. We have covered that this is an *inherent* problem in a unified build 
mechanism and that this would be an issue even if Mac were the only platform. 
The *singular* question at play is whether contributors have visibility into 
the repercussions of their mistakenly omitted includes. The proposal is to have 
a bot to provide this visibility, and Igalia already has one ready to offer. No 
single platform can provide complete visibility because no single platform 
compiles everything, but incomplete visibility would still save a huge amount 
of labor regularly performed by the community.

The fact that this labor can not only be overlooked but outright *defended* is, 
quite frankly, appalling—it is the most disrespectful behavior I've witnessed 
in my time contributing to this community.

Ross

Get Outlook for Android  

From: Alexey Proskuryakov via webkit-dev mailto:webkit-dev@lists.webkit.org> >
Sent: Wednesday, September 7, 2022 5:05:40 PM
To: Michael Catanzaro mailto:mcatanz...@gnome.org> >
Cc: webkit-dev@lists.webkit.org  
mailto:webkit-dev@lists.webkit.org> >
Subject: Re: [webkit-dev] Deployment of new EWS Non-Unified builder
 
I can't speak for everyone, but the reason why I haven't been responding was 
that the discussion went in circles, and didn't address the concerns raised.

There is new evidence showing that maintaining the non-unified build is very 
hard. We knew that from the start, which is why the plan was to not maintain 
it. Seems like this newly posted evidence only reinforces the decision.

- Alexey

> 7 сент. 2022 г., в 10:41 AM, Michael Catanzaro via webkit-dev 
> mailto:webkit-dev@lists.webkit.org> > 
> написал(а):
> 
> 
> At this point I would just go ahead and create the EWS bot. Even if it's not 
> going to be a default build configuration, we're still wasting a bunch of 
> time and effort to keep it working, and the EWS would help fix that.
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org  
> https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/webkit-dev__;!!JmoZiZGBv3RvKRSx!_-YAPcPo_76ayjWZ1XeMQ8GeKGm4GosQLa_S2VyNAFELCChziGVCya-6PVjS2uwD27UxfA_DnSX4FpDUJ7rwqxwrjek$
>  
> 
>   


___
webkit-dev mailing list
webkit-dev@lists.webkit.org  

Re: [webkit-dev] Deployment of new EWS Non-Unified builder

[Kirsling, Ross via webkit-dev]

This conversation has had some diverging threads (which is what makes mailing 
list-based communication so difficult), but I'm disappointed that this should 
mean that the crux of the conversation was lost.

There is no such thing as "not maintaining the non-unified build"; there never 
has been. We have covered that this is an *inherent* problem in a unified build 
mechanism and that this would be an issue even if Mac were the only platform. 
The *singular* question at play is whether contributors have visibility into 
the repercussions of their mistakenly omitted includes. The proposal is to have 
a bot to provide this visibility, and Igalia already has one ready to offer. No 
single platform can provide complete visibility because no single platform 
compiles everything, but incomplete visibility would still save a huge amount 
of labor regularly performed by the community.

The fact that this labor can not only be overlooked but outright *defended* is, 
quite frankly, appalling—it is the most disrespectful behavior I've witnessed 
in my time contributing to this community.

Ross

Get Outlook for Android

From: Alexey Proskuryakov via webkit-dev 
Sent: Wednesday, September 7, 2022 5:05:40 PM
To: Michael Catanzaro 
Cc: webkit-dev@lists.webkit.org 
Subject: Re: [webkit-dev] Deployment of new EWS Non-Unified builder


I can't speak for everyone, but the reason why I haven't been responding was 
that the discussion went in circles, and didn't address the concerns raised.

There is new evidence showing that maintaining the non-unified build is very 
hard. We knew that from the start, which is why the plan was to not maintain 
it. Seems like this newly posted evidence only reinforces the decision.

- Alexey

> 7 сент. 2022 г., в 10:41 AM, Michael Catanzaro via webkit-dev 
>  написал(а):
>
>
> At this point I would just go ahead and create the EWS bot. Even if it's not 
> going to be a default build configuration, we're still wasting a bunch of 
> time and effort to keep it working, and the EWS would help fix that.
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/webkit-dev__;!!JmoZiZGBv3RvKRSx!_-YAPcPo_76ayjWZ1XeMQ8GeKGm4GosQLa_S2VyNAFELCChziGVCya-6PVjS2uwD27UxfA_DnSX4FpDUJ7rwqxwrjek$


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/webkit-dev__;!!JmoZiZGBv3RvKRSx!_-YAPcPo_76ayjWZ1XeMQ8GeKGm4GosQLa_S2VyNAFELCChziGVCya-6PVjS2uwD27UxfA_DnSX4FpDUJ7rwqxwrjek$
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Deployment of new EWS Non-Unified builder

[Alexey Proskuryakov via webkit-dev]

I can't speak for everyone, but the reason why I haven't been responding was 
that the discussion went in circles, and didn't address the concerns raised.

There is new evidence showing that maintaining the non-unified build is very 
hard. We knew that from the start, which is why the plan was to not maintain 
it. Seems like this newly posted evidence only reinforces the decision.

- Alexey

> 7 сент. 2022 г., в 10:41 AM, Michael Catanzaro via webkit-dev 
>  написал(а):
> 
> 
> At this point I would just go ahead and create the EWS bot. Even if it's not 
> going to be a default build configuration, we're still wasting a bunch of 
> time and effort to keep it working, and the EWS would help fix that.
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Deployment of new EWS Non-Unified builder

[Michael Catanzaro via webkit-dev]

At this point I would just go ahead and create the EWS bot. Even if 
it's not going to be a default build configuration, we're still wasting 
a bunch of time and effort to keep it working, and the EWS would help 
fix that.



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Deployment of new EWS Non-Unified builder

[Michael Catanzaro via webkit-dev]

On Sat, May 21 2022 at 09:43:06 AM -0500, Michael Catanzaro 
 wrote:
I would go even further and consider enabling unified builds only in 
DEVELOPER_MODE (for CMake ports). For non-developer builds, 
compilation time is much less important than limiting RAM usage to 
reasonable levels. Using ninja's default parallelization level, I 
recently started hitting OOM failures even on a machine with 64 GB 
RAM! We have many people complaining that they cannot build on more 
normal machines with 16 GB RAM. If we have an EWS to ensure the 
non-unified build actually works, then it should be safe enough to 
make it the normal supported path for non-developers, rather than 
just a "best effort, let's hope it works today" thing.


I withdraw this proposal.

I thought that non-unified builds would significantly reduce peak RAM 
usage, but I was wrong. In fact, non-unified builds seem to require 
substantially more RAM at link time, perhaps because there are more 
object files to link together. On a desktop with 64 GB of RAM, I'm not 
able to link a non-unified build successfully without running out of 
RAM, but using a unified build it works fine. So my proposal was just 
totally off base.


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev