Re: [webkit-dev] New EWS Non-Unified builder
On Mon, May 3 2021 at 12:21:57 PM -0700, Sam Weinig via webkit-dev wrote: So, does anyone have any recent measurements of clean build times with and without unified sources enabled? If so, what is the current delta? That would be painful to check, because you would first have to fix all the current non-unified build failures. If somebody wants to spend all day chasing the current round of build failures, I would be interested to see the measurements ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
Hi, I am curious about the current state of our knowledge about the various benefits we are gaining from Unified Source. I think it is important to reevaluate it from time to time to ensure we are still getting the benefits we want (as there are some not so insignificant downsides). I know when it was introduced, it was a win for clean builds but I don’t believe there have been any recent measurements. So, does anyone have any recent measurements of clean build times with and without unified sources enabled? If so, what is the current delta? - Sam > On Apr 28, 2021, at 11:45 PM, dpino via webkit-dev > wrote: > > Hi everyone, > > In Igalia we have been discussing the need of deploying a new builder > which builds WebKit using non-unified sources, and we know that at least > the folks at Sony are also in favor. > > One side effect of Unified Source building is that it hides compilation > errors. The kinds of errors that usually get hidden by unified builds > are missing headers inclusions and missing definitions of functions > declared inline; the latter being tricky to debug because it results in > mysterious linker errors. This is caused by unified builds stashing > several .cpp files together for compilation, so the definitions and > header inclusions done in one “leak” into the others. As for missing > header inclusion errors, a source file might include a header definition > as a co-lateral effect of being stashed together with another file that > indeed includes the missing header. > > These hidden compilation errors may arise later at some point if unified > source files are stashed together in a different manner. > > The current situation is requiring periodical maintenance. You can check > build fixes commits due to unified source compilation with: > >$ git log --pretty=short --grep "Non-unified" > > Here are some examples: >https://bugs.webkit.org/show_bug.cgi?id=222652 >https://bugs.webkit.org/show_bug.cgi?id=222755 >https://bugs.webkit.org/show_bug.cgi?id=221701 > > A new builder which builds WebKit with non-unified Source will highly > help to improve this situation. Compilation errors will be detected as > soon as possible and will save a lot of time not only for the developers > who are currently doing this manual maintenance but for anyone who would > like to build WebKit, and may stumble on compilation errors accidentally > introduced due to unified sources. > > While correct compilation of the codebase can only be guaranteed with > non-unified source builds, we do not propose switching the current EWS > compilation builders to non-unified because it's slower and the EWS > LayoutTests and API test bots use the products built by the EWS builders > — we do not want to delay getting results from those. That's why we are > proposing a new builder: it will run on parallel, resulting in no > slowdown for the other EWS builders, which will keep using unified > builds. > > How this new builder will impact developers? The EWS LayoutTest bots > take at least 1 hour to complete a build. We think that as long as this > new EWS Non-Unified builder is within that time budget, this new EWS > wont' slow down development speed. > > Thoughts? > > Best regards, > > Diego > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On Fri, Apr 30, 2021 at 10:16 AM Alexey Proskuryakov via webkit-dev < webkit-dev@lists.webkit.org> wrote: > These points from my yesterday email remain without responses: > > 1. Cannot have an EWS without corresponding post-commit queue. > Yeah, we should first add a bot. 2. It doesn't appear like we looked into whether there are any ways to > mitigate the problem other that this most costly one. The reason build may break when a new file is added or removed is because all existing files shift. We could mitigate this if we always added a new file at the end of existing unified build files, and always left an empty spot when removing a file. I believe we don't unify files across top-level directories so I could imagine, for example, we can say that we always add a file at the end of Sources.txt per top-level directory. We could then periodically sort files lexicologically with an understanding that this will require a lot of build fixes to be applied at the same time. For ports that don't use unified builds, one solution might be to create wrapper translation units or unified header files. Say we're compiling A.cpp and B.cpp as Unified1.cpp, then we'd basically create the union of header files included in A.cpp and B.cpp and put into Unified1.h. Then this port will build wrapper translation units A-unified.cpp and B-unified.cpp each of which respectively includes A.cpp and B.cpp as well as Unified1.h. - R. Niwa > 30 апр. 2021 г., в 8:43 AM, Darin Adler via webkit-dev < > webkit-dev@lists.webkit.org> написал(а): > > > > OK. I acknowledge my view on this is different from the others > commenting here, perhaps because of different ideas about what is hard and > requires expertise. I won’t stand in the way of changes you want to make. > You know my view now. > > > > — Darin > > ___ > > webkit-dev mailing list > > webkit-dev@lists.webkit.org > > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On Thu, Apr 29, 2021 at 9:18 PM Darin Adler via webkit-dev < webkit-dev@lists.webkit.org> wrote: > > On Apr 29, 2021, at 9:06 PM, Tim Horton via webkit-dev < > webkit-dev@lists.webkit.org> wrote: > > > > it is definitely highly annoying > > It’s possible that where my thinking differs from others on this is that I > don’t think shifting annoying work from one set of commits (the ones adding > a new file) to a different set (the ones unknowingly adding need for a new > include for non-unified builds) is a significant improvement. Adding more > EWS bubbles has a cost. > One key benefit of keeping non-unified builds working is that it would make build brokerage more predictable. Today, when you're adding, removing, or renaming an existing new file, it may compile just fine on the particular port / platform you're using locally yet can still cause a build failure in other ports / platforms since different ports / platforms almost always compile a different set of translation units. If we kept unified builds always working, then the build failure will happen more consistently, making it easier to detect before uploading a patch instead of after observing EWS failures or worse after landing commits because EWS didn't have coverage. Every time someone new starts working on the WebKit project, the topic of mysterious build failures caused by the unified build system comes up as a topic. It's yet another random thing a new developer would have to learn. Anything we can do to reduce the total amount of random knowledge someone has to have to have to be productive in the WebKit project is a positive change in my opinion. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
This seems to be a discussion around “when is doing things lazily valuable.” If the work will always have to be done eventually, then doing it lazily is worse than doing it up front. But of the work has a significant chance of never being needed to be done, then doing it lazily is better. I think one could make a case that, on an infinite timescale, fixing a file’s includes will always have to be done, because it will always, eventually, one day, cause breakage. > On Apr 30, 2021, at 8:44 AM, Darin Adler via webkit-dev > wrote: > > OK. I acknowledge my view on this is different from the others commenting > here, perhaps because of different ideas about what is hard and requires > expertise. I won’t stand in the way of changes you want to make. You know my > view now. > > — Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
(Re-sending from correct address) These points from my yesterday email remain without responses: 1. Cannot have an EWS without corresponding post-commit queue. 2. It doesn't appear like we looked into whether there are any ways to mitigate the problem other that this most costly one. - Alexey > 30 апр. 2021 г., в 8:43 AM, Darin Adler via webkit-dev > написал(а): > > OK. I acknowledge my view on this is different from the others commenting > here, perhaps because of different ideas about what is hard and requires > expertise. I won’t stand in the way of changes you want to make. You know my > view now. > > — Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
OK. I acknowledge my view on this is different from the others commenting here, perhaps because of different ideas about what is hard and requires expertise. I won’t stand in the way of changes you want to make. You know my view now. — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On 30/04/2021 06:17, Darin Adler via webkit-dev wrote: >> On Apr 29, 2021, at 9:06 PM, Tim Horton via webkit-dev >> wrote: >> >> it is definitely highly annoying > > It’s possible that where my thinking differs from others on this is that I > don’t think shifting annoying work from one set of commits (the ones adding a > new file) to a different set (the ones unknowingly adding need for a new > include for non-unified builds) is a significant improvement. I think the issue is not about shifting the work from one sets of commit to another, but about shifting the work from one developer that is knowledgable on the code to other that maybe is not. When you work on a patch, knowing what your includes should be on your patch is easy if the include-breakage is related with the code you are touching because you are understanding the code you are working on. But when totally unrelated code breaks because of your patch, then is not so easy to quickly fix the issue. At least for those of us that don't know everything about the internals of WebKit. Is also not hard, but it would be much easier to fix for the person that introduced that code since they were familiar with it. So meanwhile the person that wrote that code would have identified the include that was missing in X time, I would need X*100 time because I first need to understand what is going on, and then understand the bare minimum of the unrelated code that broke to identify the issue, to finally start hunting down via greps the missing include file. And this assumes you are already familiar with this issue of missing include files happening randomly, because otherwise the first time you discover this you will spend even more time wondering what is happening. > Adding more EWS bubbles has a cost. > Indeed. But if it saves time to developers working on the codebase (which I truly believe it does) then I think it really outweighs the cost IMHO. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
> On Apr 29, 2021, at 9:06 PM, Tim Horton via webkit-dev > wrote: > > it is definitely highly annoying It’s possible that where my thinking differs from others on this is that I don’t think shifting annoying work from one set of commits (the ones adding a new file) to a different set (the ones unknowingly adding need for a new include for non-unified builds) is a significant improvement. Adding more EWS bubbles has a cost. Keep in mind that I am just as passionate as the next person about getting includes “right”. I’m just not sure that this would be a step in the right direction. — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
> On Apr 29, 2021, at 9:02 PM, Darin Adler wrote: > >> On Apr 29, 2021, at 9:01 PM, Tim Horton wrote: >> >> This is a huge problem for people on the Apple platforms using the Xcode >> projects. Nearly every time someone adds a file they have to add random >> headers to unrelated code. > > OK, sorry, I must be out of touch with the rest of you about how difficult > this is. I’ve done that many times, and it was always easy for me. I'm not saying it's hard (depending on your familiarity with the code that breaks, or if it's one of those 'using namespace' situations, or something even more subtle), but it is definitely highly annoying, and judging by the number of times I (and I'm sure others too) have had to explain what's up to other people it seems like "huge" is a only-maybe-slightly-overstated estimate :) Maybe others disagree! > — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
> On Apr 29, 2021, at 7:15 PM, Darin Adler via webkit-dev > wrote: > >> On Apr 29, 2021, at 12:04 PM, Ross Kirsling via webkit-dev >> wrote: >> >> Yeah, I think it's important to clarify that nobody is "using >> non-Unified-Source building for their development", at least to my >> knowledge. Being broken by the shifting sands of unified sources is an >> everybody problem (or at the very least an "everybody that builds via CMake >> problem", which is ultimately an everybody problem). > > How is this a bigger problem in CMake than for people on the Apple platforms > who are using Xcode projects? Can we create greater parity between the two? I don't follow. This is a huge problem for people on the Apple platforms using the Xcode projects. Nearly every time someone adds a file they have to add random headers to unrelated code. > — Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
> On Apr 29, 2021, at 9:01 PM, Tim Horton wrote: > > This is a huge problem for people on the Apple platforms using the Xcode > projects. Nearly every time someone adds a file they have to add random > headers to unrelated code. OK, sorry, I must be out of touch with the rest of you about how difficult this is. I’ve done that many times, and it was always easy for me. — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
> On Apr 29, 2021, at 12:04 PM, Ross Kirsling via webkit-dev > wrote: > > Yeah, I think it's important to clarify that nobody is "using > non-Unified-Source building for their development", at least to my knowledge. > Being broken by the shifting sands of unified sources is an everybody problem > (or at the very least an "everybody that builds via CMake problem", which is > ultimately an everybody problem). How is this a bigger problem in CMake than for people on the Apple platforms who are using Xcode projects? Can we create greater parity between the two? — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On 2021-04-29 20:07, BJ Burg wrote: > Given that there is some disagreement as to whether this is even a > problem at all, perhaps we could try it out for a while. Diego, do you > have the resources to set up an EWS bot in this configuration? > Yes, we have two workers ready for this new EWS Non-Unified builder. The target platform is WPE. There's also a patch with the configuration changes at https://bugs.webkit.org/show_bug.cgi?id=225045 The patch is currently being tested in the UAT. Diego ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On Thu, 29 Apr 2021 19:04:55 + Ross Kirsling via webkit-dev wrote: > Yeah, I think it's important to clarify that nobody is "using > non-Unified-Source building for their development", at least to my > knowledge. Being broken by the shifting sands of unified sources is an > everybody problem (or at the very least an "everybody that builds via CMake > problem", which is ultimately an everybody problem). This is pretty accurate. I do have a build directory configured manually as non-unified so I can get its “compile_commands.json” to point tooling at (mainly clangd, for code completion and linting), but actual builds I do them with the “build-webkit” script. Another situation in which I explicitly opt-in to non-unified builds is around the time we make a release branch for the GTK and WPE ports; get to solve the non-unified build issues and then land the patches in the release branch as well. The goal here is to make sure that what we release can be built non-unified, which means header includes and whatnot are correct (at least for these ports) and we can have the confidence that people doing cross-builds (where the set of files in each unified build compilation unit can shift) will not have problems with the builds. Before I did the above we used to have complaints and distributions sometimes carrying odd patches playing whack-a-mole to get builds done instead of properly fixing things. Cheers, —Adrián > > From: Alex Christensen via webkit-dev > Sent: Thursday, April 29, 2021 11:03 AM > To: Darin Adler > Cc: webkit-dev@lists.webkit.org > Subject: Re: [webkit-dev] New EWS Non-Unified builder > > I don’t see the goal as “keep non-Unified-Source building” but rather > “prevent unrelated build fixes when we add another file later”. Right now > when we add a new file we often have to sprinkle includes, declarations, and > other build fixes in files unrelated to the current change. If we had a bot > building without unification we would be informed at the time we write the > problematic code. > > > On Apr 29, 2021, at 9:55 AM, Darin Adler via webkit-dev > > wrote: > > > > Given the issue is that there are people that are using non-Unified-Source > > building for their development, the best fix is to add post-commit and EWS > > builders for one of those platforms. I do not support the idea of adding an > > additional builder just to “keep non-Unified-Source building” if no > > actively-supported platform is not choosing that build style. > > > > Specifically, if Sony people are most affected by this, I suggest we find a > > way to add Sony PlayStation post-commit and EWS builders. > > > > I am not convinced that we should add some kind of abstract “correct > > compilation” that is a separate builder. > > > > — Darin > > ___ > > webkit-dev mailing list > > webkit-dev@lists.webkit.org > > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev pgpstgQfzBoDJ.pgp Description: PGP signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On Thu, 29 Apr 2021 15:56:31 + Don Olmstead via webkit-dev wrote: > When the Mac CMake build is in a working state I'd request an EWS that is > Non-Unified as well since Mac builds cover more code. This would be a great addition, I agree. While I don't have a Mac around, I can try and spare some cycles helping out with CMake Mac to get the build in shape :) Cheers, —Adrian > -Original Message- > From: Alex Christensen via webkit-dev > Sent: Thursday, April 29, 2021 8:16 AM > To: dpino > Cc: webkit-dev@lists.webkit.org > Subject: Re: [webkit-dev] New EWS Non-Unified builder > > I’d be excited to have this. Those build failures have been bothering me > ever since we started using unified builds. We would have a way to see more > problems in our code that are currently hidden. > > > On Apr 28, 2021, at 11:45 PM, dpino via webkit-dev > > wrote: > > > > Hi everyone, > > > > In Igalia we have been discussing the need of deploying a new builder > > which builds WebKit using non-unified sources, and we know that at > > least the folks at Sony are also in favor. > > > > One side effect of Unified Source building is that it hides > > compilation errors. The kinds of errors that usually get hidden by > > unified builds are missing headers inclusions and missing definitions > > of functions declared inline; the latter being tricky to debug because > > it results in mysterious linker errors. This is caused by unified > > builds stashing several .cpp files together for compilation, so the > > definitions and header inclusions done in one “leak” into the others. > > As for missing header inclusion errors, a source file might include a > > header definition as a co-lateral effect of being stashed together > > with another file that indeed includes the missing header. > > > > These hidden compilation errors may arise later at some point if > > unified source files are stashed together in a different manner. > > > > The current situation is requiring periodical maintenance. You can > > check build fixes commits due to unified source compilation with: > > > >$ git log --pretty=short --grep "Non-unified" > > > > Here are some examples: > > > > https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=222652__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_y9h8CnFZ7s60jhM-e_2Uo6c$ > > > > > > https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=222755__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_y9h8CnFZ7s60jhM-cnURaLI$ > > > > > > https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=22 > > 1701__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_ > > y9h8CnFZ7s60jhM-qQl-arU$ > > > > A new builder which builds WebKit with non-unified Source will highly > > help to improve this situation. Compilation errors will be detected as > > soon as possible and will save a lot of time not only for the > > developers who are currently doing this manual maintenance but for > > anyone who would like to build WebKit, and may stumble on compilation > > errors accidentally introduced due to unified sources. > > > > While correct compilation of the codebase can only be guaranteed with > > non-unified source builds, we do not propose switching the current EWS > > compilation builders to non-unified because it's slower and the EWS > > LayoutTests and API test bots use the products built by the EWS > > builders — we do not want to delay getting results from those. That's > > why we are proposing a new builder: it will run on parallel, resulting > > in no slowdown for the other EWS builders, which will keep using > > unified builds. > > > > How this new builder will impact developers? The EWS LayoutTest bots > > take at least 1 hour to complete a build. We think that as long as > > this new EWS Non-Unified builder is within that time budget, this new > > EWS wont' slow down development speed. > > > > Thoughts? > > > > Best regards, > > > > Diego > > ___ > > webkit-dev mailing list > > webkit-dev@lists.webkit.org > > https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/ > > webkit-dev__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjAC > > a4ZTS_y9h8CnFZ7s60jhM-gSTmLtg$ > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://urldefense.com/v3/__https:/
Re: [webkit-dev] New EWS Non-Unified builder
Yeah, I think it's important to clarify that nobody is "using non-Unified-Source building for their development", at least to my knowledge. Being broken by the shifting sands of unified sources is an everybody problem (or at the very least an "everybody that builds via CMake problem", which is ultimately an everybody problem). Ross From: Alex Christensen via webkit-dev Sent: Thursday, April 29, 2021 11:03 AM To: Darin Adler Cc: webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] New EWS Non-Unified builder I don’t see the goal as “keep non-Unified-Source building” but rather “prevent unrelated build fixes when we add another file later”. Right now when we add a new file we often have to sprinkle includes, declarations, and other build fixes in files unrelated to the current change. If we had a bot building without unification we would be informed at the time we write the problematic code. > On Apr 29, 2021, at 9:55 AM, Darin Adler via webkit-dev > wrote: > > Given the issue is that there are people that are using non-Unified-Source > building for their development, the best fix is to add post-commit and EWS > builders for one of those platforms. I do not support the idea of adding an > additional builder just to “keep non-Unified-Source building” if no > actively-supported platform is not choosing that build style. > > Specifically, if Sony people are most affected by this, I suggest we find a > way to add Sony PlayStation post-commit and EWS builders. > > I am not convinced that we should add some kind of abstract “correct > compilation” that is a separate builder. > > — Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
I'm in favor of a non-Unified EWS builder. It seems it would largely prevent the current situation where I have to add random headers to make my patch build, usually after the patch has been reviewed and revised. This "surprise, someone forgot a header" situation has no effect on our production builds, but it does affect engineering velocity quite a bit when multiple patches are being juggled. Just as it's easier to fix style checker problems before committing, it's easier to address missing includes before committing. And I am not aware of a better way to improve the situation, other than to add some basic pre-commit checks. Given that there is some disagreement as to whether this is even a problem at all, perhaps we could try it out for a while. Diego, do you have the resources to set up an EWS bot in this configuration? BJ Burg (they/them) > On Apr 28, 2021, at 11:45 PM, dpino via webkit-dev > wrote: > > Hi everyone, > > In Igalia we have been discussing the need of deploying a new builder > which builds WebKit using non-unified sources, and we know that at least > the folks at Sony are also in favor. > > One side effect of Unified Source building is that it hides compilation > errors. The kinds of errors that usually get hidden by unified builds > are missing headers inclusions and missing definitions of functions > declared inline; the latter being tricky to debug because it results in > mysterious linker errors. This is caused by unified builds stashing > several .cpp files together for compilation, so the definitions and > header inclusions done in one “leak” into the others. As for missing > header inclusion errors, a source file might include a header definition > as a co-lateral effect of being stashed together with another file that > indeed includes the missing header. > > These hidden compilation errors may arise later at some point if unified > source files are stashed together in a different manner. > > The current situation is requiring periodical maintenance. You can check > build fixes commits due to unified source compilation with: > >$ git log --pretty=short --grep "Non-unified" > > Here are some examples: >https://bugs.webkit.org/show_bug.cgi?id=222652 >https://bugs.webkit.org/show_bug.cgi?id=222755 >https://bugs.webkit.org/show_bug.cgi?id=221701 > > A new builder which builds WebKit with non-unified Source will highly > help to improve this situation. Compilation errors will be detected as > soon as possible and will save a lot of time not only for the developers > who are currently doing this manual maintenance but for anyone who would > like to build WebKit, and may stumble on compilation errors accidentally > introduced due to unified sources. > > While correct compilation of the codebase can only be guaranteed with > non-unified source builds, we do not propose switching the current EWS > compilation builders to non-unified because it's slower and the EWS > LayoutTests and API test bots use the products built by the EWS builders > — we do not want to delay getting results from those. That's why we are > proposing a new builder: it will run on parallel, resulting in no > slowdown for the other EWS builders, which will keep using unified > builds. > > How this new builder will impact developers? The EWS LayoutTest bots > take at least 1 hour to complete a build. We think that as long as this > new EWS Non-Unified builder is within that time budget, this new EWS > wont' slow down development speed. > > Thoughts? > > Best regards, > > Diego > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
I don’t see the goal as “keep non-Unified-Source building” but rather “prevent unrelated build fixes when we add another file later”. Right now when we add a new file we often have to sprinkle includes, declarations, and other build fixes in files unrelated to the current change. If we had a bot building without unification we would be informed at the time we write the problematic code. > On Apr 29, 2021, at 9:55 AM, Darin Adler via webkit-dev > wrote: > > Given the issue is that there are people that are using non-Unified-Source > building for their development, the best fix is to add post-commit and EWS > builders for one of those platforms. I do not support the idea of adding an > additional builder just to “keep non-Unified-Source building” if no > actively-supported platform is not choosing that build style. > > Specifically, if Sony people are most affected by this, I suggest we find a > way to add Sony PlayStation post-commit and EWS builders. > > I am not convinced that we should add some kind of abstract “correct > compilation” that is a separate builder. > > — Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
Given the issue is that there are people that are using non-Unified-Source building for their development, the best fix is to add post-commit and EWS builders for one of those platforms. I do not support the idea of adding an additional builder just to “keep non-Unified-Source building” if no actively-supported platform is not choosing that build style. Specifically, if Sony people are most affected by this, I suggest we find a way to add Sony PlayStation post-commit and EWS builders. I am not convinced that we should add some kind of abstract “correct compilation” that is a separate builder. — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
Hi, Do we have a post-commit non-unified builder? Starting with EWS wouldn't seem like a workable plan to me, as breakages would creep in anyway, and it's really hard to notice and isolate those from EWS results alone. Giving all WebKit developers the responsibility to support more build configurations, particularly ones that are not required for any shipping product, terrifies me. I can see how unpredictable per platform build failures when files are added or removed are a downside of our unified builds. This doesn't seem like an intrinsic part of unified builds though. Can we do anything to make reshuffling the files more controlled, so that build failures happen predictably when one is willing to deal with them? - Alexey > 28 апр. 2021 г., в 11:45 PM, dpino via webkit-dev > написал(а): > > Hi everyone, > > In Igalia we have been discussing the need of deploying a new builder > which builds WebKit using non-unified sources, and we know that at least > the folks at Sony are also in favor. > > One side effect of Unified Source building is that it hides compilation > errors. The kinds of errors that usually get hidden by unified builds > are missing headers inclusions and missing definitions of functions > declared inline; the latter being tricky to debug because it results in > mysterious linker errors. This is caused by unified builds stashing > several .cpp files together for compilation, so the definitions and > header inclusions done in one “leak” into the others. As for missing > header inclusion errors, a source file might include a header definition > as a co-lateral effect of being stashed together with another file that > indeed includes the missing header. > > These hidden compilation errors may arise later at some point if unified > source files are stashed together in a different manner. > > The current situation is requiring periodical maintenance. You can check > build fixes commits due to unified source compilation with: > >$ git log --pretty=short --grep "Non-unified" > > Here are some examples: >https://bugs.webkit.org/show_bug.cgi?id=222652 >https://bugs.webkit.org/show_bug.cgi?id=222755 >https://bugs.webkit.org/show_bug.cgi?id=221701 > > A new builder which builds WebKit with non-unified Source will highly > help to improve this situation. Compilation errors will be detected as > soon as possible and will save a lot of time not only for the developers > who are currently doing this manual maintenance but for anyone who would > like to build WebKit, and may stumble on compilation errors accidentally > introduced due to unified sources. > > While correct compilation of the codebase can only be guaranteed with > non-unified source builds, we do not propose switching the current EWS > compilation builders to non-unified because it's slower and the EWS > LayoutTests and API test bots use the products built by the EWS builders > — we do not want to delay getting results from those. That's why we are > proposing a new builder: it will run on parallel, resulting in no > slowdown for the other EWS builders, which will keep using unified > builds. > > How this new builder will impact developers? The EWS LayoutTest bots > take at least 1 hour to complete a build. We think that as long as this > new EWS Non-Unified builder is within that time budget, this new EWS > wont' slow down development speed. > > Thoughts? > > Best regards, > > Diego > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
When the Mac CMake build is in a working state I'd request an EWS that is Non-Unified as well since Mac builds cover more code. -Original Message- From: Alex Christensen via webkit-dev Sent: Thursday, April 29, 2021 8:16 AM To: dpino Cc: webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] New EWS Non-Unified builder I’d be excited to have this. Those build failures have been bothering me ever since we started using unified builds. We would have a way to see more problems in our code that are currently hidden. > On Apr 28, 2021, at 11:45 PM, dpino via webkit-dev > wrote: > > Hi everyone, > > In Igalia we have been discussing the need of deploying a new builder > which builds WebKit using non-unified sources, and we know that at > least the folks at Sony are also in favor. > > One side effect of Unified Source building is that it hides > compilation errors. The kinds of errors that usually get hidden by > unified builds are missing headers inclusions and missing definitions > of functions declared inline; the latter being tricky to debug because > it results in mysterious linker errors. This is caused by unified > builds stashing several .cpp files together for compilation, so the > definitions and header inclusions done in one “leak” into the others. > As for missing header inclusion errors, a source file might include a > header definition as a co-lateral effect of being stashed together > with another file that indeed includes the missing header. > > These hidden compilation errors may arise later at some point if > unified source files are stashed together in a different manner. > > The current situation is requiring periodical maintenance. You can > check build fixes commits due to unified source compilation with: > >$ git log --pretty=short --grep "Non-unified" > > Here are some examples: > > https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=222652__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_y9h8CnFZ7s60jhM-e_2Uo6c$ > > > https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=222755__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_y9h8CnFZ7s60jhM-cnURaLI$ > > > https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=22 > 1701__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_ > y9h8CnFZ7s60jhM-qQl-arU$ > > A new builder which builds WebKit with non-unified Source will highly > help to improve this situation. Compilation errors will be detected as > soon as possible and will save a lot of time not only for the > developers who are currently doing this manual maintenance but for > anyone who would like to build WebKit, and may stumble on compilation > errors accidentally introduced due to unified sources. > > While correct compilation of the codebase can only be guaranteed with > non-unified source builds, we do not propose switching the current EWS > compilation builders to non-unified because it's slower and the EWS > LayoutTests and API test bots use the products built by the EWS > builders — we do not want to delay getting results from those. That's > why we are proposing a new builder: it will run on parallel, resulting > in no slowdown for the other EWS builders, which will keep using > unified builds. > > How this new builder will impact developers? The EWS LayoutTest bots > take at least 1 hour to complete a build. We think that as long as > this new EWS Non-Unified builder is within that time budget, this new > EWS wont' slow down development speed. > > Thoughts? > > Best regards, > > Diego > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/ > webkit-dev__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjAC > a4ZTS_y9h8CnFZ7s60jhM-gSTmLtg$ ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/webkit-dev__;!!JmoZiZGBv3RvKRSx!t22i0IAUrm1zV3LCteGHgJVUHzashK_F9tqjACa4ZTS_y9h8CnFZ7s60jhM-gSTmLtg$ ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
I’d be excited to have this. Those build failures have been bothering me ever since we started using unified builds. We would have a way to see more problems in our code that are currently hidden. > On Apr 28, 2021, at 11:45 PM, dpino via webkit-dev > wrote: > > Hi everyone, > > In Igalia we have been discussing the need of deploying a new builder > which builds WebKit using non-unified sources, and we know that at least > the folks at Sony are also in favor. > > One side effect of Unified Source building is that it hides compilation > errors. The kinds of errors that usually get hidden by unified builds > are missing headers inclusions and missing definitions of functions > declared inline; the latter being tricky to debug because it results in > mysterious linker errors. This is caused by unified builds stashing > several .cpp files together for compilation, so the definitions and > header inclusions done in one “leak” into the others. As for missing > header inclusion errors, a source file might include a header definition > as a co-lateral effect of being stashed together with another file that > indeed includes the missing header. > > These hidden compilation errors may arise later at some point if unified > source files are stashed together in a different manner. > > The current situation is requiring periodical maintenance. You can check > build fixes commits due to unified source compilation with: > >$ git log --pretty=short --grep "Non-unified" > > Here are some examples: >https://bugs.webkit.org/show_bug.cgi?id=222652 >https://bugs.webkit.org/show_bug.cgi?id=222755 >https://bugs.webkit.org/show_bug.cgi?id=221701 > > A new builder which builds WebKit with non-unified Source will highly > help to improve this situation. Compilation errors will be detected as > soon as possible and will save a lot of time not only for the developers > who are currently doing this manual maintenance but for anyone who would > like to build WebKit, and may stumble on compilation errors accidentally > introduced due to unified sources. > > While correct compilation of the codebase can only be guaranteed with > non-unified source builds, we do not propose switching the current EWS > compilation builders to non-unified because it's slower and the EWS > LayoutTests and API test bots use the products built by the EWS builders > — we do not want to delay getting results from those. That's why we are > proposing a new builder: it will run on parallel, resulting in no > slowdown for the other EWS builders, which will keep using unified > builds. > > How this new builder will impact developers? The EWS LayoutTest bots > take at least 1 hour to complete a build. We think that as long as this > new EWS Non-Unified builder is within that time budget, this new EWS > wont' slow down development speed. > > Thoughts? > > Best regards, > > Diego > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
