Re: [webkit-dev] WebKit Documentation
On 9/20/22 05:28, Brandon Stewart via webkit-dev wrote: > Hi WebKit Developers, > > Documentation is an important part of any open source project, especially for > a larger project like WebKit. Being able to ramp up during the onboarding > process, reading up on architectural decisions, and learning how to perform > common procedures are all features the documentation should tackle. WebKit > has a large set of documentation already, but it is scattered around a wide > range of platforms (Trac, GitHub Wiki, markdown files in the code, Git > commits, etc...), and some of the information is out of date. > > A few months ago, I started working on a new documentation solution based on > the DocC documentation framework. This provides an easy way to add and edit > documentation through markdown files. I have already ported a large section > of Trac, all of the GitHub Wiki, and all of the non third party markdown > files in the code over to this platform. I have tested this on macOS and > Linux and have found it works extremely well. (Windows should be able to use > WSL2 at the moment, while a few remaining issues get sorted out). The only > dependency for this project is a recent installation of Swift. > > You can already download the documentation and preview it locally, but we are > looking to publish it online for easy viewing. We were looking to get your > feedback if we would want to publish the documentation on GitHub Pages or > webkit.org. > > The documentation source can be found at > https://github.com/webkit/documentation. > > - Brandon > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev H, It has been more than 5 months since this new documentation repository was announced and as far as I know it hasn't been published yet. Is there any news when this site will be publicly available? Right now in Igalia we need to publish some information publicly and we were discussing where it's the right place to put it. * WebKit Documentation (https://github.com/webkit/documentation.git), IMHO it doesn't make sense because the information needs to be public. * Trac (https://trac.webkit.org/wiki), since WebKit Documentation hasn't been published yet, the fact is that we have been updating Trac pages in the last months. * WebKit GitHub Wiki (https://github.com/WebKit/WebKit/wiki), it seems like the best candidate. It says that new documentation should live there instead of being added to Trac. But on the other hand when WebKit Documentation was announced some people advocated for keep using the GH Wiki but that idea was discarded. It seems like WebKit GitHub Wiki is were new documents should be published until WebKit Documentation site is not available? Please confirm whether this assumption is correct or not. -- Diego___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Deployment of new EWS Non-Unified builder
On 6/8/22 02:40, Elliott Williams via webkit-dev wrote: > >> On Jun 7, 2022, at 10:27, Olmstead, Don via webkit-dev >> wrote: >> >> If we wanted to try any tooling around identifying when an include or >> forward declaration should be used we need a functioning non-unified build. >> We could try IWYU on the codebase, >> https://github.com/include-what-you-use/include-what-you-use , and see what >> happens. > I’ve been thinking about this, too. For Mac builds, we already have the > infrastructure to emit the compile database needed to run IWYU as a > post-build step. We could limit it to the files changed in a commit, which > ought to be very fast, and the verification would be higher-quality than a > non-unified builder since it’ll catch unused includes. In theory, we could > run it during local builds, too. > > But any meaningful integration would require figuring out how to build and > deploy it reliably, which seems annoying since it requires libclang, and > needs to be rebuilt whenever we change compilers. This sounds great. Any solution that helps to solve or miminize the current issues related to unified builds has my support. -- Diego ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Deployment of new EWS Non-Unified builder
On 6/7/22 01:39, Geoffrey Garen via webkit-dev wrote: >> As such, I also think that the non-unified EWS being green should not be a >> blocker to landing a patch. But I think having it there for information will >> help the situation. At minimum, even if every engineer simply ignores the >> non-unified EWS, it also makes it easier for someone trying to fix a trim >> missing include build issue to scan through PRs to look for this EWS failure >> in order to narrow down on which patches (and therefore possible includes) >> to focus on. > > Is this the proposal on the table -- to have an EWS bot, but also not block > patches on it? > > That's surprising to me, and not how EWS bots usually work. If we just want > an optional record of where a particular build configuration started failing, > Isn't that just... a not-EWS bot? The post-commit Non-Unified build bot already covers this use case: - https://build.webkit.org/#/builders/133 This bot is useful to pinpoint which commit introduced a change that broke non-unified builds. Although if it's not kept in good shape it becomes less and less useful over time, as errors accumulate and mask one another. So the proposal is that the EWS Non-Unified bot should work as all the other EWS bots do. -- Diego___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Deployment of new EWS Non-Unified builder
Hi Alexey, On 6/2/22 06:26, Alexey Proskuryakov wrote: > Hi, > > I'm not sure if we have a consensus on whether it is a project goal to keep > non-unified build working at all times. As discussed last year, setting up > post-commit bots is a pre-requisite for having EWS, so this part is resolved. > But proactively maintaining the non-unified build is strictly more work than > fixing it on demand. So the immediate response continues to be that we > shouldn't be doing more work when we can do less. Sorry, I disagree with your point of view. Now that we have the post-commit non-unified build it has become evident to me that actively maintaining non-unified builds is less work than doing it on demand. When the post-commit non-unified is green and a build fails is trivia to spot which commit introduced a regression and what it might be the causes. When maintaining the bot on demand, errors eventually accumulate and becomes less trivia how to fix them. A non-unified EWS bot has the same benefits as the post-commit non-unified EWS, plus is the developer who attempts to land the patch who is in charge of fixing the build. This is more efficient that offloading this work to other developer. Take for instance, this non-unified build fix I recently landed: https://github.com/webkit/webkit/compare/3a5a57d060ee%5E..3a5a57d060ee It took me a long time to figure out I had to include "JSNodeCustom.h" in some files, but possibly for the developer authorizing the patch that introduced the regression, this build error was trivial. > You mention that embedders who build with non-default flags are more likely > to hit issues. Building with non-default flags would be resulting in missing > includes for non-unified builds too, do you have an estimate of how much this > problem grows due to unified builds? I don't have stats to answer how often it happens satisfactorily, but I know that both in WebKitGTK, WPE and other WebKit ports build errors will eventually happen while building with unified sources. IMHO the earlier you can find them the better. We have even made WebKitGTK releases that failed due to unified sources build errors https://stackoverflow.com/questions/64744576/incomplete-webkitgtk-build. > How do we decide if everyone is responsible for the convenience of downstream > embedders? I think there's a wrong perception that non-unified build errors are errors happening in the WebKit ports part of the code. Most of the times they are build errors in WebCore, JavaScriptCore and many other parts of the codebase. If you build Mac with non-unified compilation probably you'll discover a bunch of build errors. I have fixed a bunch of non-unified build errors in the Mac port while building the Playwright's version of WebKit for Mac: - [macOS] Unreviewed, non-unified build fixes https://bugs.webkit.org/show_bug.cgi?id=239889 - [macOS] Unreviewed, non-unified build fixes https://bugs.webkit.org/show_bug.cgi?id=237586 - [macOS] Non-unified build fixes https://bugs.webkit.org/show_bug.cgi?id=236752 These patches landed upstream, they're not errors in the downstream code of Playwright. So it's not a downstream embedder who introduces a build error in the codebase, simply they're more likely to discover it because the introduced downstream changes causes the UnifiedSource files to be arranged in a different way. Who is the responsible for the build error? The developer who introduced the build error obviously, not the person who discovered it. >From my point of view, a project's code should be guaranteed to be always buildable under any supported configuration and it shouldn't rely on collateral effects to build successful (which is the current situation with unified sources). I understand the convenience of unified sources and the advantage they bring. A EWS non-unified bot will guarantee the entire codebase is always buildable while letting us keep using unified sources. From time to time, the EWS non-unified bot will report a build error, and the most suitable person to fix that error will be in charge of it. And we know, thanks to the post-commit non-unified bot we deployed, the non-unified bot won't introduce delays as many times it's even faster than the equivalent unified sources build bot, and of course it's always faster than the EWS LayoutTests bots. > It sounds like none of actively supported ports builds non-unified by > default, which I understand from the fact that a special post-commit queue > had to be set up for that. > > Perhaps part of the argument is that even though proactively maintaining the > non-unified build is more work, this work is somehow easier than fixing > builds on demand. If so, can you elaborate on why this is the case? Exactly, this how I see it. If we agree that a person authorizing a patch is always in a better position to fix a build error than somebody else stranger to that patch, and if we value the time of everybody equally, then we will
[webkit-dev] Deployment of new EWS Non-Unified builder
Hi, Last year we started a thread to discuss the possibility of deploying a new EWS bot that builds WebKit with Non-Unified sources [1]. This thread explains the technical reasons why a non-unified build bot is desirable. If you're aware of the problems introduced by unified sources compilation, skip the next paragraph. Unified sources compilation consists of merging several source files together into one big file to improve building times. Usually the same sources files are grouped together, but compilation flags or downstream changes can create different source file groups. When that happens, you may hit a build error. Since unified sources group files together, missing headers in one file can be satisfied by another file in the same group. What's happening in WebKit development currently is that source code that breaks non-unified compilation frequently lands without even being noticed. The breaks are usually related to missing headers. Embedders that need to support WebKit builds with different flags or maintain downstream changes are more likely to hit these compilation errors. Last year's attempt to deploy an EWS Non-Unified builder ended up in a deployment of a post-commit bot plus two workers running in the UAT. It was actually hard to get the Non-Unified builder working [2], something that we achieved at the beginning of this year (kudos to Adrián and Lauro). Since then we have been maintaining the post-commit bot very closely. There are periods of time where the bot has remained green for a long period of time. But lately maintaining the bot green has become harder and harder, specially due to the big refactorizations that have happened in WebKit source code lately. The bot very rarely stays green longer than 2 or 3 days without close maintenance. We believe that we all should share the responsibility of keeping the non-unified build working, and therefore we want to proceed with the deployment of a EWS Non-Unified builder. Initially, we'll provide two workers on a modern host with 8 cores assigned each. We have found this setup faster for most patches than many of the existing EWS queues, as it only runs a build in non-unified mode, without test or upload steps. If this were not fast enough, we would consider increasing the number of workers in the queue. We set up a page[3] with a rough comparison to the regular (unified) bot and the build times are pretty similar. Diego [1] https://www.mail-archive.com/webkit-dev@lists.webkit.org/msg30077.html [2] https://bugs.webkit.org/show_bug.cgi?id=226088 [3] https://people.igalia.com/lmoura/webkit-bots-dashboard/unified.html ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New EWS Non-Unified builder
On 2021-04-29 20:07, BJ Burg wrote: > Given that there is some disagreement as to whether this is even a > problem at all, perhaps we could try it out for a while. Diego, do you > have the resources to set up an EWS bot in this configuration? > Yes, we have two workers ready for this new EWS Non-Unified builder. The target platform is WPE. There's also a patch with the configuration changes at https://bugs.webkit.org/show_bug.cgi?id=225045 The patch is currently being tested in the UAT. Diego ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] New EWS Non-Unified builder
Hi everyone, In Igalia we have been discussing the need of deploying a new builder which builds WebKit using non-unified sources, and we know that at least the folks at Sony are also in favor. One side effect of Unified Source building is that it hides compilation errors. The kinds of errors that usually get hidden by unified builds are missing headers inclusions and missing definitions of functions declared inline; the latter being tricky to debug because it results in mysterious linker errors. This is caused by unified builds stashing several .cpp files together for compilation, so the definitions and header inclusions done in one “leak” into the others. As for missing header inclusion errors, a source file might include a header definition as a co-lateral effect of being stashed together with another file that indeed includes the missing header. These hidden compilation errors may arise later at some point if unified source files are stashed together in a different manner. The current situation is requiring periodical maintenance. You can check build fixes commits due to unified source compilation with: $ git log --pretty=short --grep "Non-unified" Here are some examples: https://bugs.webkit.org/show_bug.cgi?id=222652 https://bugs.webkit.org/show_bug.cgi?id=222755 https://bugs.webkit.org/show_bug.cgi?id=221701 A new builder which builds WebKit with non-unified Source will highly help to improve this situation. Compilation errors will be detected as soon as possible and will save a lot of time not only for the developers who are currently doing this manual maintenance but for anyone who would like to build WebKit, and may stumble on compilation errors accidentally introduced due to unified sources. While correct compilation of the codebase can only be guaranteed with non-unified source builds, we do not propose switching the current EWS compilation builders to non-unified because it's slower and the EWS LayoutTests and API test bots use the products built by the EWS builders — we do not want to delay getting results from those. That's why we are proposing a new builder: it will run on parallel, resulting in no slowdown for the other EWS builders, which will keep using unified builds. How this new builder will impact developers? The EWS LayoutTest bots take at least 1 hour to complete a build. We think that as long as this new EWS Non-Unified builder is within that time budget, this new EWS wont' slow down development speed. Thoughts? Best regards, Diego ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
