Re: [webkit-dev] PSA: Changes in the recommended GTK/WPE developer workflow
Hi,
A few updates about this topic. Read below!
On Tue, 2020-03-17 at 17:31 +, Philippe Normand wrote:
> Hi,
>
> Until now the most recommended way to work on the GTK and WPE ports
> was
> to use a custom JHBuild[0] moduleset ([1][2]) for build/runtime
> dependencies. This setup was deployed almost 10 years ago [3] and
> while
> it is an opt-in approach, it is highly recommended, especially if you
> want to run layout tests locally.
>
> If you ever executed Tools/Scripts/update-webkitgtk-libs or
> Tools/Scripts/update-webkitwpe-libs it means you currently have a
> DependenciesGTK and/or DependenciesWPE folder in WebKitBuild/ and
> that
> your workflow is JHBuild-based.
>
> JHBuild has various shortcomings, even though it served us well all
> these years:
>
> - if the moduleset is modified, the bots remove the whole
> Dependencies{GTK,WPE} folders and rebuild everything from scratch
> - no cross-compilation support
> - no clear separation between the host system and the JHBuild-based
> sysroot
> - poor system requirements management (See the install-dependencies
> scripts in Tools/{wpe,gtk})
> - time lost rebuilding the dependencies (can take up to an hour on my
> build machine)
>
> So in order to improve our lives a bit, we decided to try a new
> workflow, this time based on Flatpak[4]. Instead of having every
> developer locally build the dependencies, the built sysroot will be
> packaged in a Flatpak SDK/Runtime and downloaded to the developer
> machine. Currently the SDK is built for X86_64 but additional
> architectures can be supported (ARMv7, Aarch64 at least).
>
> The advantages of this new approach:
>
> - cross-compilation can easily be achieved
> - clear separation between the host system and the SDK sysroot
> - the SDK runs in a sandbox
> - unified toolchain for WebKit build (currently both GCC 9.2.0 and
> clang 8.0.1)
> - no time lost rebuilding dependencies :)
> - consistent layout tests coverage across different hosts
> - separate build directories for each port (example:
> WebKitBuild/GTK/Release
> mounted as /app/WebKit/WebKitBuild/Release in the SDK sandbox)
>
> As a bonus, this setup allows for:
>
> - integration with sccache, for faster WebKit builds
> - builtin IceCC support without additional manual steps (you still
> need
> a scheduler running on the host system though)
>
> The only disadvantage of this new approach is that hacking on
> dependencies is currently not trivial to accomplish. We plan to
> enable
> this most likely via the Flapjack[5] tool.
>
Flapjack was discarded, for now, because Buildstream provides a good
workflow already.
> Once the patch from bug#205658 [6] lands, this workflow will be
> available. How do I use it? Easy:
>
> 1. Make sure your flatpak version is recent enough, 1.4.4 is the
> minimum version we support. Backports for Debian stable are available
> and for Ubuntu LTS, a PPA is available. See the Flathub instructions
> (though you don't need to manually add the flathub remote) [7].
>
> 2. Run update-webkitgtk-libs or update-webkitwpe-libs as usual. This
> will download and install the SDK in WebKitBuild/UserFlatpak/
>
> 3. Run build-webkit as usual
>
> The SDK will be used when running the tests (API, layout, webkitpy,
> etc) and the MiniBrowser.
>
> For the time being we keep the JHBuild workflow in the SVN, although
> if
> everything goes well, it will be removed in the coming months. Hence
> we
> encourage all GTK/WPE developers to try this new workflow!
>
> If you still want to keep using JHBuild, make sure to set the
> WEBKIT_JHBUILD=1 environment variable in your shell. But keep in mind
> that we intend to remove support for JHBuild if Flatpak works as
> expected (so please try to test it, and report any issue with it).
> The
> GTK/WPE buildbots and EWS are currently running with JHBuild and will
> be switched one-by-one to Flatpak soon.
>
The bots now use the SDK, thanks to Diego Pino, Lauro Mauro and Carlos
López. Many thanks to them :)
> The SDK sources are currently hosted on Github[8], but we might move
> them to WebKit's SVN.
>
This is done now, as of:
https://trac.webkit.org/r261274
Philippe
> Philippe
>
> [0] https://developer.gnome.org/jhbuild/stable/
> [1] https://trac.webkit.org/browser/trunk/Tools/gtk/jhbuild.modules
> [2] https://trac.webkit.org/browser/trunk/Tools/wpe/jhbuild.modules
> [3] https://bugs.webkit.org/show_bug.cgi?id=73425
> [4] https://flatpak.org/
> [5] https://github.com/endlessm/flapjack
> [6] https://bugs.webkit.org/show_bug.cgi?id=205658
> [7] https://flatpak.org/setup/
> [8] https://github.com/igalia/webkit-flatpak-sdk
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: Changes in the recommended GTK/WPE developer workflow
On Tue, 2020-03-17 at 17:31 +, Philippe Normand wrote: > The only disadvantage of this new approach is that hacking on > dependencies is currently not trivial to accomplish. We plan to > enable > this most likely via the Flapjack[5] tool. > I forgot to mention (quoting the ChangeLog): As an additional commodity, the new environment supports the GStreamer gst-build-based workflow. Just set the GST_BUILD_PATH environment variable to your gst-build path. This feature was contributed by Thibault Saunier. > Once the patch from bug#205658 [6] lands, this workflow will be > available. The patch landed in r258626. Happy hacking! Philippe ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] PSA: Changes in the recommended GTK/WPE developer workflow
Hi,
Until now the most recommended way to work on the GTK and WPE ports was
to use a custom JHBuild[0] moduleset ([1][2]) for build/runtime
dependencies. This setup was deployed almost 10 years ago [3] and while
it is an opt-in approach, it is highly recommended, especially if you
want to run layout tests locally.
If you ever executed Tools/Scripts/update-webkitgtk-libs or
Tools/Scripts/update-webkitwpe-libs it means you currently have a
DependenciesGTK and/or DependenciesWPE folder in WebKitBuild/ and that
your workflow is JHBuild-based.
JHBuild has various shortcomings, even though it served us well all
these years:
- if the moduleset is modified, the bots remove the whole
Dependencies{GTK,WPE} folders and rebuild everything from scratch
- no cross-compilation support
- no clear separation between the host system and the JHBuild-based
sysroot
- poor system requirements management (See the install-dependencies
scripts in Tools/{wpe,gtk})
- time lost rebuilding the dependencies (can take up to an hour on my
build machine)
So in order to improve our lives a bit, we decided to try a new
workflow, this time based on Flatpak[4]. Instead of having every
developer locally build the dependencies, the built sysroot will be
packaged in a Flatpak SDK/Runtime and downloaded to the developer
machine. Currently the SDK is built for X86_64 but additional
architectures can be supported (ARMv7, Aarch64 at least).
The advantages of this new approach:
- cross-compilation can easily be achieved
- clear separation between the host system and the SDK sysroot
- the SDK runs in a sandbox
- unified toolchain for WebKit build (currently both GCC 9.2.0 and
clang 8.0.1)
- no time lost rebuilding dependencies :)
- consistent layout tests coverage across different hosts
- separate build directories for each port (example:
WebKitBuild/GTK/Release
mounted as /app/WebKit/WebKitBuild/Release in the SDK sandbox)
As a bonus, this setup allows for:
- integration with sccache, for faster WebKit builds
- builtin IceCC support without additional manual steps (you still need
a scheduler running on the host system though)
The only disadvantage of this new approach is that hacking on
dependencies is currently not trivial to accomplish. We plan to enable
this most likely via the Flapjack[5] tool.
Once the patch from bug#205658 [6] lands, this workflow will be
available. How do I use it? Easy:
1. Make sure your flatpak version is recent enough, 1.4.4 is the
minimum version we support. Backports for Debian stable are available
and for Ubuntu LTS, a PPA is available. See the Flathub instructions
(though you don't need to manually add the flathub remote) [7].
2. Run update-webkitgtk-libs or update-webkitwpe-libs as usual. This
will download and install the SDK in WebKitBuild/UserFlatpak/
3. Run build-webkit as usual
The SDK will be used when running the tests (API, layout, webkitpy,
etc) and the MiniBrowser.
For the time being we keep the JHBuild workflow in the SVN, although if
everything goes well, it will be removed in the coming months. Hence we
encourage all GTK/WPE developers to try this new workflow!
If you still want to keep using JHBuild, make sure to set the
WEBKIT_JHBUILD=1 environment variable in your shell. But keep in mind
that we intend to remove support for JHBuild if Flatpak works as
expected (so please try to test it, and report any issue with it). The
GTK/WPE buildbots and EWS are currently running with JHBuild and will
be switched one-by-one to Flatpak soon.
The SDK sources are currently hosted on Github[8], but we might move
them to WebKit's SVN.
Philippe
[0] https://developer.gnome.org/jhbuild/stable/
[1] https://trac.webkit.org/browser/trunk/Tools/gtk/jhbuild.modules
[2] https://trac.webkit.org/browser/trunk/Tools/wpe/jhbuild.modules
[3] https://bugs.webkit.org/show_bug.cgi?id=73425
[4] https://flatpak.org/
[5] https://github.com/endlessm/flapjack
[6] https://bugs.webkit.org/show_bug.cgi?id=205658
[7] https://flatpak.org/setup/
[8] https://github.com/igalia/webkit-flatpak-sdk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
