Title: [281184] trunk/Tools
Revision
281184
Author
[email protected]
Date
2021-08-18 02:13:47 -0700 (Wed, 18 Aug 2021)

Log Message

[Flatpak SDK] Increase the startup performance of webkit-flatpak
https://bugs.webkit.org/show_bug.cgi?id=229185

Reviewed by Philippe Normand.

Improve flatpakutils.py by ensuring that we don't make redundant
calls to flatpak during initialization of the data structures. This
saves 5 seconds on every call to build-webkit and run-webkit-tests.

Before:
    $ time ./Tools/Scripts/webkit-flatpak -c true
    real    0m6,297s
    user    0m0,786s
    sys     0m0,513s

After:
    $ time ./Tools/Scripts/webkit-flatpak -c true
    real    0m1,243s
    user    0m0,375s
    sys     0m0,162s

* flatpak/flatpakutils.py:
(FlatpakPackages.__init__): Separate the update into another
method so that it can be called directly when a package is installed.
Add new packages directly to self.packages.
(FlatpakRepos.__init__): Ensure that we only update our repositories
and package list once we've initialized our list of repositories.
(FlatpakRepos.add): Return True if this method actually changed anything
and accept a parameter determining whether an update is done to
the repository and package list.
(FlatpakRepos.is_package_installed): Added this method which replaces
FlatpakRepo.is_app_installed.
(FlatpakRepo.__init__): Remove the app registry because it is duplicated
by FlatpakPackages.
(FlatpakPackage.is_installed): Now call FlatpakRepos.is_package_installed.
(WebkitFlatpak._reset_repository): Set up both repositories here and
pass then as part of the FlatpakRepos constructor.
(WebkitFlatpak.main): Separate out the check which deletes the Flatpak
repository from the one that checks if the packages are installed. This
makes sure we don't install packages and then immediately delete them.
(WebkitFlatpak.check_installed_packages): rename _get_packages to _get_dependency_packages
in order to make it clear that it doesn't return installed packages.
(WebkitFlatpak._get_dependency_packages): Ditto.
(WebkitFlatpak.install_all): Ditto.
(FlatpakRepo.is_app_installed): Deleted.
(WebkitFlatpak._get_packages): Deleted.

Modified Paths

Diff

Modified: trunk/Tools/ChangeLog (281183 => 281184)


--- trunk/Tools/ChangeLog	2021-08-18 08:26:33 UTC (rev 281183)
+++ trunk/Tools/ChangeLog	2021-08-18 09:13:47 UTC (rev 281184)
@@ -1,3 +1,52 @@
+2021-08-18  Martin Robinson  <[email protected]>
+
+        [Flatpak SDK] Increase the startup performance of webkit-flatpak
+        https://bugs.webkit.org/show_bug.cgi?id=229185
+
+        Reviewed by Philippe Normand.
+
+        Improve flatpakutils.py by ensuring that we don't make redundant
+        calls to flatpak during initialization of the data structures. This
+        saves 5 seconds on every call to build-webkit and run-webkit-tests.
+
+        Before:
+            $ time ./Tools/Scripts/webkit-flatpak -c true
+            real    0m6,297s
+            user    0m0,786s
+            sys     0m0,513s
+
+        After:
+            $ time ./Tools/Scripts/webkit-flatpak -c true
+            real    0m1,243s
+            user    0m0,375s
+            sys     0m0,162s
+
+        * flatpak/flatpakutils.py:
+        (FlatpakPackages.__init__): Separate the update into another
+        method so that it can be called directly when a package is installed.
+        Add new packages directly to self.packages.
+        (FlatpakRepos.__init__): Ensure that we only update our repositories
+        and package list once we've initialized our list of repositories.
+        (FlatpakRepos.add): Return True if this method actually changed anything
+        and accept a parameter determining whether an update is done to
+        the repository and package list.
+        (FlatpakRepos.is_package_installed): Added this method which replaces
+        FlatpakRepo.is_app_installed.
+        (FlatpakRepo.__init__): Remove the app registry because it is duplicated
+        by FlatpakPackages.
+        (FlatpakPackage.is_installed): Now call FlatpakRepos.is_package_installed.
+        (WebkitFlatpak._reset_repository): Set up both repositories here and
+        pass then as part of the FlatpakRepos constructor.
+        (WebkitFlatpak.main): Separate out the check which deletes the Flatpak
+        repository from the one that checks if the packages are installed. This
+        makes sure we don't install packages and then immediately delete them.
+        (WebkitFlatpak.check_installed_packages): rename _get_packages to _get_dependency_packages
+        in order to make it clear that it doesn't return installed packages.
+        (WebkitFlatpak._get_dependency_packages): Ditto.
+        (WebkitFlatpak.install_all): Ditto.
+        (FlatpakRepo.is_app_installed): Deleted.
+        (WebkitFlatpak._get_packages): Deleted.
+
 2021-08-17  Jonathan Bedard  <[email protected]>
 
         [webkitcorepy] Return SourceFileLoader in find_module

Modified: trunk/Tools/flatpak/flatpakutils.py (281183 => 281184)


--- trunk/Tools/flatpak/flatpakutils.py	2021-08-18 08:26:33 UTC (rev 281183)
+++ trunk/Tools/flatpak/flatpakutils.py	2021-08-18 09:13:47 UTC (rev 281184)
@@ -228,10 +228,11 @@
 
     def __init__(self, repos, user=True):
         FlatpakObject.__init__(self, user=user)
-
         self.repos = repos
+        self.update()
 
-        packs = []
+    def update(self):
+        self.packages = []
         out = self.flatpak("list", "--columns=application,arch,branch,origin", "-a", gather_output=True)
         package_defs = [line for line in out.split("\n") if line]
         for package_def in package_defs:
@@ -239,11 +240,8 @@
 
             # If installed from a file, the package is in no repo
             repo = self.repos.repos.get(origin)
+            self.packages.append(FlatpakPackage(name, branch, repo, arch))
 
-            packs.append(FlatpakPackage(name, branch, repo, arch))
-
-        self.packages = packs
-
     def __iter__(self):
         for package in self.packages:
             yield package
@@ -251,11 +249,18 @@
 
 class FlatpakRepos(FlatpakObject):
 
-    def __init__(self, user=True):
+    def __init__(self, repos, user=True):
         FlatpakObject.__init__(self, user=user)
-        self.repos = {}
         self.update()
 
+        updated = False
+        for repo in repos:
+            updated = self.add(repo, update=False) or updated
+
+        # Only fetch the remote and package list again if we updated anything.
+        if updated:
+            self.update()
+
     def update(self):
         self.repos = {}
         out = self.flatpak("remote-list", "--columns=name,title,url", gather_output=True)
@@ -271,37 +276,48 @@
 
         self.packages = FlatpakPackages(self)
 
-    def add(self, repo, override=True):
+    def add(self, repo, update=True, override=True):
         try:
+            repo.repos = self
+
             same_name = None
             for name, tmprepo in self.repos.items():
                 if repo.url == tmprepo.url:
-                    return tmprepo
+                    return False
                 elif repo.name == name:
                     same_name = tmprepo
 
             if same_name:
-                if override:
+                if override and same_name.url != repo.url:
                     self.flatpak("remote-modify", repo.name, "--url="" + repo.url)
                     same_name.url = ""
+                    return True
+                else:
+                    return False
 
-                    return same_name
-                else:
-                    return None
+            args = ["remote-add", repo.name, "--if-not-exists"]
+            if repo.repo_file:
+                args.extend(["--from", repo.repo_file.name])
             else:
-                args = ["remote-add", repo.name, "--if-not-exists"]
-                if repo.repo_file:
-                    args.extend(["--from", repo.repo_file.name])
-                else:
-                    args.extend(["--no-gpg-verify", repo.url])
-                self.flatpak(*args, comment="Adding repo %s" % repo.name)
-
-            repo.repos = self
-            return repo
+                args.extend(["--no-gpg-verify", repo.url])
+            self.flatpak(*args, comment="Adding repo %s" % repo.name)
+            return True
         finally:
-            self.update()
+            if update:
+                self.packages = FlatpakPackages(self)
 
+    def is_package_installed(self, name, branch=None, arch=None):
+        for package in self.packages:
+            if name != package.name:
+                continue
+            if branch and branch != package.branch:
+                continue
+            if arch and arch != package.arch:
+                continue
+            return True
+        return False
 
+
 class FlatpakRepo(FlatpakObject):
 
     def __init__(self, name, desc=None, url=""
@@ -325,21 +341,6 @@
         else:
             assert url
 
-        self._app_registry = {}
-        output = self.flatpak("list", "--columns=application,branch", "-a", gather_output=True)
-        for line in output.splitlines():
-            name, branch = line.split("\t")
-            self._app_registry[name] = branch
-
-    def is_app_installed(self, name, branch=None):
-        if branch:
-            try:
-                return self._app_registry[name] == branch
-            except KeyError:
-                return False
-        else:
-            return name in self._app_registry.keys()
-
     @property
     def repo_file(self):
         if self._repo_file:
@@ -373,16 +374,11 @@
         return "%s/%s/%s" % (self.name, self.arch, self.branch)
 
     def is_installed(self, branch):
+        # Bundles installed from files do not have repositories.
         if not self.repo:
-            # Bundle installed from file
             return True
+        return self.repo.repos.is_package_installed(self.name, branch, self.arch)
 
-        for package in self.repo.repos.packages:
-            if package.name == self.name and package.branch == branch and package.arch == self.arch:
-                return True
-
-        return False
-
     def install(self):
         if not self.repo:
             return False
@@ -391,6 +387,7 @@
         args = ("install", self.repo.name, self.name, "--reinstall", branch)
         comment = "Installing from " + self.repo.name + " " + self.name + " " + self.arch + " " + branch
         self.flatpak(*args, comment=comment)
+        self.repo.repos.packages.update()
 
     def update(self):
         if not self.is_installed(self.branch):
@@ -603,14 +600,17 @@
         return True
 
     def _reset_repository(self):
-        self.repos = FlatpakRepos()
         url = ""
         repo_file = "https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo"
         if self.user_repo:
             url = "" % self.user_repo
             repo_file = None
-        self.sdk_repo = self.repos.add(FlatpakRepo("webkit-sdk", url="" repo_file=repo_file))
 
+        self.sdk_repo = FlatpakRepo("webkit-sdk", url="" repo_file=repo_file)
+        self.flathub_repo = FlatpakRepo("flathub", url=""
+                                        repo_file="https://dl.flathub.org/repo/flathub.flatpakrepo")
+        self.repos = FlatpakRepos([self.sdk_repo, self. flathub_repo])
+
     def setup_builddir(self):
         if os.path.exists(os.path.join(self.flatpak_build_path, "metadata")):
             return True
@@ -986,17 +986,21 @@
             repo.flatpak("update", comment="Updating Flatpak %s environment" % self.build_type)
             regenerate_toolchains = (repo.version("org.webkit.Sdk") != version_before_update) or not self.check_toolchains_generated()
 
-            for package in self._get_packages():
-                if package.name.startswith("org.webkit") and repo.is_app_installed(package.name) \
-                   and not repo.is_app_installed(package.name, branch=self.sdk_branch):
+            # If we have an out-of-date package, simply remove the entire flatpak directory and start over.
+            for package in self._get_dependency_packages():
+                if package.name.startswith("org.webkit") \
+                   and self.repos.is_package_installed(package.name) \
+                   and not self.repos.is_package_installed(package.name, branch=self.sdk_branch):
                     Console.message("New SDK version available, removing local UserFlatpak directory before switching to new version")
                     shutil.rmtree(self.flatpak_build_path)
                     self._reset_repository()
-                    regenerate_toolchains = True
                     break
-                elif not repo.is_app_installed(package.name):
+
+            for package in self._get_dependency_packages():
+                if not self.repos.is_package_installed(package.name):
                     package.install()
                     regenerate_toolchains = True
+
         else:
             regenerate_toolchains = self.regenerate_toolchains
 
@@ -1100,7 +1104,7 @@
             return (relative_filename, sccache_toolchains)
 
     def check_installed_packages(self):
-        for package in self._get_packages():
+        for package in self._get_dependency_packages():
             if package.name.startswith("org.webkit") and not package.is_installed(self.sdk_branch):
                 Console.error_message("Flatpak package %s not installed. Please update your SDK: Tools/Scripts/update-webkit-flatpak", package)
                 return False
@@ -1128,7 +1132,7 @@
 
         return 0
 
-    def _get_packages(self):
+    def _get_dependency_packages(self):
         arch = platform.machine()
         self.runtime = FlatpakPackage("org.webkit.Platform", self.sdk_branch,
                                       self.sdk_repo, arch)
@@ -1138,9 +1142,6 @@
         packages.append(FlatpakPackage('org.webkit.Sdk.Debug', self.sdk_branch,
                                        self.sdk_repo, arch))
 
-        self.flathub_repo = self.repos.add(FlatpakRepo("flathub", url=""
-                                                       repo_file="https://dl.flathub.org/repo/flathub.flatpakrepo"))
-
         fdo_branch = "20.08"
         extensions = ("rust-stable", "llvm11")
         for name in extensions:
@@ -1153,7 +1154,7 @@
         if os.path.exists(os.path.join(self.flatpak_build_path, "runtime", "org.webkit.Sdk")):
             return
         Console.message("Installing %s dependencies in %s", self.build_type, self.flatpak_build_path)
-        for package in self._get_packages():
+        for package in self._get_dependency_packages():
             if not package.is_installed(self.sdk_branch):
                 package.install()
 
_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes

Reply via email to