So I did an experiment with bash loadables (builtins).
Turns out bash has a bunch of builtin examples.
I had ptxdist build me a bash-5.2 for x86_64.
With the "loadables" make target.
$ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
--- ../lib/ptxdist-2022.11.0/rules/bash.make 2022-10-21 16:54:50.000000000
+0200
+++ rules/bash.make 2022-11-08 12:51:38.609794625 +0100
@@ -77,6 +77,16 @@
--$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
# ----------------------------------------------------------------------------
+# Compile
+# ----------------------------------------------------------------------------
+
+$(STATEDIR)/bash.compile:
+ @$(call targetinfo)
+ @$(call world/compile, BASH)
+ @$(call compile, BASH, loadables)
+ @$(call touch)
+
+# ----------------------------------------------------------------------------
Since mkdir is slightly less than half of the external binary calls I had it
replaced.
I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
and created a shell-loadables directory there.
$ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir
scripts/lib/shell-loadables/
$ ls -1 scripts/lib/shell-loadables/
mkdir
$ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh
scripts/lib/ptxd_make_xpkg_pkg.sh
--- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh 2022-10-21
16:54:50.000000000 +0200
+++ scripts/lib/ptxd_make_xpkg_pkg.sh 2022-11-08 13:08:18.074550829 +0100
@@ -682,6 +682,8 @@
ptxd_install_file() {
local cmd="file"
+ #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]:
${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
+ enable -f scripts/lib/shell-loadables/mkdir mkdir
ptxd_install_file_impl "$@" ||
ptxd_install_error "install_file failed!"
}
without mkdir builtin:
finished target timezone.targetinstall.post
real 0m17,043s
user 0m13,522s
sys 0m4,138s
with mkdir builtin:
finished target timezone.targetinstall.post
real 0m12,345s
user 0m9,899s
sys 0m2,916s
On 11/8/22 11:13 AM, Christian Melki wrote:
On 11/4/22 8:12 PM, Alexander Dahl wrote:
Hello Christian,
Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
Hello,
I have question regarding the speedup of daily work.
We have frontend and backend of our webgui deployed with separate packages.
Only task of these package is to deploy the files with
@$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/,
/var/www/,no)
Compile and install stages are empty. The just call targetinfo and touch to
skip the stages.
The frontend depends on the backend and the backend obviously depends on our
application which is called by the backend.
So our web-frontend.in file looks like this:
## SECTION=project_specific
config WEB_FRONTEND
bool
select APP_LAYER
select WEB_BACKEND
prompt "e-mode Web Frontend"
help
As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and
Frontend are executed again. However this is only a Run-Time only dependency. It is a
third-party archive and install_tree takes quite some time even on fast build hosts. Even
it if is just a minute it is annoying to spend the time waiting during image creation.
Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the
documentation here:
https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
So my expectation would be that if i change the webfrontend.in file like this:
config WEB_FRONTEND
bool
select APP_LAYER if RUNTIME
select WEB_BACKEND if RUNTIME
prompt "e-mode Web Frontend"
help
That sounds reasonable and I would have done it the same.
The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just
execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist
clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect
that the web-frontend is deployed earlier if no build dependency is given.
Am i missing something, oder is the "if RUNTIME" Switch not working properly in
my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
Not sure how that should behave. However if you want to speed up the
build: make sure you call ptxdist with -q or --quiet parameter. The
output on screen takes suprisingly much time, even with modern
terminals, and especially when doing targetinstall of many many files
(as usually the case with web frontends. been there, done that.)
Greets
Alex
I have a slight disagreement here. I don't think the console is slow.
So I did some investigation, mostly since the slowness bugs me too.
I did a:
export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]:
${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
as time measurement and trace in the shellscript in question (mostly
scripts/lib/ptxd_make_xpkg_pkg.sh).
This pretty linear progressive cpu consumption, albeit the big chunks were due
to calling of external binaries.
Then I did a timezone package clean and targetinstall.
When targetinstalling a zone there was calls to several binaries.
For one zone I had external calls to (in order):
echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir,
flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install,
install, chmod, chmod, chown, echo.
Some of these could be internal builtins, but the consumed time suggested
otherwise.
Either way. Each install took about 26 ms and I could account the majority of
that time in forking external programs and waiting for them to return.
So my conclusion is: The whole thing is a bit slow and bash doesn't help.
Regards,
Christian
Thanks for any feedback.
BR,
Christian
