On 11/11/22 17:08, Michael Olbrich wrote:
> Hi,
>
> On Tue, Nov 08, 2022 at 01:10:32PM +0100, Christian Melki wrote:
>> So I did an experiment with bash loadables (builtins).
>> Turns out bash has a bunch of builtin examples.
>>
>> I had ptxdist build me a bash-5.2 for x86_64.
>> With the "loadables" make target.
>>
>> $ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
>> --- ../lib/ptxdist-2022.11.0/rules/bash.make 2022-10-21 16:54:50.000000000
>> +0200
>> +++ rules/bash.make 2022-11-08 12:51:38.609794625 +0100
>> @@ -77,6 +77,16 @@
>> --$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
>> #
>> ----------------------------------------------------------------------------
>> +# Compile
>> +#
>> ----------------------------------------------------------------------------
>> +
>> +$(STATEDIR)/bash.compile:
>> + @$(call targetinfo)
>> + @$(call world/compile, BASH)
>> + @$(call compile, BASH, loadables)
>> + @$(call touch)
>> +
>> +#
>> ----------------------------------------------------------------------------
>>
>> Since mkdir is slightly less than half of the external binary calls I had it
>> replaced.
>> I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
>> and created a shell-loadables directory there.
>>
>> $ cp
>> platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir
>> scripts/lib/shell-loadables/
>>
>> $ ls -1 scripts/lib/shell-loadables/
>> mkdir
>>
>> $ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh
>> scripts/lib/ptxd_make_xpkg_pkg.sh
>> --- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh
>> 2022-10-21 16:54:50.000000000 +0200
>> +++ scripts/lib/ptxd_make_xpkg_pkg.sh 2022-11-08 13:08:18.074550829
>> +0100
>> @@ -682,6 +682,8 @@
>> ptxd_install_file() {
>> local cmd="file"
>> + #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]:
>> ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
>> + enable -f scripts/lib/shell-loadables/mkdir mkdir
>> ptxd_install_file_impl "$@" ||
>> ptxd_install_error "install_file failed!"
>> }
>>
>> without mkdir builtin:
>> finished target timezone.targetinstall.post
>>
>> real 0m17,043s
>> user 0m13,522s
>> sys 0m4,138s
>>
>> with mkdir builtin:
>> finished target timezone.targetinstall.post
>>
>> real 0m12,345s
>> user 0m9,899s
>> sys 0m2,916s
>
> So loadable builtins in bash is interesting, but not something we can rely
> on at this time. But this whole thing got me thinking. Most mkdirs are
> don't actually do anything. They are just in case the directory does not
> exist yet. So we can do that:
>
> if [ ! -d "${dir}" ]; then
> mkdir -p "${dir}"
> fi
>
> That avoid most of the mkdirs and the speedup is considerably. Similar to
> the builtin I think. And I found some more things to improve:
>
> The 'flock' is only needed when building packages in parallel, so we can
> skip it for things like "ptxdist drop foo.compile; ptxdist targetinstall
> foo", and that's where the slow targetinstall hurts the moset.
>
> Do the chmod/chown with the 'install' when possible.
>
> And some stuff to improve the ptxdist startup time.
>
> For "time ptxdist targetinstall timezone" I get:
>
> Before:
> real 0m29.167s
> user 0m19.389s
> sys 0m9.903s
>
> After:
> real 0m8.887s
> user 0m5.954s
> sys 0m2.470s
>
> This stuff should hit master pretty soon.
>
> Thanks for the inspiration :-).
>
> Michael
>
Super!
The end goal was reached either way.
A faster ptxdist! Wee! :D
And thanks for the kind words,
Christian
>> On 11/8/22 11:13 AM, Christian Melki wrote:
>>>
>>>
>>> On 11/4/22 8:12 PM, Alexander Dahl wrote:
>>>> Hello Christian,
>>>>
>>>> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>>>>> Hello,
>>>>>
>>>>> I have question regarding the speedup of daily work.
>>>>>
>>>>> We have frontend and backend of our webgui deployed with separate
>>>>> packages. Only task of these package is to deploy the files with
>>>>>
>>>>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/,
>>>>> /var/www/,no)
>>>>>
>>>>> Compile and install stages are empty. The just call targetinfo and touch
>>>>> to skip the stages.
>>>>>
>>>>> The frontend depends on the backend and the backend obviously depends on
>>>>> our application which is called by the backend.
>>>>> So our web-frontend.in file looks like this:
>>>>> ## SECTION=project_specific
>>>>>
>>>>> config WEB_FRONTEND
>>>>> bool
>>>>> select APP_LAYER
>>>>> select WEB_BACKEND
>>>>> prompt "e-mode Web Frontend"
>>>>> help
>>>>>
>>>>> As expected if i clean and compile APP_LAYER the targetinstallstage of
>>>>> Backend and Frontend are executed again. However this is only a Run-Time
>>>>> only dependency. It is a third-party archive and install_tree takes quite
>>>>> some time even on fast build hosts. Even it if is just a minute it is
>>>>> annoying to spend the time waiting during image creation. Trying to solve
>>>>> that i found "if RUNTIME" für Run-Time only Dependencys in the
>>>>> documentation here:
>>>>>
>>>>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>>>>
>>>>> So my expectation would be that if i change the webfrontend.in file like
>>>>> this:
>>>>>
>>>>> config WEB_FRONTEND
>>>>> bool
>>>>> select APP_LAYER if RUNTIME
>>>>> select WEB_BACKEND if RUNTIME
>>>>> prompt "e-mode Web Frontend"
>>>>> help
>>>>
>>>> That sounds reasonable and I would have done it the same.
>>>>
>>>>> The "if RUNTIME" would make sure that the targetinstall stage is not
>>>>> executed again if i just execute a "ptxdist clean app-layer" followed by
>>>>> a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is
>>>>> clear that all targetinstall stages are executed again, but i would
>>>>> expect that the web-frontend is deployed earlier if no build dependency
>>>>> is given.
>>>>>
>>>>> Am i missing something, oder is the "if RUNTIME" Switch not working
>>>>> properly in my ptxdist-2018.12 version? Or does it have no effect on
>>>>> targetinstall stages?
>>>>
>>>> Not sure how that should behave. However if you want to speed up the
>>>> build: make sure you call ptxdist with -q or --quiet parameter. The
>>>> output on screen takes suprisingly much time, even with modern
>>>> terminals, and especially when doing targetinstall of many many files
>>>> (as usually the case with web frontends. been there, done that.)
>>>>
>>>> Greets
>>>> Alex
>>>>
>>>
>>> I have a slight disagreement here. I don't think the console is slow.
>>> So I did some investigation, mostly since the slowness bugs me too.
>>> I did a:
>>> export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]:
>>> ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
>>> as time measurement and trace in the shellscript in question (mostly
>>> scripts/lib/ptxd_make_xpkg_pkg.sh).
>>> This pretty linear progressive cpu consumption, albeit the big chunks were
>>> due to calling of external binaries.
>>>
>>> Then I did a timezone package clean and targetinstall.
>>> When targetinstalling a zone there was calls to several binaries.
>>>
>>> For one zone I had external calls to (in order):
>>> echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir,
>>> flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir,
>>> install, install, chmod, chmod, chown, echo.
>>>
>>> Some of these could be internal builtins, but the consumed time suggested
>>> otherwise.
>>> Either way. Each install took about 26 ms and I could account the majority
>>> of that time in forking external programs and waiting for them to return.
>>>
>>> So my conclusion is: The whole thing is a bit slow and bash doesn't help.
>>>
>>> Regards,
>>> Christian
>>>
>>>>>
>>>>> Thanks for any feedback.
>>>>> BR,
>>>>> Christian
>>>>
>>>
>>
>>
>
