This looks reasonable to me. I'll queue the change and do some regression testing. Once that passes, I can put it in my next pull request.
Bruce On Fri, Nov 4, 2022 at 7:32 PM Volodymyr Babchuk <[email protected]> wrote: > > Some file systems, like ZFS, are very slow at appending to existing > files. Due to Copy-On-Write nature, they create a new copy of a file > each time we do ">>" in a shell script. This becomes very noticeable > if shell script does lots and lots of appends, like sanitize_fragment() > function in kconf_check. On my setup, do_kernel_configcheck task takes > literally hours to complete. > > To fix this issue, we can store sanitized_list and fragment_errors.txt > files on tmpfs, which is extremely fast at writing. As most distros > use tmpfs for /tmp, logical step is to use `mktemp` to create > temporary files. > > After completing writing to temporary locations, we can move those two > files back to ${LOGDIR}. > > Also, function 'cleanup' was added to remove temporary files in case > of abnormal exit. > > With this patch, do_kernel_configcheck task completes in ~2 minutes on > my setup, which is a great improvement. > > Signed-off-by: Volodymyr Babchuk <[email protected]> > --- > tools/kconf_check | 20 +++++++++++++++++++- > 1 file changed, 19 insertions(+), 1 deletion(-) > > diff --git a/tools/kconf_check b/tools/kconf_check > index 57d3132..5f25f17 100755 > --- a/tools/kconf_check > +++ b/tools/kconf_check > @@ -195,6 +195,21 @@ function classify_options() { > fi > } > > +# Temporary files to store sanitized list and fragment errors. > +# Create those files in /tmp because /tmp in most cases is stored in RAM > +# This greatly increases performance, as we will append data to those files > +# and some file systems (like ZFS) are very slow at appends. > +FRAGMENT_ERRORS_TMP=`mktemp` > +SANITIZED_LIST_TMP=`mktemp` > + > +# Clean temporary files from /tmp on exit > +function cleanup { > + rm -f $FRAGMENT_ERRORS_TMP > + rm -f $SANITIZED_LIST_TMP > +} > + > +trap cleanup EXIT > + > # step #1. > # run merge_config.sh (without regenerating the .config), and capture its > output. > # This gets us redefined options, and a combined (but unprocessed) config. > @@ -295,7 +310,7 @@ SED_CONFIG_EXP="s/^\(# > \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/p" > rm -f ${LOGDIR}/fragment_errors.txt > declare -A CONFIGMAP > for c in ${configs_abs}; do > - sanitize_fragment ${c} ${LOGDIR}/sanitized_list >> > ${LOGDIR}/fragment_errors.txt > + sanitize_fragment ${c} $SANITIZED_LIST_TMP >> $FRAGMENT_ERRORS_TMP > cfg_list=$(sed -n "$SED_CONFIG_EXP" ${c}) > for option in ${cfg_list}; do > if [ -z "${CONFIGMAP[${option}]}" ]; then > @@ -310,6 +325,9 @@ for c in ${configs_abs}; do > fi > done > > +mv $FRAGMENT_ERRORS_TMP ${LOGDIR}/fragment_errors.txt > +mv $SANITIZED_LIST_TMP ${LOGDIR}/sanitized_list > + > # If we had meta data, we can use the classifications to group the options. > if [ -n "${metadata}" ]; then > rm -f $LOGDIR/avail_hardware.cfg > -- > 2.38.1 -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11858): https://lists.yoctoproject.org/g/linux-yocto/message/11858 Mute This Topic: https://lists.yoctoproject.org/mt/94817916/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
