Currently the following drawbacks exist with applying patches with quilt and git: * When a developer is working on a source tree in build-*/* and the contents of the patches directory changes due to an updated patch the changes do not find its way to the working tree, unless the developer performs a 'quilt pop -a && quilt push -a'. If someone else edited the same patch as the developer is working on, the risk exist that the changes of the other developer get lost (this happens if a developer does a 'quilt pop -a' until failure followed by a 'quilt refresh' without looking carefully at the changes) * Changes to source files must always be explicitly added to a patch otherwise the change dangles around and will get lost on a package rebuild, which will (not infrequently) only be discovered by other developers who runs into broken code. * By using the GIT tool (PTXCONF_SETUP_PATCHIN_GIT) to apply patches only the tree is being _generated_ by means of git. There is no useful tool yet to extract the new changes done by a developer out of the git-tree. The current git-alias 'ptx-patches' dumps the entire series of patches applied to the patches directory, but this always results in a patch series that is entirely different compared to the series that was used to generate the git-tree, even if no single line of code has been touched of that package.
To improve these issues the following changes are made by this patch: * When PTXCONF_SETUP_PATCHIN_GIT is set, git is being used as patch apply tool as usual, but a label (patches_current) is created on the last commit that has been applied to indicate the reference point of the contents of the patches directory. An md5sum is stored in the tag that matches all these patches. * When the contents of the patches directory changes, the md5sum of the patches will change. This will be the trigger to rebuild the source tree with the new patch series. All local commits in the git repo of that package will be automatically rebased on top of the new series. This will only happen in case the git-tree has no local uncommitted changes. If that is the case the user will be notified and the build will be stopped. * This regeneration of the build tree is done in a new intermediate stage called 'update'. This intermediate stage has found its place between the 'extract' and the 'prepare' stage. The 'update' stage will be executed on every incremental build, and will touch its stage file _only_ when something has changed or if the 'extract' stage is newer compared to the 'update' stage. This makes sure the 'prepare' stage will run, and so the 'compile' and all other subsequent stages will run when a patch has changed. Note: This stage is transparant in case 'quilt' or 'patch' is being used as patch tool. * On every incremental build all local changes to the source tree in build-*/* that are committed in the local git repo will be exported to a per package 'patches/staged' directory. The developer can move the patches that are stored here (by hand) to the normal 'patches' directory and add them by hand to the series file. This prevents the situation that the developer will loose his local committed changes if by accident a 'make clean' is being executed. Note that exporting the patches to a staged directory instead of the normal patches directory also prevents that the source tree is automatically being rebuild after every commit in the git repo, which can safe a huge amount of time. Every change in the patches directory would result in a regeneration of the build tree as described in the 2nd bullet above. Regeneration of the source tree usually ends up in a complete rebuild of the package which could take quite some time, while an incremental build of the package can be very fast. * Using git to apply patches is rather slow, PTXCONF_SETUP_PATCHIN_GIT_PKG_LIST can be used to specify the packages that need to be extracted by using git. * Adding dependencies can slow down incremental build times by seconds, since on every incremental build the md5sums are calculated of the patch-series. (Experience show for a 6MB patchset about 1 second). If this is annoying or unwanted, the setting PTXCONF_SETUP_PATCHIN_GIT_DEPENDENCIES can be used to disable the dependency check globally. To enable the dependencies only on a single package the PTXCONF_SETUP_PATCHIN_GIT_PKG_LIST can be used to limit the number of packages that use git as patch tool. * An option --no-git is added to the cmd-line to disable the git-usage temporarily. It also disables the dependencies mechanism for that build, even if the source package was extracted with git. * The git-alias ptx-patches still works as usual. * quilt and patch works as usual. * If a source package does not have a .gitignore file on its own, a default file is being generated for that package. Signed-off-by: Remy Bohmer <[email protected]> --- bin/ptxdist | 9 +- config/setup/Kconfig | 42 ++++++ rules/post/function-touch.make | 3 +- rules/post/ptxd_make_world_patchin.make | 32 +++++ scripts/bash_completion | 6 +- scripts/lib/ptxd_lib_dgen.awk | 3 +- scripts/lib/ptxd_make_world_patchin.sh | 230 ++++++++++++++++++++++++++++++- 7 files changed, 317 insertions(+), 8 deletions(-) diff --git a/bin/ptxdist b/bin/ptxdist index a14cfde..f952f8c 100755 --- a/bin/ptxdist +++ b/bin/ptxdist @@ -805,6 +805,7 @@ Build Actions: get <package> get package sources extract <package> extract package + update <package> update patches applied to package prepare <package> run configure stages for package compile <package> compile the sources install <package> install host side components into sysroot/ @@ -862,6 +863,7 @@ Options: after an error. --git use git to apply patches + --no-git NOT use git to apply patches EOF } @@ -1354,6 +1356,11 @@ parse_first() --git) # overwrite default from ptxdistrc export PTXCONF_SETUP_PATCHIN_GIT=y + export PTXCONF_SETUP_PATCHIN_GIT_PKG_LIST="*" + ;; + --no-git) + # overwrite default from ptxdistrc + export PTXCONF_SETUP_PATCHIN_GIT=n ;; --) while [ ${#} -ne 0 ]; do @@ -1739,7 +1746,7 @@ parse_second() ######## standard target, directly into make - extract|prepare|compile|install|targetinstall|tags) + extract|update|prepare|compile|install|targetinstall|tags) local cmd_post local pkg="${1}" diff --git a/config/setup/Kconfig b/config/setup/Kconfig index f5ed9c9..54e8e87 100644 --- a/config/setup/Kconfig +++ b/config/setup/Kconfig @@ -260,6 +260,48 @@ config SETUP_PATCHIN_GIT _NOTE_: this is an experimental feature +config SETUP_PATCHIN_GIT_DEPENDENCIES + bool "enable build time dependencies to applied patches" + depends on SETUP_PATCHIN_GIT + help + Activating this option, ptxdist will maintain dependencies + to the patches that were applied while extracting the + package and creating the git tree. + + When some patch has changed in the patches directory, ptxdist + will regenerate the entire git tree based on the new contents + of the patches directory. When this is completed, all local + changes to the package will be rebased on top of the new git + tree. This will only be done if there are no uncommitted changes + inside the tree, otherwise the build will stop and report a + 'dirty-tree' message. + + Local changes that were committed in the git-repo of the package + will be automatically exported to a 'staged' directory inside + the patches directory, they need to be moved manually to the + patches directory and manually be added to the series file. + This to prevent unwanted rebasing of the git-tree, which usually + results in a almost complete rebuild of the package (due to changing + timstamps of many files inside the package) + + Dependencies to patches are calculated based on md5-sums, not on + timestamps of the patch files to prevent unneccessary rebasing of + the git-tree and thus speed up (incremental) builds. + + _NOTE_: this is an experimental feature + +config SETUP_PATCHIN_GIT_PKG_LIST + prompt "packages that need to use 'git' to apply patches" + string + default "*" + depends on SETUP_PATCHIN_GIT + help + Specify the list of packages that need to use "git" instead + of quilt to apply patches. + (Items must be seperated with spaces, e.g.: 'kernel u-boot busybox') + Use '*' to use git for ALL packages + + _NOTE_: this is an experimental feature config SETUP_HOST_CC prompt "host CC" diff --git a/rules/post/function-touch.make b/rules/post/function-touch.make index 0a7919e..d5704d8 100644 --- a/rules/post/function-touch.make +++ b/rules/post/function-touch.make @@ -16,9 +16,10 @@ ifdef PTXDIST_QUIET _touch_opt_output := echo "finished: $(PTX_COLOR_GREEN)$${target}$(PTX_COLOR_OFF)" >&$(PTXDIST_FD_STDOUT); endif +# $(1): options to be passed to the touch command touch = \ target="$(strip $(@))"; \ - touch "$${target}"; \ + touch "$${target}" $(1); \ target="$${target\#\#*/}"; \ $(_touch_opt_output) \ echo "finished target $${target}" diff --git a/rules/post/ptxd_make_world_patchin.make b/rules/post/ptxd_make_world_patchin.make index 48c72a1..39c9652 100644 --- a/rules/post/ptxd_make_world_patchin.make +++ b/rules/post/ptxd_make_world_patchin.make @@ -18,4 +18,36 @@ patchin = \ pkg_deprecated_patchin_series="$(call ptx/escape,$(3))" \ $(call world/patchin, $(1)) +world/update = \ + pkg_target_update=$@; \ + pkg_target_extract=$${pkg_target_update/%update/extract}; \ + pkg_deprecated_patchin_dir="$(call ptx/escape,$(2))" \ + pkg_deprecated_patchin_series="$(call ptx/escape,$(3))" \ + $(call world/env, $(1)) \ + pkg_patch_series="$(call ptx/escape,$(call remove_quotes, $(PTXCONF_$(strip $(1))_SERIES)))" \ + ptxd_make_world_update + +update = \ + $(call targetinfo) \ + $(call world/update,$(1),$($(1)_DIR)); \ + if [ $$? -eq 0 ]; then \ + $(call touch,-r $@); \ + else \ + echo "package not up-to-date"; \ + $(call touch); \ + fi + +# The update target is dependant on FORCE to force it to be called on every +# incremental 'ptxdist go'. It checks for changed patches, and if some patch has +# changed, it will update the *.update target. This update triggers the rebuild +# of prepare, compile and so on. + +### --- for KLIBC packages only --- +$(STATEDIR)/klibc-%.update: FORCE + @$(call update,$(PTX_MAP_TO_PACKAGE_klibc-$(*))) + +### --- all but KLIBC packages --- +$(STATEDIR)/%.update: FORCE + @$(call update,$(PTX_MAP_TO_PACKAGE_$(*))) + # vim: syntax=make diff --git a/scripts/bash_completion b/scripts/bash_completion index f24672f..e7b77c5 100644 --- a/scripts/bash_completion +++ b/scripts/bash_completion @@ -14,7 +14,7 @@ _ptxdist_completion() cur=( $(_get_cword) ) opts=' --collectionconfig --debug --force --force-download --j-extern --j-intern -k --platformconfig --ptxconfig --quiet --toolchain' - cmds=' allmodconfig allnoconfig allyesconfig autobuild boardsetup clean clone collection compile distclean drop export_src extract get go images install kernelconfig menu menuconfig migrate newpackage oldconfig platform platformconfig prepare print projects ptxdist select setup tags targetinstall test toolchain u_boot_config version' + cmds=' allmodconfig allnoconfig allyesconfig autobuild boardsetup clean clone collection compile distclean drop export_src extract update get go images install kernelconfig menu menuconfig migrate newpackage oldconfig platform platformconfig prepare print projects ptxdist select setup tags targetinstall test toolchain u_boot_config version' # if no commands were given, complete on commands themselves if [ $COMP_CWORD -eq 1 ]; then @@ -40,14 +40,14 @@ _ptxdist_completion() compgen -f /dummy >/dev/null [ $COMP_CWORD -lt 3 ] && _filedir ;; - get|extract|prepare|compile|install|targetinstall|tags) + get|extract|update|prepare|compile|install|targetinstall|tags) COMPREPLY=( $( compgen -W "${opts} $( ptxdist print PTX_PACKAGES_SELECTED )" -- $cur ) ) ;; clean) COMPREPLY=( $( compgen -W "${opts} $( ptxdist print PTX_PACKAGES_SELECTED ) root" -- $cur ) ) ;; drop) - COMPREPLY=( $( compgen -W "${opts} $( pushd $(ptxdist print PTXDIST_PLATFORMDIR)/state >/dev/null; ls +(*.get|*.extract|*.prepare|*.compile|*.install|*.targetinstall); popd >/dev/null )" -- $cur ) ) + COMPREPLY=( $( compgen -W "${opts} $( pushd $(ptxdist print PTXDIST_PLATFORMDIR)/state >/dev/null; ls +(*.get|*.extract|*.update|*.prepare|*.compile|*.install|*.targetinstall); popd >/dev/null )" -- $cur ) ) ;; newpackage) COMPREPLY=( $( compgen -W "${opts} target host cross klibc arc-autoconf-lib src-autoconf-prog src-autoconf-proglib src-cmake-prog src-linux-dirver src-make-prog src-stellaris font simple" -- $cur) ) diff --git a/scripts/lib/ptxd_lib_dgen.awk b/scripts/lib/ptxd_lib_dgen.awk index d446412..3583731 100644 --- a/scripts/lib/ptxd_lib_dgen.awk +++ b/scripts/lib/ptxd_lib_dgen.awk @@ -300,7 +300,8 @@ END { # default deps # print "$(STATEDIR)/" this_pkg ".extract: " "$(STATEDIR)/" this_pkg ".get" > DGEN_DEPS_POST; - print "$(STATEDIR)/" this_pkg ".prepare: " "$(STATEDIR)/" this_pkg ".extract" > DGEN_DEPS_POST; + print "$(STATEDIR)/" this_pkg ".update: " "$(STATEDIR)/" this_pkg ".extract" > DGEN_DEPS_POST; + print "$(STATEDIR)/" this_pkg ".prepare: " "$(STATEDIR)/" this_pkg ".update" > DGEN_DEPS_POST; print "$(STATEDIR)/" this_pkg ".tags: " "$(STATEDIR)/" this_pkg ".prepare" > DGEN_DEPS_POST; print "$(STATEDIR)/" this_pkg ".compile: " "$(STATEDIR)/" this_pkg ".prepare" > DGEN_DEPS_POST; print "$(STATEDIR)/" this_pkg ".install: " "$(STATEDIR)/" this_pkg ".compile" > DGEN_DEPS_POST; diff --git a/scripts/lib/ptxd_make_world_patchin.sh b/scripts/lib/ptxd_make_world_patchin.sh index 57f7a61..a59abb9 100644 --- a/scripts/lib/ptxd_make_world_patchin.sh +++ b/scripts/lib/ptxd_make_world_patchin.sh @@ -28,6 +28,8 @@ ptxd_make_world_patchin_apply_init() # local path="${PTXDIST_PATH_PATCHES//://${pkg_pkg}/generic } \ ${PTXDIST_PATH_PATCHES//://${pkg_pkg} }" + local pkg_patch_tool_git_ok=no + local pkg_label="${pkg_stamp%%.*}" # find patch_dir if ! ptxd_get_path "${path}"; then @@ -59,7 +61,19 @@ ptxd_make_world_patchin_apply_init() fi # decide which tool to use - if [ "${PTXCONF_SETUP_PATCHIN_GIT}" ] && which git > /dev/null 2>&1; then + if [ "${PTXCONF_SETUP_PATCHIN_GIT}" = "y" ]; then + if [ "${PTXCONF_SETUP_PATCHIN_GIT_PKG_LIST}" = "*" ]; then + pkg_patch_tool_git_ok=yes + else + for each_pkg in ${PTXCONF_SETUP_PATCHIN_GIT_PKG_LIST}; do + if [ "${each_pkg}" = "${pkg_label}" ]; then + pkg_patch_tool_git_ok=yes + break + fi + done + fi + fi + if [ "${pkg_patch_tool_git_ok}" = "yes" ] && which git > /dev/null 2>&1; then pkg_patch_tool=git elif which quilt > /dev/null 2>&1; then pkg_patch_tool=quilt @@ -71,6 +85,69 @@ export -f ptxd_make_world_patchin_apply_init # +# Create a gignore file in the git-tree if it does not exist yet +# +ptxd_make_world_patchin_create_gitignore() +{ + [ ! -f ${pkg_patchin_dir}/.gitignore ] || return + cat > ${pkg_patchin_dir}/.gitignore << EOF +# +# Normal rules +# +.* +*.o +*.lo +*.o.* +*.a +*.s +*.so +*.so.dbg +*.i +*.lst +*.elf +*.bin +*.gz +*.bz2 +*.lzma +*.patch +*.gcno + +# +# Top-level generic files +# +tags +TAGS +!.gitignore + +# stgit generated dirs +patches-* + +# quilt's files +patches +series + +# ptxdist files +.ptxdist/ + +# cscope files +cscope.* +ncscope.* + +# gnu global files +GPATH +GRTAGS +GSYMS +GTAGS + +*.orig +*~ +\#*# +EOF +} +export -f ptxd_make_world_patchin_create_gitignore + + +# # initialize git database in $pkg_patchin_dir and do initial commit # ptxd_make_world_patchin_apply_git_init() @@ -81,6 +158,7 @@ ptxd_make_world_patchin_apply_git_init() # is already git repo? if [ "${git_dir}" != ".git" ]; then echo "patchin: git: initializing repository" + ptxd_make_world_patchin_create_gitignore git init -q && git add -f . && git commit -q -m "initial commit" --author="ptxdist-${PTXDIST_VERSION_FULL} <[email protected]>" && @@ -101,6 +179,7 @@ export -f ptxd_make_world_patchin_apply_git_init # ptxd_make_world_patchin_apply_git_compat() { + rm -rf "${pkg_patchin_dir}/.ptxdist/git-patches" || return mkdir "${pkg_patchin_dir}/.ptxdist/git-patches" || return local patch para @@ -115,6 +194,7 @@ ptxd_make_world_patchin_apply_git_compat() case "${patch}" in ""|"#"*) continue ;; # skip empty lines and comments + *staged*) ptxd_bailout "Patches from the staged directory are NOT allowed in the 'series' file";; *.gz) cat="zcat" ;; *.bz2) cat="bzcat" ;; *) @@ -145,10 +225,156 @@ ptxd_make_world_patchin_apply_git() git quiltimport \ --patches "${pkg_patchin_dir}/.ptxdist/git-patches" \ - --author "unknown author <[email protected]>" + --author "unknown author <[email protected]>" || return + + # Store the md5 of all the patches we just applied. This is used to determine + # if the original patch series has been changed. When those have changed + # we need to rebuild the git tree from scratch and rebase all local modifications + # on top of the new tree. + patches_current_md5=$(find ${pkg_patchin_dir}/.ptxdist/patches/ -type f | \ + grep -v '/staged/' | sort | xargs cat | md5sum | cut -d' ' -f1) + git tag -f -m "md5:${patches_current_md5}" patches-current } export -f ptxd_make_world_patchin_apply_git +ptxd_make_world_update_cleanup() +{ + # Cleanup possible interrupted re-generation of the tree + if [ "$(git branch -a | grep temp_new | grep '*')" ]; then + # Tree may be dirty, force reset it. + git rebase --abort > /dev/null 2>&1 + git reset --hard temp_new > /dev/null 2>&1 + fi + if [ "$(git branch -a | grep temp_old | grep '*')" ]; then + # Tree may be dirty, force reset it. + git rebase --abort > /dev/null 2>&1 + git reset --hard temp_old > /dev/null 2>&1 + fi + if [ "$(git branch -a | grep master | grep '*')" = "" ]; then + echo "Checking out 'master' branch" + git checkout master || \ + ptxd_bailout "Failed to checkout the 'master' branch of the git tree" + fi + git branch -D temp_old > /dev/null 2>&1 + git branch -D temp_new > /dev/null 2>&1 +} +export -f ptxd_make_world_update_cleanup + +ptxd_make_world_update_export_staged() +{ + # + # Export the developers committed/staged patches to the patches/staged directory + # These are the patches on top of the patches-current tag + # Strip off the first line containing the commit ID and the last line containing + # the git version. Before exporting the patch, check if it really has changed + # compared to the previous version + # + rm -rf "${pkg_patch_staged_dir}/tmp" + mkdir -p "${pkg_patch_staged_dir}/tmp/export" "${pkg_patch_staged_dir}/tmp/compare" || \ + ptxd_bailout "Failed to create directory ${pkg_patch_staged_dir}" + git format-patch -N -s -o "${pkg_patch_staged_dir}/tmp/export" patches-current > /dev/null || \ + ptxd_bailout "Failed to export patches to ${pkg_patch_staged_dir}/tmp/export" + if [ $(ls -1 ${pkg_patch_staged_dir}/tmp/export/ | wc -l) -gt 0 ]; then + for eachpatch in ${pkg_patch_staged_dir}/tmp/export/*; do + tail -n +2 ${eachpatch} | head -n -2 > ${pkg_patch_staged_dir}/tmp/compare/$(basename ${eachpatch}) || \ + ptxd_bailout "Failed to strip commit ids from patches ${eachpatch}" + done + for eachpatch in ${pkg_patch_staged_dir}/tmp/compare/*; do + patchname=$(basename ${eachpatch}) + if [ -f "${pkg_patch_staged_dir}/${patchname}" ]; then + cmp -s "${eachpatch}" "${pkg_patch_staged_dir}/${patchname}" + if [ $? -ne 0 ]; then + # Patch has been updated, copy patch and force recompile of the tree + echo "Patch ${patchname} has been updated" + cp -f "${eachpatch}" "${pkg_patch_staged_dir}/${patchname}" + pkg_update_touch_stage=1 + fi + else + # just copy the file, and force recompile + echo "Patch ${patchname} has been created" + cp "${eachpatch}" "${pkg_patch_staged_dir}/${patchname}" + pkg_update_touch_stage=1 + fi + done + fi + rm -rf "${pkg_patch_staged_dir}/tmp" + # Housekeeping: Remove the stage directory if it is empty. + rmdir --ignore-fail-on-non-empty ${pkg_patch_staged_dir} +} +export -f ptxd_make_world_update_export_staged + +# +# The ptxd_make_world_update routine compares the patches series currently in +# the patches directory with the series that were applied while extracting +# the package. Once it detects the patches of the package has changed +# it will rebuild the entire git-tree from the base up with the new patch +# series. Once these series has been applied the local changes made by the +# developer will be rebased on top of the new tree. +# +# It will also output the changes of the developer to a directory +# called 'patches/staged', but they will not automatically be added to the +# series file. It is up to the developer to move the patches from the +# staged dir and to add them to the series file by hand. They are just +# stored in the staged dir to be sure these changes will not get lost. +# +# returns nonzero in case the <PKG>.update needs to be updated, zero if not +ptxd_make_world_update() +{ + local pkg_update_touch_stage=0 + local pkg_patchin_dir=${pkg_deprecated_patchin_dir:-${pkg_dir}} + local pkg_patch_staged_dir=${pkg_patchin_dir}/.ptxdist/patches/staged + + # in case the extract phase has (re-)run, we always need to update the + # update target to force rerun of prepare, compile and so on. + # (This also works in case ${pkg_target_update} does not exist yet + [ ${pkg_target_update} -ot ${pkg_target_extract} ] && pkg_update_touch_stage=1 + + # Check if the git dependencies mechanism is enabled + [ "${PTXCONF_SETUP_PATCHIN_GIT}" = "y" ] || return ${pkg_update_touch_stage} + [ ${PTXCONF_SETUP_PATCHIN_GIT_DEPENDENCIES} ] || return ${pkg_update_touch_stage} + [ -d ${pkg_patchin_dir}/.git ] || return ${pkg_update_touch_stage} + + # This link might get broken on version upgrades of ptxdist while + # using a project build with older version of ptxdist. In that case + # a clean build might be required + [ $(readlink -e ${pkg_patchin_dir}/.ptxdist/patches) ] || + ptxd_bailout "Symbolic link '${pkg_patchin_dir}/.ptxdist/patches' is broken, perform a 'ptxdist clean <PKG>'" + + pushd "${pkg_patchin_dir}" > /dev/null + + ptxd_make_world_update_cleanup + ptxd_make_world_update_export_staged + + # + # When the patches directory contents have changed, we need to rebuild the + # entire git tree and move our current work on top of it. + # + patches_current_md5=$(find ${pkg_patchin_dir}/.ptxdist/patches/ -type f | \ + grep -v '/staged/' | sort | xargs cat | md5sum | cut -d' ' -f1) + patches_stored_md5=$(git show -s --pretty=format: patches-current | grep 'md5' | cut -d':' -f2) + if [ "${patches_current_md5}" != "${patches_stored_md5}" ]; then + echo "--> Patch series has changed! Rebuilding the git tree" + # Check for uncommitted changes + if git update-index --refresh --unmerged | read dummy; then + ptxd_bailout "Working tree dirty, refusing to update the git tree" + fi + if git diff-index --name-only HEAD | read dummy; then + ptxd_bailout "Working tree dirty, refusing(2) to update the git tree" + fi + # Now do the rebuild of the git-tree and rebase + (git checkout -b temp_old patches-current && + git checkout -b temp_new base && + ptxd_make_world_patchin_apply_git && + git rebase --onto temp_new temp_old master && + git branch -D temp_old && + git branch -d temp_new) || \ + ptxd_bailout "Failed to rebuild the git tree in ${pkg_patchin_dir}" + pkg_update_touch_stage=1 + fi + popd > /dev/null + return ${pkg_update_touch_stage} +} +export -f ptxd_make_world_update # # apply patch series with quilt -- ptxdist mailing list [email protected]
