Re: [gentoo-portage-dev] [PATCH 0/4] Add sync-rcu support for rsync (bug 662070)
On Mon, 6 Aug 2018 00:40:29 -0700 Zac Medico wrote: > Add a boolean sync-rcu repos.conf setting that behaves as follows: > > sync-rcu = yes|no > > Enable read-copy-update (RCU) behavior for sync operations. The > current latest immutable version of a repository will be > referenced by a symlink found where the repository would normally be > located (see the location setting). Repository consumers should > resolve the cannonical path of this symlink before attempt to access > the repository, and all operations should be read-only, since > the repository is considered immutable. Updates occur by atomic > replacement of the symlink, which causes new consumers to use the > new immutable version, while any earlier consumers continue to > use the cannonical path that was resolved earlier. This option > requires sync-allow-hardlinks and sync-rcu-store-dir options to > be enabled, and currently also requires that sync-type is set > to rsync. This option is disabled by default, since the symlink > usage would require special handling for scenarios involving bind > mounts and chroots. > > sync-rcu-store-dir > > Directory path reserved for sync-rcu storage. This directory must > have a unique value for each repository (do not set it in the > DEFAULT section). This directory must not contain any other files > or directories aside from those that are created automatically > when sync-rcu is enabled. > > sync-rcu-spare-snapshots = 1 > > Number of spare snapshots for sync-rcu to retain with expired > ttl. This protects the previous latest snapshot from being removed > immediately after a new version becomes available, since it might > still be used by running processes. > > sync-rcu-ttl-days = 7 > > Number of days for sync-rcu to retain previous immutable snapshots > of a repository. After the ttl of a particular snapshot has > expired, it will be remove automatically (the latest snapshot > is exempt, and sync-rcu-spare-snapshots configures the number of > previous snapshots that are exempt). If the ttl is set too low, > then a snapshot could expire while it is in use by a running > process. > > Zac Medico (4): > Implement asyncio.iscoroutinefunction for compat_coroutine > Add _sync_decorator module > rsync: split out repo storage framework > Add sync-rcu support for rsync (bug 662070) > > lib/portage/repository/config.py | 36 ++- > lib/portage/repository/storage/__init__.py | 0 > .../repository/storage/hardlink_quarantine.py | 95 > lib/portage/repository/storage/hardlink_rcu.py | 251 > + > lib/portage/repository/storage/inplace.py | 49 > lib/portage/repository/storage/interface.py| 87 +++ > lib/portage/sync/controller.py | 1 + > lib/portage/sync/modules/rsync/rsync.py| 85 ++- > lib/portage/sync/syncbase.py | 33 > +++ .../tests/util/futures/test_compat_coroutine.py| 14 ++ > lib/portage/util/futures/_asyncio/__init__.py | 14 ++ > lib/portage/util/futures/_sync_decorator.py| 54 + > lib/portage/util/futures/compat_coroutine.py | 12 + > man/portage.5 | 35 +++ 14 files > changed, 700 insertions(+), 66 deletions(-) create mode 100644 > lib/portage/repository/storage/__init__.py create mode 100644 > lib/portage/repository/storage/hardlink_quarantine.py create mode > 100644 lib/portage/repository/storage/hardlink_rcu.py create mode > 100644 lib/portage/repository/storage/inplace.py create mode 100644 > lib/portage/repository/storage/interface.py create mode 100644 > lib/portage/util/futures/_sync_decorator.py > series looks good, just the one typo
Re: [gentoo-portage-dev] [PATCH 3/4] rsync: split out repo storage framework
On Mon, 6 Aug 2018 00:40:32 -0700 Zac Medico wrote: > Since there aremany ways to manage repository storage, split out a > repo storage framework. The HardlinkQuarantineRepoStorage class > implements the existing default behavior, and the InplaceRepoStorage > class implements the legacy behavior (when sync-allow-hardlinks is > disabled in repos.conf). > > Each class implements RepoStorageInterface, which uses coroutine > methods since coroutines are well-suited to the I/O bound tasks that > these methods perform. The _sync_decorator is used to convert > coroutine methods to synchronous methods, for smooth integration into > the surrounding synchronous code. > > Bug: https://bugs.gentoo.org/662070 > missing space in first line of commit message s/aremany/are many ---
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On Thu, 9 Aug 2018 16:04:42 -0700 Zac Medico wrote: > For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available, > so the code from bug 630292 works for USERLAND=BSD. > > Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support > (bug 630292)) Reported-by: Michał Górny > Bug: https://bugs.gentoo.org/663256 > --- > bin/isolated-functions.sh | 6 +- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh > index 28ca94532..cac42a4c5 100644 > --- a/bin/isolated-functions.sh > +++ b/bin/isolated-functions.sh > @@ -448,7 +448,11 @@ fi > if [[ -z ${XARGS} ]] ; then > case ${USERLAND} in > BSD) > - export XARGS="xargs" > + if type -P gxargs > /dev/null; then > + export XARGS="gxargs -r" > + else > + export XARGS="xargs" > + fi > ;; > *) > export XARGS="xargs -r" LGTM
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On 10/08/18 00:17, M. J. Everitt wrote: > On 10/08/18 00:04, Zac Medico wrote: >> For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available, >> so the code from bug 630292 works for USERLAND=BSD. >> >> Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support (bug >> 630292)) >> Reported-by: Michał Górny >> Bug: https://bugs.gentoo.org/663256 >> --- >> bin/isolated-functions.sh | 6 +- >> 1 file changed, 5 insertions(+), 1 deletion(-) >> >> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh >> index 28ca94532..cac42a4c5 100644 >> --- a/bin/isolated-functions.sh >> +++ b/bin/isolated-functions.sh >> @@ -448,7 +448,11 @@ fi >> if [[ -z ${XARGS} ]] ; then >> case ${USERLAND} in >> BSD) >> -export XARGS="xargs" >> +if type -P gxargs > /dev/null; then >> +export XARGS="gxargs -r" >> +else >> +export XARGS="xargs" >> +fi >> ;; >> *) >> export XARGS="xargs -r" > Isn't the else clause there redundant? > Oops, sorry missed the minus before the first export .. my bad .. signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On 10/08/18 00:04, Zac Medico wrote: > For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available, > so the code from bug 630292 works for USERLAND=BSD. > > Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support (bug > 630292)) > Reported-by: Michał Górny > Bug: https://bugs.gentoo.org/663256 > --- > bin/isolated-functions.sh | 6 +- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh > index 28ca94532..cac42a4c5 100644 > --- a/bin/isolated-functions.sh > +++ b/bin/isolated-functions.sh > @@ -448,7 +448,11 @@ fi > if [[ -z ${XARGS} ]] ; then > case ${USERLAND} in > BSD) > - export XARGS="xargs" > + if type -P gxargs > /dev/null; then > + export XARGS="gxargs -r" > + else > + export XARGS="xargs" > + fi > ;; > *) > export XARGS="xargs -r" Isn't the else clause there redundant? signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available, so the code from bug 630292 works for USERLAND=BSD. Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support (bug 630292)) Reported-by: Michał Górny Bug: https://bugs.gentoo.org/663256 --- bin/isolated-functions.sh | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh index 28ca94532..cac42a4c5 100644 --- a/bin/isolated-functions.sh +++ b/bin/isolated-functions.sh @@ -448,7 +448,11 @@ fi if [[ -z ${XARGS} ]] ; then case ${USERLAND} in BSD) - export XARGS="xargs" + if type -P gxargs > /dev/null; then + export XARGS="gxargs -r" + else + export XARGS="xargs" + fi ;; *) export XARGS="xargs -r" -- 2.16.4
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
On 08/09/2018 02:13 AM, Michał Górny wrote: > W dniu czw, 09.08.2018 o godzinie 11∶09 +0200, użytkownik Ulrich Mueller > napisał: >>> On Thu, 09 Aug 2018, Michał Górny wrote: + if xargs --help | grep -q -- --max-procs=; then >>> '--help' is not a valid argument on FreeBSD, so this spews errors >>> to stderr: >> >> IIRC, GNU findutils is required by PMS? >> > > Yes, using gxargs is another option. I suppose the usual ebuild env > override is not used in this phase. The 'alias xargs=gxargs' from profiles/default/bsd/fbsd/profile.bashrc is ineffective because it doesn't apply to the XARGS variable that portage defines in isolated-functions.sh: case ${USERLAND} in BSD) export XARGS="xargs" ;; *) export XARGS="xargs -r" ;; esac I guess it should simply use gxargs if available. However, I don't see anything in PMS about the 'g' prefix, though I do see this in the findutils ebuilds: program_prefix=$(usex userland_GNU '' g) -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
W dniu czw, 09.08.2018 o godzinie 11∶09 +0200, użytkownik Ulrich Mueller napisał: > > > > > > On Thu, 09 Aug 2018, Michał Górny wrote: > > > + if xargs --help | grep -q -- --max-procs=; then > > '--help' is not a valid argument on FreeBSD, so this spews errors > > to stderr: > > IIRC, GNU findutils is required by PMS? > Yes, using gxargs is another option. I suppose the usual ebuild env override is not used in this phase. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
> On Thu, 09 Aug 2018, Michał Górny wrote: >> +if xargs --help | grep -q -- --max-procs=; then > '--help' is not a valid argument on FreeBSD, so this spews errors > to stderr: IIRC, GNU findutils is required by PMS? Ulrich
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
W dniu śro, 25.07.2018 o godzinie 12∶59 -0700, użytkownik Zac Medico napisał: > If xargs supports the --max-procs option then use the makeopts_jobs > function from helper-functions.sh to generate a --max-procs argument. > > Bug: https://bugs.gentoo.org/630292 > --- > bin/install-qa-check.d/60pngfix | 14 -- > 1 file changed, 12 insertions(+), 2 deletions(-) > > diff --git a/bin/install-qa-check.d/60pngfix b/bin/install-qa-check.d/60pngfix > index 8d53040b6..2bf020079 100644 > --- a/bin/install-qa-check.d/60pngfix > +++ b/bin/install-qa-check.d/60pngfix > @@ -1,7 +1,17 @@ > # Check for issues with PNG files > > +source "${PORTAGE_BIN_PATH}/helper-functions.sh" || exit 1 > + > pngfix_check() { > - local pngfix=$(type -P pngfix) > + local jobs pngfix=$(type -P pngfix) xargs=(${XARGS}) > + > + if xargs --help | grep -q -- --max-procs=; then '--help' is not a valid argument on FreeBSD, so this spews errors to stderr: * Final size of build directory: 3655 KiB (3.5 MiB) * Final size of installed tree:24 KiB xargs: illegal option -- - usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements] [-S replsize]] [-J replstr] [-L number] [-n number [-x]] [-P maxprocs] [-s size] [utility [argument ...]] strip: strip --strip-unneeded -R .comment -R .GCC.command.line -R .note.gnu.gold-version usr/lib/libgraphene-1.0.so.0.800.2 Also note that -P is more portable than --max-procs. > + jobs=$(makeopts_jobs) > + if [[ ${jobs} -gt 1 ]]; then > + xargs+=(--max-procs=${jobs}) > + fi > + fi > + > if [[ -n ${pngfix} ]] ; then > local pngout=() > local next > @@ -25,7 +35,7 @@ pngfix_check() { > fi > eqawarn " ${pngout[@]:7}: ${error}" > fi > - done < <(find "${ED}" -type f -name '*.png' -exec "${pngfix}" > {} +) > + done < <(find "${ED}" -type f -name '*.png' -print0 | > "${xargs[@]}" -0 "${pngfix}") > fi > } > -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part