Re: [gentoo-portage-dev] [PATCH 0/4] Add sync-rcu support for rsync (bug 662070)
On Mon, 6 Aug 2018 00:40:29 -0700 Zac Medico wrote: > Add a boolean sync-rcu repos.conf setting that behaves as follows: > > sync-rcu = yes|no > > Enable read-copy-update (RCU) behavior for sync operations. The > current latest immutable version of a repository will be > referenced by a symlink found where the repository would normally be > located (see the location setting). Repository consumers should > resolve the cannonical path of this symlink before attempt to access > the repository, and all operations should be read-only, since > the repository is considered immutable. Updates occur by atomic > replacement of the symlink, which causes new consumers to use the > new immutable version, while any earlier consumers continue to > use the cannonical path that was resolved earlier. This option > requires sync-allow-hardlinks and sync-rcu-store-dir options to > be enabled, and currently also requires that sync-type is set > to rsync. This option is disabled by default, since the symlink > usage would require special handling for scenarios involving bind > mounts and chroots. > > sync-rcu-store-dir > > Directory path reserved for sync-rcu storage. This directory must > have a unique value for each repository (do not set it in the > DEFAULT section). This directory must not contain any other files > or directories aside from those that are created automatically > when sync-rcu is enabled. > > sync-rcu-spare-snapshots = 1 > > Number of spare snapshots for sync-rcu to retain with expired > ttl. This protects the previous latest snapshot from being removed > immediately after a new version becomes available, since it might > still be used by running processes. > > sync-rcu-ttl-days = 7 > > Number of days for sync-rcu to retain previous immutable snapshots > of a repository. After the ttl of a particular snapshot has > expired, it will be remove automatically (the latest snapshot > is exempt, and sync-rcu-spare-snapshots configures the number of > previous snapshots that are exempt). If the ttl is set too low, > then a snapshot could expire while it is in use by a running > process. > > Zac Medico (4): > Implement asyncio.iscoroutinefunction for compat_coroutine > Add _sync_decorator module > rsync: split out repo storage framework > Add sync-rcu support for rsync (bug 662070) > > lib/portage/repository/config.py | 36 ++- > lib/portage/repository/storage/__init__.py | 0 > .../repository/storage/hardlink_quarantine.py | 95 > lib/portage/repository/storage/hardlink_rcu.py | 251 > + > lib/portage/repository/storage/inplace.py | 49 > lib/portage/repository/storage/interface.py| 87 +++ > lib/portage/sync/controller.py | 1 + > lib/portage/sync/modules/rsync/rsync.py| 85 ++- > lib/portage/sync/syncbase.py | 33 > +++ .../tests/util/futures/test_compat_coroutine.py| 14 ++ > lib/portage/util/futures/_asyncio/__init__.py | 14 ++ > lib/portage/util/futures/_sync_decorator.py| 54 + > lib/portage/util/futures/compat_coroutine.py | 12 + > man/portage.5 | 35 +++ 14 files > changed, 700 insertions(+), 66 deletions(-) create mode 100644 > lib/portage/repository/storage/__init__.py create mode 100644 > lib/portage/repository/storage/hardlink_quarantine.py create mode > 100644 lib/portage/repository/storage/hardlink_rcu.py create mode > 100644 lib/portage/repository/storage/inplace.py create mode 100644 > lib/portage/repository/storage/interface.py create mode 100644 > lib/portage/util/futures/_sync_decorator.py > series looks good, just the one typo
Re: [gentoo-portage-dev] [PATCH 3/4] rsync: split out repo storage framework
On Mon, 6 Aug 2018 00:40:32 -0700 Zac Medico wrote: > Since there aremany ways to manage repository storage, split out a > repo storage framework. The HardlinkQuarantineRepoStorage class > implements the existing default behavior, and the InplaceRepoStorage > class implements the legacy behavior (when sync-allow-hardlinks is > disabled in repos.conf). > > Each class implements RepoStorageInterface, which uses coroutine > methods since coroutines are well-suited to the I/O bound tasks that > these methods perform. The _sync_decorator is used to convert > coroutine methods to synchronous methods, for smooth integration into > the surrounding synchronous code. > > Bug: https://bugs.gentoo.org/662070 > missing space in first line of commit message s/aremany/are many ---
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On Thu, 9 Aug 2018 16:04:42 -0700
Zac Medico wrote:
> For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available,
> so the code from bug 630292 works for USERLAND=BSD.
>
> Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support
> (bug 630292)) Reported-by: Michał Górny
> Bug: https://bugs.gentoo.org/663256
> ---
> bin/isolated-functions.sh | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index 28ca94532..cac42a4c5 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -448,7 +448,11 @@ fi
> if [[ -z ${XARGS} ]] ; then
> case ${USERLAND} in
> BSD)
> - export XARGS="xargs"
> + if type -P gxargs > /dev/null; then
> + export XARGS="gxargs -r"
> + else
> + export XARGS="xargs"
> + fi
> ;;
> *)
> export XARGS="xargs -r"
LGTM
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On 10/08/18 00:17, M. J. Everitt wrote:
> On 10/08/18 00:04, Zac Medico wrote:
>> For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available,
>> so the code from bug 630292 works for USERLAND=BSD.
>>
>> Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support (bug
>> 630292))
>> Reported-by: Michał Górny
>> Bug: https://bugs.gentoo.org/663256
>> ---
>> bin/isolated-functions.sh | 6 +-
>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
>> index 28ca94532..cac42a4c5 100644
>> --- a/bin/isolated-functions.sh
>> +++ b/bin/isolated-functions.sh
>> @@ -448,7 +448,11 @@ fi
>> if [[ -z ${XARGS} ]] ; then
>> case ${USERLAND} in
>> BSD)
>> -export XARGS="xargs"
>> +if type -P gxargs > /dev/null; then
>> +export XARGS="gxargs -r"
>> +else
>> +export XARGS="xargs"
>> +fi
>> ;;
>> *)
>> export XARGS="xargs -r"
> Isn't the else clause there redundant?
>
Oops, sorry missed the minus before the first export .. my bad ..
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On 10/08/18 00:04, Zac Medico wrote:
> For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available,
> so the code from bug 630292 works for USERLAND=BSD.
>
> Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support (bug
> 630292))
> Reported-by: Michał Górny
> Bug: https://bugs.gentoo.org/663256
> ---
> bin/isolated-functions.sh | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index 28ca94532..cac42a4c5 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -448,7 +448,11 @@ fi
> if [[ -z ${XARGS} ]] ; then
> case ${USERLAND} in
> BSD)
> - export XARGS="xargs"
> + if type -P gxargs > /dev/null; then
> + export XARGS="gxargs -r"
> + else
> + export XARGS="xargs"
> + fi
> ;;
> *)
> export XARGS="xargs -r"
Isn't the else clause there redundant?
signature.asc
Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available,
so the code from bug 630292 works for USERLAND=BSD.
Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support (bug 630292))
Reported-by: Michał Górny
Bug: https://bugs.gentoo.org/663256
---
bin/isolated-functions.sh | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index 28ca94532..cac42a4c5 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -448,7 +448,11 @@ fi
if [[ -z ${XARGS} ]] ; then
case ${USERLAND} in
BSD)
- export XARGS="xargs"
+ if type -P gxargs > /dev/null; then
+ export XARGS="gxargs -r"
+ else
+ export XARGS="xargs"
+ fi
;;
*)
export XARGS="xargs -r"
--
2.16.4
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
On 08/09/2018 02:13 AM, Michał Górny wrote:
> W dniu czw, 09.08.2018 o godzinie 11∶09 +0200, użytkownik Ulrich Mueller
> napisał:
>>> On Thu, 09 Aug 2018, Michał Górny wrote:
+ if xargs --help | grep -q -- --max-procs=; then
>>> '--help' is not a valid argument on FreeBSD, so this spews errors
>>> to stderr:
>>
>> IIRC, GNU findutils is required by PMS?
>>
>
> Yes, using gxargs is another option. I suppose the usual ebuild env
> override is not used in this phase.
The 'alias xargs=gxargs' from profiles/default/bsd/fbsd/profile.bashrc
is ineffective because it doesn't apply to the XARGS variable that
portage defines in isolated-functions.sh:
case ${USERLAND} in
BSD)
export XARGS="xargs"
;;
*)
export XARGS="xargs -r"
;;
esac
I guess it should simply use gxargs if available. However, I don't see
anything in PMS about the 'g' prefix, though I do see this in the
findutils ebuilds:
program_prefix=$(usex userland_GNU '' g)
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
W dniu czw, 09.08.2018 o godzinie 11∶09 +0200, użytkownik Ulrich Mueller napisał: > > > > > > On Thu, 09 Aug 2018, Michał Górny wrote: > > > + if xargs --help | grep -q -- --max-procs=; then > > '--help' is not a valid argument on FreeBSD, so this spews errors > > to stderr: > > IIRC, GNU findutils is required by PMS? > Yes, using gxargs is another option. I suppose the usual ebuild env override is not used in this phase. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
> On Thu, 09 Aug 2018, Michał Górny wrote: >> +if xargs --help | grep -q -- --max-procs=; then > '--help' is not a valid argument on FreeBSD, so this spews errors > to stderr: IIRC, GNU findutils is required by PMS? Ulrich
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/60pngfix: parallel support (bug 630292)
W dniu śro, 25.07.2018 o godzinie 12∶59 -0700, użytkownik Zac Medico
napisał:
> If xargs supports the --max-procs option then use the makeopts_jobs
> function from helper-functions.sh to generate a --max-procs argument.
>
> Bug: https://bugs.gentoo.org/630292
> ---
> bin/install-qa-check.d/60pngfix | 14 --
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/bin/install-qa-check.d/60pngfix b/bin/install-qa-check.d/60pngfix
> index 8d53040b6..2bf020079 100644
> --- a/bin/install-qa-check.d/60pngfix
> +++ b/bin/install-qa-check.d/60pngfix
> @@ -1,7 +1,17 @@
> # Check for issues with PNG files
>
> +source "${PORTAGE_BIN_PATH}/helper-functions.sh" || exit 1
> +
> pngfix_check() {
> - local pngfix=$(type -P pngfix)
> + local jobs pngfix=$(type -P pngfix) xargs=(${XARGS})
> +
> + if xargs --help | grep -q -- --max-procs=; then
'--help' is not a valid argument on FreeBSD, so this spews errors
to stderr:
* Final size of build directory: 3655 KiB (3.5 MiB)
* Final size of installed tree:24 KiB
xargs: illegal option -- -
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements] [-S replsize]]
[-J replstr] [-L number] [-n number [-x]] [-P maxprocs]
[-s size] [utility [argument ...]]
strip: strip --strip-unneeded -R .comment -R .GCC.command.line -R
.note.gnu.gold-version
usr/lib/libgraphene-1.0.so.0.800.2
Also note that -P is more portable than --max-procs.
> + jobs=$(makeopts_jobs)
> + if [[ ${jobs} -gt 1 ]]; then
> + xargs+=(--max-procs=${jobs})
> + fi
> + fi
> +
> if [[ -n ${pngfix} ]] ; then
> local pngout=()
> local next
> @@ -25,7 +35,7 @@ pngfix_check() {
> fi
> eqawarn " ${pngout[@]:7}: ${error}"
> fi
> - done < <(find "${ED}" -type f -name '*.png' -exec "${pngfix}"
> {} +)
> + done < <(find "${ED}" -type f -name '*.png' -print0 |
> "${xargs[@]}" -0 "${pngfix}")
> fi
> }
>
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
