Re: [gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)
On 07/08/2018 01:48 AM, Fabian Groffen wrote: > Would it be worth having a flag to keep long standing sync behaviour for > situations where either diskspace, or verification results in problems > (like bug #657320)? > > Fabian Yeah, also the hardlinks will not work with overlayfs [1], so I've posted a news item [2] explaining that people can set set "sync-allow-hardlinks = no" in repos.conf to disable the new behavior. [1] https://www.brunsware.de/blog/gentoo/portage-tree-squashfs-overlayfs.html [2] https://archives.gentoo.org/gentoo-dev/message/004d9ca4c918e55638a919b87b5d52dd -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)
Would it be worth having a flag to keep long standing sync behaviour for
situations where either diskspace, or verification results in problems
(like bug #657320)?
Fabian
On 07-07-2018 18:55:52 -0700, Zac Medico wrote:
> Sync into a quarantine subdirectory, using the rsync --link-dest option
> to create hardlinks to identical files in the previous snapshot of the
> repository. If hardlinks are not supported, then show a warning message
> and sync directly to the normal repository location.
>
> If verification succeeds, then the quarantine subdirectory is synced
> to the normal repository location, and the quarantine subdirectory
> is deleted. If verification fails, then the quarantine directory is
> preserved for purposes of analysis.
>
> Even if verification happens to be disabled, the quarantine directory
> is still useful for making the repository update more atomic, so that
> it is less likely that normal repository location will be observed in
> a partially synced state.
>
> The new behavior may conflict with configurations that restrict the
> use of hardlinks, such as overlay filesystems. Therefore, users will
> have to set "sync-allow-hardlinks = no" in repos.conf if they have
> a configuration that prevents the use of hardlinks, but this should
> not be very common.
>
> Bug: https://bugs.gentoo.org/660410
> ---
> [PATCH v2] makes it possible to disable the new behavior by setting
> "sync-allow-hardlinks = no" in repos.conf
>
> cnf/repos.conf | 1 +
> man/portage.5 | 8 +++
> pym/portage/repository/config.py| 8 ++-
> pym/portage/sync/modules/rsync/rsync.py | 89
> ++---
> 4 files changed, 97 insertions(+), 9 deletions(-)
>
> diff --git a/cnf/repos.conf b/cnf/repos.conf
> index 352073cfd5..419f6d1182 100644
> --- a/cnf/repos.conf
> +++ b/cnf/repos.conf
> @@ -6,6 +6,7 @@ location = /usr/portage
> sync-type = rsync
> sync-uri = rsync://rsync.gentoo.org/gentoo-portage
> auto-sync = yes
> +sync-allow-hardlinks = yes
> sync-rsync-verify-jobs = 1
> sync-rsync-verify-metamanifest = yes
> sync-rsync-verify-max-age = 24
> diff --git a/man/portage.5 b/man/portage.5
> index 5adb07d821..acc80791be 100644
> --- a/man/portage.5
> +++ b/man/portage.5
> @@ -973,6 +973,14 @@ files). Defaults to true.
> .br
> Valid values: true, false.
> .TP
> +.B sync\-allow\-hardlinks = yes|no
> +Allow sync plugins to use hardlinks in order to ensure that a repository
> +remains in a valid state if something goes wrong during the sync operation.
> +For example, if signature verification fails during a sync operation,
> +the previous state of the repository will be preserved. This option may
> +conflict with configurations that restrict the use of hardlinks, such as
> +overlay filesystems.
> +.TP
> .B sync\-cvs\-repo
> Specifies CVS repository.
> .TP
> diff --git a/pym/portage/repository/config.py
> b/pym/portage/repository/config.py
> index 1d897bb903..c7440369c2 100644
> --- a/pym/portage/repository/config.py
> +++ b/pym/portage/repository/config.py
> @@ -86,6 +86,7 @@ class RepoConfig(object):
> 'sync_type', 'sync_umask', 'sync_uri', 'sync_user',
> 'thin_manifest',
> 'update_changelog', '_eapis_banned', '_eapis_deprecated',
> '_masters_orig', 'module_specific_options',
> 'manifest_required_hashes',
> + 'sync_allow_hardlinks',
> 'sync_openpgp_key_path',
> 'sync_openpgp_key_refresh_retry_count',
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -188,6 +189,9 @@ class RepoConfig(object):
> self.strict_misc_digests = repo_opts.get(
> 'strict-misc-digests', 'true').lower() == 'true'
>
> + self.sync_allow_hardlinks = repo_opts.get(
> + 'sync-allow-hardlinks', 'true').lower() in ('true',
> 'yes')
> +
> self.sync_openpgp_key_path = repo_opts.get(
> 'sync-openpgp-key-path', None)
>
> @@ -534,6 +538,7 @@ class RepoConfigLoader(object):
> 'clone_depth',
> 'eclass_overrides',
> 'force', 'masters',
> 'priority', 'strict_misc_digests',
> 'sync_depth',
> 'sync_hooks_only_on_change',
> + 'sync_allow_hardlinks',
> 'sync_openpgp_key_path',
>
> 'sync_openpgp_key_refresh_retry_count',
>
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -962,7 +967,8 @@ class RepoConfigLoader(object):
> def config_string(self):
> bool_keys = ("strict_misc_digests",)
> str_or_int_keys = ("auto_sync", "clone_depth", "format",
> "location",
> -
[gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)
Sync into a quarantine subdirectory, using the rsync --link-dest option
to create hardlinks to identical files in the previous snapshot of the
repository. If hardlinks are not supported, then show a warning message
and sync directly to the normal repository location.
If verification succeeds, then the quarantine subdirectory is synced
to the normal repository location, and the quarantine subdirectory
is deleted. If verification fails, then the quarantine directory is
preserved for purposes of analysis.
Even if verification happens to be disabled, the quarantine directory
is still useful for making the repository update more atomic, so that
it is less likely that normal repository location will be observed in
a partially synced state.
The new behavior may conflict with configurations that restrict the
use of hardlinks, such as overlay filesystems. Therefore, users will
have to set "sync-allow-hardlinks = no" in repos.conf if they have
a configuration that prevents the use of hardlinks, but this should
not be very common.
Bug: https://bugs.gentoo.org/660410
---
[PATCH v2] makes it possible to disable the new behavior by setting
"sync-allow-hardlinks = no" in repos.conf
cnf/repos.conf | 1 +
man/portage.5 | 8 +++
pym/portage/repository/config.py| 8 ++-
pym/portage/sync/modules/rsync/rsync.py | 89 ++---
4 files changed, 97 insertions(+), 9 deletions(-)
diff --git a/cnf/repos.conf b/cnf/repos.conf
index 352073cfd5..419f6d1182 100644
--- a/cnf/repos.conf
+++ b/cnf/repos.conf
@@ -6,6 +6,7 @@ location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes
+sync-allow-hardlinks = yes
sync-rsync-verify-jobs = 1
sync-rsync-verify-metamanifest = yes
sync-rsync-verify-max-age = 24
diff --git a/man/portage.5 b/man/portage.5
index 5adb07d821..acc80791be 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -973,6 +973,14 @@ files). Defaults to true.
.br
Valid values: true, false.
.TP
+.B sync\-allow\-hardlinks = yes|no
+Allow sync plugins to use hardlinks in order to ensure that a repository
+remains in a valid state if something goes wrong during the sync operation.
+For example, if signature verification fails during a sync operation,
+the previous state of the repository will be preserved. This option may
+conflict with configurations that restrict the use of hardlinks, such as
+overlay filesystems.
+.TP
.B sync\-cvs\-repo
Specifies CVS repository.
.TP
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 1d897bb903..c7440369c2 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -86,6 +86,7 @@ class RepoConfig(object):
'sync_type', 'sync_umask', 'sync_uri', 'sync_user',
'thin_manifest',
'update_changelog', '_eapis_banned', '_eapis_deprecated',
'_masters_orig', 'module_specific_options',
'manifest_required_hashes',
+ 'sync_allow_hardlinks',
'sync_openpgp_key_path',
'sync_openpgp_key_refresh_retry_count',
'sync_openpgp_key_refresh_retry_delay_max',
@@ -188,6 +189,9 @@ class RepoConfig(object):
self.strict_misc_digests = repo_opts.get(
'strict-misc-digests', 'true').lower() == 'true'
+ self.sync_allow_hardlinks = repo_opts.get(
+ 'sync-allow-hardlinks', 'true').lower() in ('true',
'yes')
+
self.sync_openpgp_key_path = repo_opts.get(
'sync-openpgp-key-path', None)
@@ -534,6 +538,7 @@ class RepoConfigLoader(object):
'clone_depth',
'eclass_overrides',
'force', 'masters',
'priority', 'strict_misc_digests',
'sync_depth',
'sync_hooks_only_on_change',
+ 'sync_allow_hardlinks',
'sync_openpgp_key_path',
'sync_openpgp_key_refresh_retry_count',
'sync_openpgp_key_refresh_retry_delay_max',
@@ -962,7 +967,8 @@ class RepoConfigLoader(object):
def config_string(self):
bool_keys = ("strict_misc_digests",)
str_or_int_keys = ("auto_sync", "clone_depth", "format",
"location",
- "main_repo", "priority", "sync_depth",
"sync_openpgp_key_path",
+ "main_repo", "priority", "sync_depth",
+ "sync_allow_hardlinks", "sync_openpgp_key_path",
"sync_openpgp_key_refresh_retry_count",
"sync_openpgp_key_refresh_retry_delay_max",
