Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 11:59 PM, Zac Medico wrote: >> It used to make a special statement for a new stable Portage and >> strongly recommended that it be emerged first. It should probably do the >> same for openpgp-keys-gentoo-release. > Sure, but it this case we have a chicken-and-egg problem, because I > needed the latest openpgp-keys-gentoo-release installed but in order to > do that I had to sync, but then verification failed because I didn't > have the latest openpgp-keys-gentoo-release. We're hopefully attributing that to a startup problem. Obviously as we go along we need to have proper key rollover procedures so this should never happen including backup keys. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 02:59 PM, Zac Medico wrote: > On 07/08/2018 02:50 PM, Aaron W. Swenson wrote: >> On July 8, 2018 5:38:48 PM EDT, Zac Medico wrote: >> >> On 07/08/2018 02:18 PM, Michał Górny wrote: >> >> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico >> napisał: >> >> On 07/08/2018 01:18 PM, Zac Medico wrote: >> >> On 07/08/2018 01:08 PM, Michał Górny wrote: >> >> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, >> użytkownik Zac Medico >> napisał: >> >> On 07/08/2018 11:42 AM, Michał Górny wrote: >> >> W dniu nie, 08.07.2018 o godzinie 11∶04 >> -0700, użytkownik Zac Medico >> napisał: >> >> On 07/08/2018 06:56 AM, Michał Górny wrote: >> >> W dniu nie, 08.07.2018 o godzinie >> 15∶02 +0200, użytkownik Kristian >> Fiskerstrand napisał: >> >> On 07/08/2018 08:53 AM, Michał >> Górny wrote: >> >> Is safe git syncing >> implemented already? If not, >> maybe finish it first and >> cover both with a single >> news item. Git is going to >> be more efficient here, so >> people may want to learn >> they have an alternative. >> >> >> Why complicate things, and >> increase wait for something that >> benefits >> most users, just to give >> alternatives to a few using >> non-default sync >> mechanism. Securing git >> distribution is a whole >> different ballpark. >> >> >> >> Let me rephrase. Let's say I'm using >> rsync. This new feature is >> something positive but it breaks my >> use case (for one of the listed >> reasons -- overlayfs, inode use, >> small fs cache). After reading this >> news item, I learn that my only >> option is to disable the new feature. >> >> Now, I would appreciate being told >> that there's an alternate sync method >> that handles secure updates without >> having all those drawbacks. >> >> >> The thing is, the normal git tree >> doesn't even provide pre-generated >> metadata, and I see then gentoo-mirror >> repo that provides metadata does >> not have commits signed with an release key: >> >> >> https://github.com/gentoo-mirror/gentoo/commits/stable >> >> So I'm really not comfortable >> recommending git to anyone at this point. >> >> >> Wrong twice. >> >> Firstly, the canonical URL is: >> >> >> https://anongit.gentoo.org/git/repo/sync/gentoo.git >> (https://gitweb.gentoo.org/repo/sync/gentoo.git) >> >> Secondly, the merge commits (i.e. top >> commits that are verified >> by Portage) are signed by dedicated key that >> is part of the infra key >> set. In other words, it works out of the box. >> >> >> Is there any documentation that shows users how >> to migrate to git, and >> what the pros and cons might be? Maybe its >> worthy of its own news item. >> >> >> Maybe. I don't really know, and don't think it's a >> good idea to show 30 >> news item of things users might like
[gentoo-dev] News Item: Portage rsync hardlink support [v2]
In v2 there's a suggestion to use git for better efficiency than rsync, as requested by Michał Górny: Title: Portage rsync hardlink support Author: Zac Medico Posted: 2018-07-11 Revision: 1 News-Item-Format: 2.0 Display-If-Installed: sys-apps/portage For users of the rsync tree, beginning with sys-apps/portage-2.3.42, the default behavior for sync operations will use hardlinks in order to ensure that a repository remains in a valid state if something goes wrong [1]. For example, if signature verification fails during a sync operation, the new hardlink behavior will preserve the previous state of the repository. The new behavior may conflict with configurations that restrict the use of hardlinks, such as overlay filesystems. Therefore, users will have to set "sync-allow-hardlinks = no" in repos.conf if they have a configuration that restricts the use of hardlinks, but this should not be very common: [DEFAULT] sync-allow-hardlinks = no Note that it possible to sync more efficiently using git [2] instead of rsync, though git consumes an increasing amount of disk space over time because shallow pull is currently not supported [3]. [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync --link-dest to implement atomic repository updates (and abort if signature verification fails) [2] https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repo [3] https://bugs.gentoo.org/552814 sync using git pull --depth=1 by default -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On Sun, Jul 8, 2018 at 5:50 PM Aaron W. Swenson wrote: > > Does Portage not call attention to critical updates? > > It used to make a special statement for a new stable Portage and strongly > recommended that it be emerged first. It should probably do the same for > openpgp-keys-gentoo-release. If that is only needed for syncing, then I'm not sure there is really a need. If you have the ebuild for the latest keys, then you can install them in any order at any time. If you don't have the ebuild but need it, then portage won't know, and syncing will simply fail unless overridden somehow. -- Rich
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 02:50 PM, Aaron W. Swenson wrote: > On July 8, 2018 5:38:48 PM EDT, Zac Medico wrote: > > On 07/08/2018 02:18 PM, Michał Górny wrote: > > W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico > napisał: > > On 07/08/2018 01:18 PM, Zac Medico wrote: > > On 07/08/2018 01:08 PM, Michał Górny wrote: > > W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, > użytkownik Zac Medico > napisał: > > On 07/08/2018 11:42 AM, Michał Górny wrote: > > W dniu nie, 08.07.2018 o godzinie 11∶04 > -0700, użytkownik Zac Medico > napisał: > > On 07/08/2018 06:56 AM, Michał Górny wrote: > > W dniu nie, 08.07.2018 o godzinie > 15∶02 +0200, użytkownik Kristian > Fiskerstrand napisał: > > On 07/08/2018 08:53 AM, Michał > Górny wrote: > > Is safe git syncing > implemented already? If not, > maybe finish it first and > cover both with a single > news item. Git is going to > be more efficient here, so > people may want to learn > they have an alternative. > > > Why complicate things, and > increase wait for something that > benefits > most users, just to give > alternatives to a few using > non-default sync > mechanism. Securing git > distribution is a whole > different ballpark. > > > > Let me rephrase. Let's say I'm using > rsync. This new feature is > something positive but it breaks my > use case (for one of the listed > reasons -- overlayfs, inode use, > small fs cache). After reading this > news item, I learn that my only > option is to disable the new feature. > > Now, I would appreciate being told > that there's an alternate sync method > that handles secure updates without > having all those drawbacks. > > > The thing is, the normal git tree > doesn't even provide pre-generated > metadata, and I see then gentoo-mirror > repo that provides metadata does > not have commits signed with an release key: > > > https://github.com/gentoo-mirror/gentoo/commits/stable > > So I'm really not comfortable > recommending git to anyone at this point. > > > Wrong twice. > > Firstly, the canonical URL is: > > > https://anongit.gentoo.org/git/repo/sync/gentoo.git > (https://gitweb.gentoo.org/repo/sync/gentoo.git) > > Secondly, the merge commits (i.e. top > commits that are verified > by Portage) are signed by dedicated key that > is part of the infra key > set. In other words, it works out of the box. > > > Is there any documentation that shows users how > to migrate to git, and > what the pros and cons might be? Maybe its > worthy of its own news item. > > > Maybe. I don't really know, and don't think it's a > good idea to show 30 > news item of things users might like on every new > Gentoo install. > > > Well if instructions for setting up git sync
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On July 8, 2018 5:38:48 PM EDT, Zac Medico wrote: >On 07/08/2018 02:18 PM, Michał Górny wrote: >> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico >> napisał: >>> On 07/08/2018 01:18 PM, Zac Medico wrote: On 07/08/2018 01:08 PM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac >Medico > napisał: >> On 07/08/2018 11:42 AM, Michał Górny wrote: >>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac >Medico >>> napisał: On 07/08/2018 06:56 AM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik >Kristian > Fiskerstrand napisał: >> On 07/08/2018 08:53 AM, Michał Górny wrote: >>> Is safe git syncing implemented already? If not, maybe >finish it first and cover both with a single news item. Git is going to >be more efficient here, so people may want to learn they have an >alternative. >> >> Why complicate things, and increase wait for something that >benefits >> most users, just to give alternatives to a few using >non-default sync >> mechanism. Securing git distribution is a whole different >ballpark. >> > > Let me rephrase. Let's say I'm using rsync. This new feature >is > something positive but it breaks my use case (for one of the >listed > reasons -- overlayfs, inode use, small fs cache). After >reading this > news item, I learn that my only option is to disable the new >feature. > > Now, I would appreciate being told that there's an alternate >sync method > that handles secure updates without having all those >drawbacks. The thing is, the normal git tree doesn't even provide >pre-generated metadata, and I see then gentoo-mirror repo that provides >metadata does not have commits signed with an release key: https://github.com/gentoo-mirror/gentoo/commits/stable So I'm really not comfortable recommending git to anyone at >this point. >>> >>> Wrong twice. >>> >>> Firstly, the canonical URL is: >>> >>> https://anongit.gentoo.org/git/repo/sync/gentoo.git >>> (https://gitweb.gentoo.org/repo/sync/gentoo.git) >>> >>> Secondly, the merge commits (i.e. top commits that are verified >>> by Portage) are signed by dedicated key that is part of the >infra key >>> set. In other words, it works out of the box. >> >> Is there any documentation that shows users how to migrate to >git, and >> what the pros and cons might be? Maybe its worthy of its own news >item. > > Maybe. I don't really know, and don't think it's a good idea to >show 30 > news item of things users might like on every new Gentoo install. Well if instructions for setting up git sync and associated >pros/cons are not documented anywhere then I won't advise anyone to use it. >>> >>> I've attempted to configure it for myself, and this is what it does: >>> >>> * Using keys from /usr/share/openpgp-keys/gentoo-release.asc >>> * Refreshing keys from keyserver ... >>>[ ok ] >>> * No valid signature found: unable to verify signature (missing >key?) >>> >> >> Please report a bug and attach your configuration along with keyring >> version. > >It works after upgrading to openpgp-keys-gentoo-release-20180706 from >openpgp-keys-gentoo-release-20180323. >-- >Thanks, >Zac Does Portage not call attention to critical updates? It used to make a special statement for a new stable Portage and strongly recommended that it be emerged first. It should probably do the same for openpgp-keys-gentoo-release. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. signature.asc Description: PGP signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 02:18 PM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico > napisał: >> On 07/08/2018 01:18 PM, Zac Medico wrote: >>> On 07/08/2018 01:08 PM, Michał Górny wrote: W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico napisał: > On 07/08/2018 11:42 AM, Michał Górny wrote: >> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico >> napisał: >>> On 07/08/2018 06:56 AM, Michał Górny wrote: W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian Fiskerstrand napisał: > On 07/08/2018 08:53 AM, Michał Górny wrote: >> Is safe git syncing implemented already? If not, maybe finish it >> first and cover both with a single news item. Git is going to be >> more efficient here, so people may want to learn they have an >> alternative. > > Why complicate things, and increase wait for something that benefits > most users, just to give alternatives to a few using non-default sync > mechanism. Securing git distribution is a whole different ballpark. > Let me rephrase. Let's say I'm using rsync. This new feature is something positive but it breaks my use case (for one of the listed reasons -- overlayfs, inode use, small fs cache). After reading this news item, I learn that my only option is to disable the new feature. Now, I would appreciate being told that there's an alternate sync method that handles secure updates without having all those drawbacks. >>> >>> The thing is, the normal git tree doesn't even provide pre-generated >>> metadata, and I see then gentoo-mirror repo that provides metadata does >>> not have commits signed with an release key: >>> >>> https://github.com/gentoo-mirror/gentoo/commits/stable >>> >>> So I'm really not comfortable recommending git to anyone at this point. >> >> Wrong twice. >> >> Firstly, the canonical URL is: >> >> https://anongit.gentoo.org/git/repo/sync/gentoo.git >> (https://gitweb.gentoo.org/repo/sync/gentoo.git) >> >> Secondly, the merge commits (i.e. top commits that are verified >> by Portage) are signed by dedicated key that is part of the infra key >> set. In other words, it works out of the box. > > Is there any documentation that shows users how to migrate to git, and > what the pros and cons might be? Maybe its worthy of its own news item. Maybe. I don't really know, and don't think it's a good idea to show 30 news item of things users might like on every new Gentoo install. >>> >>> Well if instructions for setting up git sync and associated pros/cons >>> are not documented anywhere then I won't advise anyone to use it. >> >> I've attempted to configure it for myself, and this is what it does: >> >> * Using keys from /usr/share/openpgp-keys/gentoo-release.asc >> * Refreshing keys from keyserver ... >>[ ok ] >> * No valid signature found: unable to verify signature (missing key?) >> > > Please report a bug and attach your configuration along with keyring > version. It works after upgrading to openpgp-keys-gentoo-release-20180706 from openpgp-keys-gentoo-release-20180323. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico napisał: > On 07/08/2018 01:18 PM, Zac Medico wrote: > > On 07/08/2018 01:08 PM, Michał Górny wrote: > > > W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico > > > napisał: > > > > On 07/08/2018 11:42 AM, Michał Górny wrote: > > > > > W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico > > > > > napisał: > > > > > > On 07/08/2018 06:56 AM, Michał Górny wrote: > > > > > > > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian > > > > > > > Fiskerstrand napisał: > > > > > > > > On 07/08/2018 08:53 AM, Michał Górny wrote: > > > > > > > > > Is safe git syncing implemented already? If not, maybe finish > > > > > > > > > it first and cover both with a single news item. Git is going > > > > > > > > > to be more efficient here, so people may want to learn they > > > > > > > > > have an alternative. > > > > > > > > > > > > > > > > Why complicate things, and increase wait for something that > > > > > > > > benefits > > > > > > > > most users, just to give alternatives to a few using > > > > > > > > non-default sync > > > > > > > > mechanism. Securing git distribution is a whole different > > > > > > > > ballpark. > > > > > > > > > > > > > > > > > > > > > > Let me rephrase. Let's say I'm using rsync. This new feature is > > > > > > > something positive but it breaks my use case (for one of the > > > > > > > listed > > > > > > > reasons -- overlayfs, inode use, small fs cache). After reading > > > > > > > this > > > > > > > news item, I learn that my only option is to disable the new > > > > > > > feature. > > > > > > > > > > > > > > Now, I would appreciate being told that there's an alternate sync > > > > > > > method > > > > > > > that handles secure updates without having all those drawbacks. > > > > > > > > > > > > The thing is, the normal git tree doesn't even provide pre-generated > > > > > > metadata, and I see then gentoo-mirror repo that provides metadata > > > > > > does > > > > > > not have commits signed with an release key: > > > > > > > > > > > > https://github.com/gentoo-mirror/gentoo/commits/stable > > > > > > > > > > > > So I'm really not comfortable recommending git to anyone at this > > > > > > point. > > > > > > > > > > Wrong twice. > > > > > > > > > > Firstly, the canonical URL is: > > > > > > > > > > https://anongit.gentoo.org/git/repo/sync/gentoo.git > > > > > (https://gitweb.gentoo.org/repo/sync/gentoo.git) > > > > > > > > > > Secondly, the merge commits (i.e. top commits that are verified > > > > > by Portage) are signed by dedicated key that is part of the infra key > > > > > set. In other words, it works out of the box. > > > > > > > > Is there any documentation that shows users how to migrate to git, and > > > > what the pros and cons might be? Maybe its worthy of its own news item. > > > > > > Maybe. I don't really know, and don't think it's a good idea to show 30 > > > news item of things users might like on every new Gentoo install. > > > > Well if instructions for setting up git sync and associated pros/cons > > are not documented anywhere then I won't advise anyone to use it. > > I've attempted to configure it for myself, and this is what it does: > > * Using keys from /usr/share/openpgp-keys/gentoo-release.asc > * Refreshing keys from keyserver ... >[ ok ] > * No valid signature found: unable to verify signature (missing key?) > Please report a bug and attach your configuration along with keyring version. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 01:18 PM, Zac Medico wrote: > On 07/08/2018 01:08 PM, Michał Górny wrote: >> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico >> napisał: >>> On 07/08/2018 11:42 AM, Michał Górny wrote: W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico napisał: > On 07/08/2018 06:56 AM, Michał Górny wrote: >> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian >> Fiskerstrand napisał: >>> On 07/08/2018 08:53 AM, Michał Górny wrote: Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative. >>> >>> Why complicate things, and increase wait for something that benefits >>> most users, just to give alternatives to a few using non-default sync >>> mechanism. Securing git distribution is a whole different ballpark. >>> >> >> Let me rephrase. Let's say I'm using rsync. This new feature is >> something positive but it breaks my use case (for one of the listed >> reasons -- overlayfs, inode use, small fs cache). After reading this >> news item, I learn that my only option is to disable the new feature. >> >> Now, I would appreciate being told that there's an alternate sync method >> that handles secure updates without having all those drawbacks. > > The thing is, the normal git tree doesn't even provide pre-generated > metadata, and I see then gentoo-mirror repo that provides metadata does > not have commits signed with an release key: > > https://github.com/gentoo-mirror/gentoo/commits/stable > > So I'm really not comfortable recommending git to anyone at this point. Wrong twice. Firstly, the canonical URL is: https://anongit.gentoo.org/git/repo/sync/gentoo.git (https://gitweb.gentoo.org/repo/sync/gentoo.git) Secondly, the merge commits (i.e. top commits that are verified by Portage) are signed by dedicated key that is part of the infra key set. In other words, it works out of the box. >>> >>> Is there any documentation that shows users how to migrate to git, and >>> what the pros and cons might be? Maybe its worthy of its own news item. >> >> Maybe. I don't really know, and don't think it's a good idea to show 30 >> news item of things users might like on every new Gentoo install. > > Well if instructions for setting up git sync and associated pros/cons > are not documented anywhere then I won't advise anyone to use it. I've attempted to configure it for myself, and this is what it does: * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys from keyserver ... [ ok ] * No valid signature found: unable to verify signature (missing key?) -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 01:08 PM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico > napisał: >> On 07/08/2018 11:42 AM, Michał Górny wrote: >>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico >>> napisał: On 07/08/2018 06:56 AM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian > Fiskerstrand napisał: >> On 07/08/2018 08:53 AM, Michał Górny wrote: >>> Is safe git syncing implemented already? If not, maybe finish it first >>> and cover both with a single news item. Git is going to be more >>> efficient here, so people may want to learn they have an alternative. >> >> Why complicate things, and increase wait for something that benefits >> most users, just to give alternatives to a few using non-default sync >> mechanism. Securing git distribution is a whole different ballpark. >> > > Let me rephrase. Let's say I'm using rsync. This new feature is > something positive but it breaks my use case (for one of the listed > reasons -- overlayfs, inode use, small fs cache). After reading this > news item, I learn that my only option is to disable the new feature. > > Now, I would appreciate being told that there's an alternate sync method > that handles secure updates without having all those drawbacks. The thing is, the normal git tree doesn't even provide pre-generated metadata, and I see then gentoo-mirror repo that provides metadata does not have commits signed with an release key: https://github.com/gentoo-mirror/gentoo/commits/stable So I'm really not comfortable recommending git to anyone at this point. >>> >>> Wrong twice. >>> >>> Firstly, the canonical URL is: >>> >>> https://anongit.gentoo.org/git/repo/sync/gentoo.git >>> (https://gitweb.gentoo.org/repo/sync/gentoo.git) >>> >>> Secondly, the merge commits (i.e. top commits that are verified >>> by Portage) are signed by dedicated key that is part of the infra key >>> set. In other words, it works out of the box. >> >> Is there any documentation that shows users how to migrate to git, and >> what the pros and cons might be? Maybe its worthy of its own news item. > > Maybe. I don't really know, and don't think it's a good idea to show 30 > news item of things users might like on every new Gentoo install. Well if instructions for setting up git sync and associated pros/cons are not documented anywhere then I won't advise anyone to use it. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico napisał: > On 07/08/2018 11:42 AM, Michał Górny wrote: > > W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico > > napisał: > > > On 07/08/2018 06:56 AM, Michał Górny wrote: > > > > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian > > > > Fiskerstrand napisał: > > > > > On 07/08/2018 08:53 AM, Michał Górny wrote: > > > > > > Is safe git syncing implemented already? If not, maybe finish it > > > > > > first and cover both with a single news item. Git is going to be > > > > > > more efficient here, so people may want to learn they have an > > > > > > alternative. > > > > > > > > > > Why complicate things, and increase wait for something that benefits > > > > > most users, just to give alternatives to a few using non-default sync > > > > > mechanism. Securing git distribution is a whole different ballpark. > > > > > > > > > > > > > Let me rephrase. Let's say I'm using rsync. This new feature is > > > > something positive but it breaks my use case (for one of the listed > > > > reasons -- overlayfs, inode use, small fs cache). After reading this > > > > news item, I learn that my only option is to disable the new feature. > > > > > > > > Now, I would appreciate being told that there's an alternate sync method > > > > that handles secure updates without having all those drawbacks. > > > > > > The thing is, the normal git tree doesn't even provide pre-generated > > > metadata, and I see then gentoo-mirror repo that provides metadata does > > > not have commits signed with an release key: > > > > > > https://github.com/gentoo-mirror/gentoo/commits/stable > > > > > > So I'm really not comfortable recommending git to anyone at this point. > > > > Wrong twice. > > > > Firstly, the canonical URL is: > > > > https://anongit.gentoo.org/git/repo/sync/gentoo.git > > (https://gitweb.gentoo.org/repo/sync/gentoo.git) > > > > Secondly, the merge commits (i.e. top commits that are verified > > by Portage) are signed by dedicated key that is part of the infra key > > set. In other words, it works out of the box. > > Is there any documentation that shows users how to migrate to git, and > what the pros and cons might be? Maybe its worthy of its own news item. Maybe. I don't really know, and don't think it's a good idea to show 30 news item of things users might like on every new Gentoo install. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On Sun, Jul 8, 2018 at 2:31 PM Kristian Fiskerstrand wrote: > > On 07/08/2018 08:10 PM, Rich Freeman wrote: > > Again, the current portage support for git verification doesn't check > > any developer keys. > > right, so why would it be material for a news item improving the status > quo for those synching using the official rsync method? > I never claimed it should be, though mgorny made a reasonable argument for giving users an alternative if they can't use the hard links. -- Rich
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 11:42 AM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico > napisał: >> On 07/08/2018 06:56 AM, Michał Górny wrote: >>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian >>> Fiskerstrand napisał: On 07/08/2018 08:53 AM, Michał Górny wrote: > Is safe git syncing implemented already? If not, maybe finish it first > and cover both with a single news item. Git is going to be more efficient > here, so people may want to learn they have an alternative. Why complicate things, and increase wait for something that benefits most users, just to give alternatives to a few using non-default sync mechanism. Securing git distribution is a whole different ballpark. >>> >>> Let me rephrase. Let's say I'm using rsync. This new feature is >>> something positive but it breaks my use case (for one of the listed >>> reasons -- overlayfs, inode use, small fs cache). After reading this >>> news item, I learn that my only option is to disable the new feature. >>> >>> Now, I would appreciate being told that there's an alternate sync method >>> that handles secure updates without having all those drawbacks. >> >> The thing is, the normal git tree doesn't even provide pre-generated >> metadata, and I see then gentoo-mirror repo that provides metadata does >> not have commits signed with an release key: >> >> https://github.com/gentoo-mirror/gentoo/commits/stable >> >> So I'm really not comfortable recommending git to anyone at this point. > > Wrong twice. > > Firstly, the canonical URL is: > > https://anongit.gentoo.org/git/repo/sync/gentoo.git > (https://gitweb.gentoo.org/repo/sync/gentoo.git) > > Secondly, the merge commits (i.e. top commits that are verified > by Portage) are signed by dedicated key that is part of the infra key > set. In other words, it works out of the box. Is there any documentation that shows users how to migrate to git, and what the pros and cons might be? Maybe its worthy of its own news item. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico napisał: > On 07/08/2018 06:56 AM, Michał Górny wrote: > > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian > > Fiskerstrand napisał: > > > On 07/08/2018 08:53 AM, Michał Górny wrote: > > > > Is safe git syncing implemented already? If not, maybe finish it first > > > > and cover both with a single news item. Git is going to be more > > > > efficient here, so people may want to learn they have an alternative. > > > > > > Why complicate things, and increase wait for something that benefits > > > most users, just to give alternatives to a few using non-default sync > > > mechanism. Securing git distribution is a whole different ballpark. > > > > > > > Let me rephrase. Let's say I'm using rsync. This new feature is > > something positive but it breaks my use case (for one of the listed > > reasons -- overlayfs, inode use, small fs cache). After reading this > > news item, I learn that my only option is to disable the new feature. > > > > Now, I would appreciate being told that there's an alternate sync method > > that handles secure updates without having all those drawbacks. > > The thing is, the normal git tree doesn't even provide pre-generated > metadata, and I see then gentoo-mirror repo that provides metadata does > not have commits signed with an release key: > > https://github.com/gentoo-mirror/gentoo/commits/stable > > So I'm really not comfortable recommending git to anyone at this point. Wrong twice. Firstly, the canonical URL is: https://anongit.gentoo.org/git/repo/sync/gentoo.git (https://gitweb.gentoo.org/repo/sync/gentoo.git) Secondly, the merge commits (i.e. top commits that are verified by Portage) are signed by dedicated key that is part of the infra key set. In other words, it works out of the box. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 08:10 PM, Rich Freeman wrote: > Again, the current portage support for git verification doesn't check > any developer keys. right, so why would it be material for a news item improving the status quo for those synching using the official rsync method? -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On Sun, Jul 8, 2018 at 1:50 PM Kristian Fiskerstrand wrote: > > On 07/08/2018 07:34 PM, Rich Freeman wrote: > > The patch is to do the verification before > > checking it out so that if it fails the tree is left in a > > last-known-good state (at least as seen by tools at the filesystem > > level - the fetched bad commits would still be visible to git). > > there is still a very different key management issue discussed. If a > developers credentials are removed from Gentoo LDAP for some reason it > will be stopped from pushing new commits immediately, but the third > party verification can be valid up until that point and after since the > developer might not have published a revocation certificate. > > the git sync method will need a way to distinguish this for end-users, > but the proper rsync synchronization will be able to trust the data at > the point we say it is OK. Again, the current portage support for git verification doesn't check any developer keys. It just checks the keys in /usr/share/openpgp-keys/gentoo-release.asc (after fetching updates). If the HEAD commit verifies, it is good, if not it is bad. It doesn't matter whether any commit in the tree other than the HEAD has a valid signature, and it will reject any signature from a developer on HEAD. The logic is somewhat similar to rsync in that regard. There is a separate proposal out there which is unimplemented to check developer keys, and we aren't talking about that. I agree that it has the challenge of figuring out whether the key was acceptable at the time the signature was made. -- Rich
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 06:56 AM, Michał Górny wrote: > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian > Fiskerstrand napisał: >> On 07/08/2018 08:53 AM, Michał Górny wrote: >>> Is safe git syncing implemented already? If not, maybe finish it first and >>> cover both with a single news item. Git is going to be more efficient here, >>> so people may want to learn they have an alternative. >> >> Why complicate things, and increase wait for something that benefits >> most users, just to give alternatives to a few using non-default sync >> mechanism. Securing git distribution is a whole different ballpark. >> > > Let me rephrase. Let's say I'm using rsync. This new feature is > something positive but it breaks my use case (for one of the listed > reasons -- overlayfs, inode use, small fs cache). After reading this > news item, I learn that my only option is to disable the new feature. > > Now, I would appreciate being told that there's an alternate sync method > that handles secure updates without having all those drawbacks. The thing is, the normal git tree doesn't even provide pre-generated metadata, and I see then gentoo-mirror repo that provides metadata does not have commits signed with an release key: https://github.com/gentoo-mirror/gentoo/commits/stable So I'm really not comfortable recommending git to anyone at this point. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 07:34 PM, Rich Freeman wrote: > The patch is to do the verification before > checking it out so that if it fails the tree is left in a > last-known-good state (at least as seen by tools at the filesystem > level - the fetched bad commits would still be visible to git). there is still a very different key management issue discussed. If a developers credentials are removed from Gentoo LDAP for some reason it will be stopped from pushing new commits immediately, but the third party verification can be valid up until that point and after since the developer might not have published a revocation certificate. the git sync method will need a way to distinguish this for end-users, but the proper rsync synchronization will be able to trust the data at the point we say it is OK. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 08/07/18 18:34, Rich Freeman wrote: > On Sun, Jul 8, 2018 at 9:02 AM Kristian Fiskerstrand wrote: >> On 07/08/2018 08:53 AM, Michał Górny wrote: >>> Is safe git syncing implemented already? If not, maybe finish it first and >>> cover both with a single news item. Git is going to be more efficient here, >>> so people may want to learn they have an alternative. >> Why complicate things, and increase wait for something that benefits >> most users, just to give alternatives to a few using non-default sync >> mechanism. Securing git distribution is a whole different ballpark. >> > I'll agree that it is different, but we're talking about verification > of the HEAD signature by infra, not verification of individual > developer keys, which was the topic of the recent thread. > > Verification is already built-into portage for git syncing (but off by > default). The problem is that portage will still checkout the tree if > it fails verification. The patch is to do the verification before > checking it out so that if it fails the tree is left in a > last-known-good state (at least as seen by tools at the filesystem > level - the fetched bad commits would still be visible to git). > Slightly radical thought here, but hear me out .. Could we use this same functionality to be able to validate the tree integrity with respect to CI testing? I mean, if the tree is 'broken' could we have some kind of warning displayed perhaps? Something that could be toggled (or default Off) would indeed be good, so that users/devs can choose what level or 'standard' of tree state they're prepared to accept. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On Sun, Jul 8, 2018 at 9:02 AM Kristian Fiskerstrand wrote: > > On 07/08/2018 08:53 AM, Michał Górny wrote: > > Is safe git syncing implemented already? If not, maybe finish it first and > > cover both with a single news item. Git is going to be more efficient here, > > so people may want to learn they have an alternative. > > Why complicate things, and increase wait for something that benefits > most users, just to give alternatives to a few using non-default sync > mechanism. Securing git distribution is a whole different ballpark. > I'll agree that it is different, but we're talking about verification of the HEAD signature by infra, not verification of individual developer keys, which was the topic of the recent thread. Verification is already built-into portage for git syncing (but off by default). The problem is that portage will still checkout the tree if it fails verification. The patch is to do the verification before checking it out so that if it fails the tree is left in a last-known-good state (at least as seen by tools at the filesystem level - the fetched bad commits would still be visible to git). -- Rich
Re: [gentoo-dev] News Item: Portage rsync hardlink support
W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian Fiskerstrand napisał: > On 07/08/2018 08:53 AM, Michał Górny wrote: > > Is safe git syncing implemented already? If not, maybe finish it first and > > cover both with a single news item. Git is going to be more efficient here, > > so people may want to learn they have an alternative. > > Why complicate things, and increase wait for something that benefits > most users, just to give alternatives to a few using non-default sync > mechanism. Securing git distribution is a whole different ballpark. > Let me rephrase. Let's say I'm using rsync. This new feature is something positive but it breaks my use case (for one of the listed reasons -- overlayfs, inode use, small fs cache). After reading this news item, I learn that my only option is to disable the new feature. Now, I would appreciate being told that there's an alternate sync method that handles secure updates without having all those drawbacks. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 08:53 AM, Michał Górny wrote: > Is safe git syncing implemented already? If not, maybe finish it first and > cover both with a single news item. Git is going to be more efficient here, > so people may want to learn they have an alternative. Why complicate things, and increase wait for something that benefits most users, just to give alternatives to a few using non-default sync mechanism. Securing git distribution is a whole different ballpark. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 08:08 AM, Zac Medico wrote: > For example, if signature verification fails during a > sync operation, the new hardlink behavior will preserve the previous > state of the repository. This seems like a nice improvement, thank you for implementing it and making it the new default. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 02:28 AM, Toralf Förster wrote: > On 07/08/2018 08:08 AM, Zac Medico wrote: >> Please review. >> >> Title: Portage rsync hardlink support >> Author: Zac Medico >> Posted: 2018-07-11 >> Revision: 1 >> News-Item-Format: 2.0 >> Display-If-Installed: sys-apps/portage >> IMO there's another heads-up for users having an unsual configuration: > > So we do speak about files under /usr/portage itself and not about that > dir (==changing its inode number), right? > > B/c otherwise there's another heads-up for people bind-mounting > /usr/portage onto chrooted images. This case will continue to work because the download happens in a subdirectory, and after it's verified we call rsync again to apply the changes to the top-level directory. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 08/07/18 10:21, Zac Medico wrote: > On 07/08/2018 02:15 AM, Michał Górny wrote: >> >> Are you sure about that? That might have been the case so far but this >> hardlink tree may actually tip the balance. > Even if it takes twice a long (which it doesn't), the difference is > negligible for most people because they usually don't sync more than > once per day. It is recommended not to hammer the rsync mirrors with updates, and so this argument holds true for "most users". signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 08:08 AM, Zac Medico wrote: > Please review. > > Title: Portage rsync hardlink support > Author: Zac Medico > Posted: 2018-07-11 > Revision: 1 > News-Item-Format: 2.0 > Display-If-Installed: sys-apps/portage > IMO there's another heads-up for users having an unsual configuration: So we do speak about files under /usr/portage itself and not about that dir (==changing its inode number), right? B/c otherwise there's another heads-up for people bind-mounting /usr/portage onto chrooted images. -- Toralf PGP 23217DA7 9B888F45 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/08/2018 02:15 AM, Michał Górny wrote: > Dnia 8 lipca 2018 09:14:06 CEST, Zac Medico napisał(a): >> On 07/07/2018 11:53 PM, Michał Górny wrote: >>> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico >> napisał(a): Please review. Title: Portage rsync hardlink support Author: Zac Medico Posted: 2018-07-11 Revision: 1 News-Item-Format: 2.0 Display-If-Installed: sys-apps/portage For users of the rsync tree, beginning with sys-apps/portage-2.3.42, the default behavior for sync operations will use hardlinks in order to ensure that a repository remains in a valid state if something goes wrong [1]. For example, if signature verification fails during >> a sync operation, the new hardlink behavior will preserve the previous state of the repository. The new behavior may conflict with configurations that restrict the use of hardlinks, such as overlay filesystems. Therefore, users will have to set "sync-allow-hardlinks = no" in repos.conf if they have a configuration that restricts the use of hardlinks, but this should not be very common: [DEFAULT] sync-allow-hardlinks = no [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync --link-dest to implement atomic repository updates (and abort if signature verification fails) >>> >>> Is safe git syncing implemented already? If not, maybe finish it >> first and cover both with a single news item. Git is going to be more >> efficient here, so people may want to learn they have an alternative. >> >> Yeah there's already a patch for git sync [1] but I'd rather not make >> this news item more complicated than it needs to be. I wouldn't have >> bothered with a news item except that I want to give people some >> warning >> in case they are using overlayfs [2]. I think the efficiency difference >> between rsync and git here are pretty negligible for most people. > > Are you sure about that? That might have been the case so far but this > hardlink tree may actually tip the balance. Even if it takes twice a long (which it doesn't), the difference is negligible for most people because they usually don't sync more than once per day. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
Dnia 8 lipca 2018 09:14:06 CEST, Zac Medico napisał(a): >On 07/07/2018 11:53 PM, Michał Górny wrote: >> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico >napisał(a): >>> Please review. >>> >>> Title: Portage rsync hardlink support >>> Author: Zac Medico >>> Posted: 2018-07-11 >>> Revision: 1 >>> News-Item-Format: 2.0 >>> Display-If-Installed: sys-apps/portage >>> >>> For users of the rsync tree, beginning with sys-apps/portage-2.3.42, >>> the default behavior for sync operations will use hardlinks in order >>> to ensure that a repository remains in a valid state if something >>> goes wrong [1]. For example, if signature verification fails during >a >>> sync operation, the new hardlink behavior will preserve the previous >>> state of the repository. >>> >>> The new behavior may conflict with configurations that restrict the >>> use of hardlinks, such as overlay filesystems. Therefore, users will >>> have to set "sync-allow-hardlinks = no" in repos.conf if they have >>> a configuration that restricts the use of hardlinks, but this should >>> not be very common: >>> >>> [DEFAULT] >>> sync-allow-hardlinks = no >>> >>> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync >>>--link-dest to implement atomic repository updates (and abort if >>>signature verification fails) >> >> Is safe git syncing implemented already? If not, maybe finish it >first and cover both with a single news item. Git is going to be more >efficient here, so people may want to learn they have an alternative. > >Yeah there's already a patch for git sync [1] but I'd rather not make >this news item more complicated than it needs to be. I wouldn't have >bothered with a news item except that I want to give people some >warning >in case they are using overlayfs [2]. I think the efficiency difference >between rsync and git here are pretty negligible for most people. Are you sure about that? That might have been the case so far but this hardlink tree may actually tip the balance. > >[1] https://bugs.gentoo.org/660372 >[2] >https://www.brunsware.de/blog/gentoo/portage-tree-squashfs-overlayfs.html -- Best regards, Michał Górny (by phone)
Re: [gentoo-dev] News Item: Portage rsync hardlink support
On 07/07/2018 11:53 PM, Michał Górny wrote: > Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico napisał(a): >> Please review. >> >> Title: Portage rsync hardlink support >> Author: Zac Medico >> Posted: 2018-07-11 >> Revision: 1 >> News-Item-Format: 2.0 >> Display-If-Installed: sys-apps/portage >> >> For users of the rsync tree, beginning with sys-apps/portage-2.3.42, >> the default behavior for sync operations will use hardlinks in order >> to ensure that a repository remains in a valid state if something >> goes wrong [1]. For example, if signature verification fails during a >> sync operation, the new hardlink behavior will preserve the previous >> state of the repository. >> >> The new behavior may conflict with configurations that restrict the >> use of hardlinks, such as overlay filesystems. Therefore, users will >> have to set "sync-allow-hardlinks = no" in repos.conf if they have >> a configuration that restricts the use of hardlinks, but this should >> not be very common: >> >> [DEFAULT] >> sync-allow-hardlinks = no >> >> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync >>--link-dest to implement atomic repository updates (and abort if >>signature verification fails) > > Is safe git syncing implemented already? If not, maybe finish it first and > cover both with a single news item. Git is going to be more efficient here, > so people may want to learn they have an alternative. Yeah there's already a patch for git sync [1] but I'd rather not make this news item more complicated than it needs to be. I wouldn't have bothered with a news item except that I want to give people some warning in case they are using overlayfs [2]. I think the efficiency difference between rsync and git here are pretty negligible for most people. [1] https://bugs.gentoo.org/660372 [2] https://www.brunsware.de/blog/gentoo/portage-tree-squashfs-overlayfs.html -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News Item: Portage rsync hardlink support
Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico napisał(a): >Please review. > >Title: Portage rsync hardlink support >Author: Zac Medico >Posted: 2018-07-11 >Revision: 1 >News-Item-Format: 2.0 >Display-If-Installed: sys-apps/portage > >For users of the rsync tree, beginning with sys-apps/portage-2.3.42, >the default behavior for sync operations will use hardlinks in order >to ensure that a repository remains in a valid state if something >goes wrong [1]. For example, if signature verification fails during a >sync operation, the new hardlink behavior will preserve the previous >state of the repository. > >The new behavior may conflict with configurations that restrict the >use of hardlinks, such as overlay filesystems. Therefore, users will >have to set "sync-allow-hardlinks = no" in repos.conf if they have >a configuration that restricts the use of hardlinks, but this should >not be very common: > >[DEFAULT] >sync-allow-hardlinks = no > >[1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync >--link-dest to implement atomic repository updates (and abort if >signature verification fails) Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative. -- Best regards, Michał Górny (by phone)
[gentoo-dev] News Item: Portage rsync hardlink support
Please review. Title: Portage rsync hardlink support Author: Zac Medico Posted: 2018-07-11 Revision: 1 News-Item-Format: 2.0 Display-If-Installed: sys-apps/portage For users of the rsync tree, beginning with sys-apps/portage-2.3.42, the default behavior for sync operations will use hardlinks in order to ensure that a repository remains in a valid state if something goes wrong [1]. For example, if signature verification fails during a sync operation, the new hardlink behavior will preserve the previous state of the repository. The new behavior may conflict with configurations that restrict the use of hardlinks, such as overlay filesystems. Therefore, users will have to set "sync-allow-hardlinks = no" in repos.conf if they have a configuration that restricts the use of hardlinks, but this should not be very common: [DEFAULT] sync-allow-hardlinks = no [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync --link-dest to implement atomic repository updates (and abort if signature verification fails) -- Thanks, Zac
