Re: [gentoo-dev] News item review: SquashDelta syncing support
* Michał Górny schrieb am 17.05.15 um 05:47 Uhr: Dnia 2015-05-16, o godz. 20:38:36 Michael Orlitzky napisał(a): On 05/16/2015 06:01 PM, Michał Górny wrote: >> >> We have gentoo-announce@g.o and gentoo-user@g.o too! > > That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is > doubtedly used by sysadmins. > This one: https://archives.gentoo.org/gentoo-announce/ First time I hear about it. Looks to be used primarily for GLSAs. I wonder if anyone actually uses it. We use it to watch GLSA announcements. -Marc -- 0x35A64134 - 8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134 signature.asc Description: Digital signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
On 05/16/2015 11:47 PM, Michał Górny wrote: >> >> This one: >> >> https://archives.gentoo.org/gentoo-announce/ > > First time I hear about it. Looks to be used primarily for GLSAs. > I wonder if anyone actually uses it. > For what it's worth, it's one of the three lists mentioned specifically on our new "mailing lists" page: https://www.gentoo.org/get-involved/mailing-lists/
Re: [gentoo-dev] News item review: SquashDelta syncing support
Michał Górny wrote: > Dnia 2015-05-16, o godz. 20:38:36 > Michael Orlitzky napisał(a): > >> On 05/16/2015 06:01 PM, Michał Górny wrote: We have gentoo-announce@g.o and gentoo-user@g.o too! >>> That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is >>> doubtedly used by sysadmins. >>> >> This one: >> >> https://archives.gentoo.org/gentoo-announce/ > First time I hear about it. Looks to be used primarily for GLSAs. > I wonder if anyone actually uses it. > I subscribe to it and wish more things, new features for example, would be posted there. Sometimes portage has a new feature and unless one reads the man page, one has no clue. Of course, my wish is not limited to portage's new features. Other new Gentoo things would be nice too. Dale :-) :-)
Re: [gentoo-dev] News item review: SquashDelta syncing support
On Sun, 17 May 2015 00:00:22 +0200 Michał Górny wrote: > No, we don't. But news items are good way to tell people that > something is happening. Look at Funtoo. Their users are happy because > developers announce stuff rather than expecting them to watch for > random things happening. When we designed news items, we deliberately said they weren't for doing this kind of thing. News items are for high impact changes that require user action. Use the regular PR channels for showing off. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
Dnia 2015-05-16, o godz. 20:38:36 Michael Orlitzky napisał(a): > On 05/16/2015 06:01 PM, Michał Górny wrote: > >> > >> We have gentoo-announce@g.o and gentoo-user@g.o too! > > > > That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is > > doubtedly used by sysadmins. > > > > This one: > > https://archives.gentoo.org/gentoo-announce/ First time I hear about it. Looks to be used primarily for GLSAs. I wonder if anyone actually uses it. -- Best regards, Michał Górny pgpSWhadLDH5Y.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
On 05/16/2015 06:00 PM, Michał Górny wrote: >> >> Do we need a news item for this at all? Everything is backward >> compatible and most people don't need to do anything at all in response >> to the news. > > No, we don't. But news items are good way to tell people that something > is happening. Look at Funtoo. Their users are happy because developers > announce stuff rather than expecting them to watch for random things > happening. > Ok, just keep in mind that you force thousands of people to type `eselect news read new` on all of their machines when you post a feature announcement.
Re: [gentoo-dev] News item review: SquashDelta syncing support
On 05/16/2015 06:01 PM, Michał Górny wrote: >> >> We have gentoo-announce@g.o and gentoo-user@g.o too! > > That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is > doubtedly used by sysadmins. > This one: https://archives.gentoo.org/gentoo-announce/
Re: [gentoo-dev] News item review: SquashDelta syncing support
Dnia 2015-05-16, o godz. 23:48:01 Alon Bar-Lev napisał(a): > On 15 May 2015 at 17:51, Michał Górny wrote: > > Please note that the current syncing code does not verify the OpenPGP > > signature to confirm the authenticity of fetched snapshots and deltas. > > This feature will be added as soon as gentoo-keys support in Portage is > > available. > > These are great news! > We can retire the webrsync. > Why not sign it similar to the portage snapshot are signed for now? > The webrsync signature validation is quite simple. All signing is in place already for a long time. Just the verification code is missing, and it wasn't added because I was told to wait for gentoo-keys. > Just a reminder: please note the rollback prevention mechanism in > webrsync, it is not enough to check signature, but also prevent older > snapshot to be used. Truth be told, the squashdelta syncing wasn't really made with rollback prevention in mind. I can't think immediately of any solution that would prevent accidental rollback while preserving the intended flexibility. -- Best regards, Michał Górny pgptMxdy9Z1rN.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
Dnia 2015-05-16, o godz. 10:59:20 Michael Orlitzky napisał(a): > On 05/15/2015 10:51 AM, Michał Górny wrote: > > Title: SquashDelta syncing support > > Do we need a news item for this at all? Everything is backward > compatible and most people don't need to do anything at all in response > to the news. No, we don't. But news items are good way to tell people that something is happening. Look at Funtoo. Their users are happy because developers announce stuff rather than expecting them to watch for random things happening. -- Best regards, Michał Górny pgpIFr_u7xIS3.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
Dnia 2015-05-15, o godz. 08:23:27 Dirkjan Ochtman napisał(a): > On Fri, May 15, 2015 at 7:51 AM, Michał Górny wrote: > > Starting with Portage 2.2.19, a new SquashDelta syncing method has been > > introduced. It is meant to provide lightweight and efficient solution > > for stable systems. The whole repository is contained within a single > > pre-generated SquashFS image file. The daily snapshot of the repository > > is first fetched from the mirrors, and afterwards updated in-place using > > deltas (without repacking). > > This sounds nice, but the news item currently leaves me wondering what > sort of improvements I should expect. It says the new method is > "lightweight and efficient", but it would be nice to quantify this a > little bit, or add a link to a page with more details. I think the > default sync method in the handbook up to now has always been rsync? A > comparison (both in terms of upside and in terms of downside) would be > nice. Also, whether we want to make this the new default at some > point, and if so, when. I'll rewrite the beginning to be more clear on advantages when I have some time for it. -- Best regards, Michał Górny pgpL0KRIMcGUv.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
Dnia 2015-05-16, o godz. 15:23:24 Michael Orlitzky napisał(a): > On 05/16/2015 03:18 PM, Pacho Ramos wrote: > > > > Maybe this could be announced in planet.gentoo.org or even main > > gentoo.org news ;) > > > > > > We have gentoo-announce@g.o and gentoo-user@g.o too! That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is doubtedly used by sysadmins. -- Best regards, Michał Górny pgpRr2oid3rbr.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] News item review: SquashDelta syncing support
On 15 May 2015 at 17:51, Michał Górny wrote: > Please note that the current syncing code does not verify the OpenPGP > signature to confirm the authenticity of fetched snapshots and deltas. > This feature will be added as soon as gentoo-keys support in Portage is > available. These are great news! We can retire the webrsync. Why not sign it similar to the portage snapshot are signed for now? The webrsync signature validation is quite simple. Just a reminder: please note the rollback prevention mechanism in webrsync, it is not enough to check signature, but also prevent older snapshot to be used. Regards, Alon
Re: [gentoo-dev] News item review: SquashDelta syncing support
On Sat, 16 May 2015 15:23:24 -0400 Michael Orlitzky wrote: > On 05/16/2015 03:18 PM, Pacho Ramos wrote: > > > > Maybe this could be announced in planet.gentoo.org or even main > > gentoo.org news ;) > > > > > > We have gentoo-announce@g.o and gentoo-user@g.o too! > > But it will have to wait. There is a few bugs in the commit made to portage for the release. It doesn't run. It is fixed in portage- though. We'll see if we can get it gpg verifying with gkeys for another release. -- Brian Dolbec
Re: [gentoo-dev] News item review: SquashDelta syncing support
On 05/16/2015 03:18 PM, Pacho Ramos wrote: > > Maybe this could be announced in planet.gentoo.org or even main > gentoo.org news ;) > > We have gentoo-announce@g.o and gentoo-user@g.o too!
Re: [gentoo-dev] News item review: SquashDelta syncing support
El sáb, 16-05-2015 a las 20:38 +0200, Alexis Ballier escribió: > On Sat, 16 May 2015 10:59:20 -0400 > Michael Orlitzky wrote: > > > On 05/15/2015 10:51 AM, Michał Górny wrote: > > > Title: SquashDelta syncing support > > > > Do we need a news item for this at all? Everything is backward > > compatible and most people don't need to do anything at all in > > response to the news. > > I tend to agree here; while the feature looks cool, I'm not sure news > items purpose is to announce new features. > Maybe this could be announced in planet.gentoo.org or even main gentoo.org news ;)
Re: [gentoo-dev] News item review: SquashDelta syncing support
On Sat, 16 May 2015 10:59:20 -0400 Michael Orlitzky wrote: > On 05/15/2015 10:51 AM, Michał Górny wrote: > > Title: SquashDelta syncing support > > Do we need a news item for this at all? Everything is backward > compatible and most people don't need to do anything at all in > response to the news. I tend to agree here; while the feature looks cool, I'm not sure news items purpose is to announce new features.
Re: [gentoo-dev] News item review: SquashDelta syncing support
On 05/15/2015 10:51 AM, Michał Górny wrote: > Title: SquashDelta syncing support Do we need a news item for this at all? Everything is backward compatible and most people don't need to do anything at all in response to the news.
Re: [gentoo-dev] News item review: SquashDelta syncing support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 15/05/15 03:33 PM, Rich Freeman wrote: > On Fri, May 15, 2015 at 2:32 PM, Ian Stakenvicius > wrote: >> >> The new item doesn't really cover this much -- that the feature >> is for supporting storage and synchronization of the gentoo repo >> on squashfs rather than on a regular filesystem. Perhaps it >> would be enough to link to an article describing the benefits of >> using a squashfs'ed portage tree, so users could chose whether >> they want this or not based on that? Similarly, it would >> probably be good to mention that this new feature deprecates >> squash_portage and the other tools/methods out there for doing >> the same thing locally. >> > > That makes sense to me. Some of the likely benefits would be: > > 1. Less disk space use. 2. Vastly less inode use. 3. Much less > CPU/IO to update. 4. I suspect much less fragmentation/write/etc > for storage on flash. Then again, on filesystems like btrfs > fragmentation might be worse due to all the internal writes. 5. > Probably better read performance (less disk IO, more CPU). > > Downsides include: 1. No way to sync more frequently than whatever > the update cycle is. It would be more like emerge-webrsync and less > like emerge --sync. 2. Impossible to tweak ebuilds without setting > up an overlay. This might be annoying for devs/etc. > Given the importance of this is to me more about the squashfs storage than the sync method, it may even be pertinent to change the title of the news item to something like: "SquashFS repo, SquashDelta syncing support" -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlVWSygACgkQ2ugaI38ACPAXdgEApXmmfrFJB1b4L0B4hKnNAuLs Njl9rWczgmR4SjMgvBwA/AwIOujrtoiQd1iT4j9oqQAjYJ9S8O/vVJe/9yWJXpj/ =WI2a -END PGP SIGNATURE-
Re: [gentoo-dev] News item review: SquashDelta syncing support
On Fri, May 15, 2015 at 2:32 PM, Ian Stakenvicius wrote: > > The new item doesn't really cover this much -- that the feature is for > supporting storage and synchronization of the gentoo repo on squashfs > rather than on a regular filesystem. Perhaps it would be enough to > link to an article describing the benefits of using a squashfs'ed > portage tree, so users could chose whether they want this or not based > on that? Similarly, it would probably be good to mention that this > new feature deprecates squash_portage and the other tools/methods out > there for doing the same thing locally. > That makes sense to me. Some of the likely benefits would be: 1. Less disk space use. 2. Vastly less inode use. 3. Much less CPU/IO to update. 4. I suspect much less fragmentation/write/etc for storage on flash. Then again, on filesystems like btrfs fragmentation might be worse due to all the internal writes. 5. Probably better read performance (less disk IO, more CPU). Downsides include: 1. No way to sync more frequently than whatever the update cycle is. It would be more like emerge-webrsync and less like emerge --sync. 2. Impossible to tweak ebuilds without setting up an overlay. This might be annoying for devs/etc. -- Rich
Re: [gentoo-dev] News item review: SquashDelta syncing support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 15/05/15 12:15 PM, Diamond wrote: > On Fri, 15 May 2015 08:23:27 -0700 Dirkjan Ochtman > wrote: > >> On Fri, May 15, 2015 at 7:51 AM, Michał Górny >> wrote: >>> Starting with Portage 2.2.19, a new SquashDelta syncing method >>> has been introduced. It is meant to provide lightweight and >>> efficient solution for stable systems. The whole repository is >>> contained within a single pre-generated SquashFS image file. >>> The daily snapshot of the repository is first fetched from the >>> mirrors, and afterwards updated in-place using deltas (without >>> repacking). >> >> This sounds nice, but the news item currently leaves me wondering >> what sort of improvements I should expect. [...] > > I've read the pdf article of Michał Górny and from my expirience > with emerge-delta-webrsync and app-portage/getdelta in the past > this good old new feature looks mostly useful for bad Internet > connections (too slow or too expensive ones) and looks mostly > useless for syncing relative to rsync method from local mirror like > I use [...] Although this thread should be a review of the news item rather than a review of the feature, I think both of these guys have a point. The main benefit to this new feature is that it allows users to use a squashfs image for their gentoo repo (portage tree) without having to (re)generate it themselves locally every time they --sync, AND without having to re-download an entire image from gentoo mirrors each time either. The new item doesn't really cover this much -- that the feature is for supporting storage and synchronization of the gentoo repo on squashfs rather than on a regular filesystem. Perhaps it would be enough to link to an article describing the benefits of using a squashfs'ed portage tree, so users could chose whether they want this or not based on that? Similarly, it would probably be good to mention that this new feature deprecates squash_portage and the other tools/methods out there for doing the same thing locally. -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlVWO9kACgkQ2ugaI38ACPA1vwD9ELIdOgSSTfly3rT5zU6dzhGb 62LtN8loiRFhKfyAe/8A/24xw95j7qav/himVRA5OOjM3qTE++iBY/2yXPgLWpI5 =1+2r -END PGP SIGNATURE-
Re: [gentoo-dev] News item review: SquashDelta syncing support
On Fri, 15 May 2015 08:23:27 -0700 Dirkjan Ochtman wrote: > On Fri, May 15, 2015 at 7:51 AM, Michał Górny > wrote: > > Starting with Portage 2.2.19, a new SquashDelta syncing method has > > been introduced. It is meant to provide lightweight and efficient > > solution for stable systems. The whole repository is contained > > within a single pre-generated SquashFS image file. The daily > > snapshot of the repository is first fetched from the mirrors, and > > afterwards updated in-place using deltas (without repacking). > > This sounds nice, but the news item currently leaves me wondering what > sort of improvements I should expect. It says the new method is > "lightweight and efficient", but it would be nice to quantify this a > little bit, or add a link to a page with more details. I think the > default sync method in the handbook up to now has always been rsync? A > comparison (both in terms of upside and in terms of downside) would be > nice. Also, whether we want to make this the new default at some > point, and if so, when. > > Cheers, > > Dirkjan > > I've read the pdf article of Michał Górny and from my expirience with emerge-delta-webrsync and app-portage/getdelta in the past this good old new feature looks mostly useful for bad Internet connections (too slow or too expensive ones) and looks mostly useless for syncing relative to rsync method from local mirror like I use http://mirror.yandex.ru/gentoo-distfiles/ from my local region. eix-sync gave me the following statistics (before introducing new portage sync with repos.conf wich has stopped upgrade in the middle atm because >=app-portage/layman-2.3.0 haven't been stabilised yet): * Time statistics: 19 seconds for syncing 17 seconds for eix-update 1 seconds for eix-diff 51 seconds total or this one the other day: * Time statistics: 37 seconds for syncing 11 seconds for eix-update 1 seconds for eix-diff 67 seconds total So it takes usually 15-40 seconds for syncing using usual rsync method. This deltas have their own drawbacks like "delta is under generation, please wait half an hour or even more" or "your state is not the same what was while generating delta on the host and lets do additional work with more deltas". )) Although, nice try with experimenting and trying to improve sync mechanism. )
Re: [gentoo-dev] News item review: SquashDelta syncing support
On Fri, May 15, 2015 at 7:51 AM, Michał Górny wrote: > Starting with Portage 2.2.19, a new SquashDelta syncing method has been > introduced. It is meant to provide lightweight and efficient solution > for stable systems. The whole repository is contained within a single > pre-generated SquashFS image file. The daily snapshot of the repository > is first fetched from the mirrors, and afterwards updated in-place using > deltas (without repacking). This sounds nice, but the news item currently leaves me wondering what sort of improvements I should expect. It says the new method is "lightweight and efficient", but it would be nice to quantify this a little bit, or add a link to a page with more details. I think the default sync method in the handbook up to now has always been rsync? A comparison (both in terms of upside and in terms of downside) would be nice. Also, whether we want to make this the new default at some point, and if so, when. Cheers, Dirkjan
[gentoo-dev] News item review: SquashDelta syncing support
Title: SquashDelta syncing support Author: Michał Górny Content-Type: text/plain Posted: 2015-05-xx Revision: 1 News-Item-Format: 1.0 Display-If-Installed: >=sys-apps/portage-2.2.19 Starting with Portage 2.2.19, a new SquashDelta syncing method has been introduced. It is meant to provide lightweight and efficient solution for stable systems. The whole repository is contained within a single pre-generated SquashFS image file. The daily snapshot of the repository is first fetched from the mirrors, and afterwards updated in-place using deltas (without repacking). In order to enable SquashDelta syncing, please install dev-util/squashmerge utility first: $ emerge -v dev-util/squashmerge Afterwards, you can use a repos.conf entry similar to the following: [gentoo] location = /var/db/repos/gentoo sync-type = squashdelta sync-uri = mirror://gentoo/../snapshots/squashfs During the initial sync, Portage will fetch the current Gentoo SquashFS snapshot from the mirrors (~105M). For the synces following, it will only fetch a single delta, and use squashmerge to quickly update the local copy. If possible, Portage will automatically mount (or remount) the SquashFS after syncing. However, you may want to add an explicit /etc/fstab entry for the filesystem to make it available before invoking 'emerge --sync': /var/cache/portage/squashfs/gentoo-current.sqfs /var/db/repos/gentoo \ squashfs defaults 0 0 Please note that the current syncing code does not verify the OpenPGP signature to confirm the authenticity of fetched snapshots and deltas. This feature will be added as soon as gentoo-keys support in Portage is available. -- Best regards, Michał Górny pgpaH0O8FrL2X.pgp Description: OpenPGP digital signature
