Re: [gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Alec Warner]

On Fri, Jan 17, 2014 at 11:10 PM, Alan McKinnon wrote:

> On 18/01/2014 09:04, Patrick Lauer wrote:
> >> which could link to the
> >> > infra page would be good here perhaps, so when an outage occurred ( at
> >> > least on the web side ) appropriate links to infra could be given.
> > The more sane fix would be low DNS TTL and rotating DNS to a different
> > IP if things are down.
> >
> >
>
>
> That is already in place:
>
>  $ dig overlays.gentoo.org
>
> ; <<>> DiG 9.9.4 <<>> overlays.gentoo.org
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49989
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4000
> ;; QUESTION SECTION:
> ;overlays.gentoo.org.   IN  A
>
> ;; ANSWER SECTION:
> overlays.gentoo.org.600 IN  CNAME   spoonbill.gentoo.org.
> spoonbill.gentoo.org.   604800  IN  A   81.93.255.5
>
>
>
> 5 minutes downtime max if a switch needs to be done.
> 5 minutes is perfectly acceptable IMHO
>

infra TTL standards are 60 minutes for service CNAMEs and 30 minutes for HA
CNAMES. The TTL is 600 here (which is 10 minutes, not 5) because I lowered
it on 1/14 in anticipation of a machine failover, it was previously 2 hours.

-A


>
>
> --
> Alan McKinnon
> alan.mckin...@gmail.com
>
>
>

Re: [gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Alan McKinnon]

On 18/01/2014 09:04, Patrick Lauer wrote:
>> which could link to the
>> > infra page would be good here perhaps, so when an outage occurred ( at
>> > least on the web side ) appropriate links to infra could be given.
> The more sane fix would be low DNS TTL and rotating DNS to a different
> IP if things are down.
> 
> 


That is already in place:

 $ dig overlays.gentoo.org

; <<>> DiG 9.9.4 <<>> overlays.gentoo.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49989
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;overlays.gentoo.org.   IN  A

;; ANSWER SECTION:
overlays.gentoo.org.600 IN  CNAME   spoonbill.gentoo.org.
spoonbill.gentoo.org.   604800  IN  A   81.93.255.5



5 minutes downtime max if a switch needs to be done.
5 minutes is perfectly acceptable IMHO



-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Patrick Lauer]

On 01/18/2014 01:23 PM, Kent Fredric wrote:
> 
> On 18 January 2014 18:02, Robin H. Johnson  > wrote:
> 
> - More people need to use the infra-status page to learn about the state
>   of Gentoo services.
> 
> 
> 
> A service middle layer like fastly or cloudflare
No,

These services break SSL and most of the time just lead to redirection
loops and other "fun".

Also ClownFlair doesn't work properly without Javascript ...

So it'd actually reduce availability and the size of our wallet, which
doesn't sound like a good strategy to me.

> which could link to the
> infra page would be good here perhaps, so when an outage occurred ( at
> least on the web side ) appropriate links to infra could be given.
The more sane fix would be low DNS TTL and rotating DNS to a different
IP if things are down.


> And the infra status page is not exactly obvious. Its not listed on the
> "gentoo sites" list on the top right, and perhaps it aught to be.

Yes, that might be quite nice (plus maybe posting longer outages in a
way that they are visible on the frontpage?)

Have fun,

Patrick

Re: [gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Alec Warner]

On Fri, Jan 17, 2014 at 9:02 PM, Robin H. Johnson wrote:

> overlays.gentoo.org service has been restored on a new system.
> Some statistics and a post-mortem follow.
>
> Special thanks to antarus and a3li for all their interactions with our
> sponsor,
> and managing most of the details. I just did the final data recovery and
> this
> writeup.
>
> Please resume using the service, and if you see something weird that you
> think is different from before, please file a bug for Infrastructure.
>
> In the process, the service moved to a new machine. The SSH keys have
> changed
> as follows:
> DSA: d6:71:99:1f:46:c9:42:95:e1:9d:be:8e:f7:76:51:b5
> RSA: 92:b5:40:16:63:a3:61:9f:d7:63:64:ba:d5:51:41:b9
> ECDSA: 96:f0:29:e6:d4:85:58:46:31:ba:0e:17:0b:8c:fa:d8
>
> As this time, we will NOT be restoring Trac due to low demand. If you
> still require an web-based SVN browser for old SVN repos, please contact
> us at in...@gentoo.org.
>

For Trac wiki users. The recommendation is to move to wiki.gentoo.org. If
you hadn't migrated, and you need a copy of your Trac wiki pages from
overlays.gentoo.org, please file a bug against infra and someone (me) will
restore them for on a request by request basis. I think the deal is that I
can pretty trivially give you a tarball of markup files (one per wiki page.)

-A


>
> If you have a dev/ repo under the list 'IMPORTANT' below, you MUST push
> to the server again.
>
> IMPORTANT: The following repos were damaged beyond repair, and were not
> available in backups. You'll need to push again, I have reset the repos to
> empty:
> dev/anarchy.git
> dev/dberkholz.git
> dev/dev-zero.git
> dev/dilfridge.git
> dev/fordfrog.git
> dev/graaff.git
> dev/maekke.git
> dev/mschiff.git
> dev/quantumsummers.git
> dev/zorry.git
>
> FYI: The following repos appeared to be empty:
> dev/b33fc0d3.git
> dev/moult.git
> dev/tomwij.git
> user/blueicefield.git
> user/disinbox.git
> user/palatis.git
> user/paragon.git
> user/vmalov.git
> user/xray.git
>
> FYI: The following repos contained dangling commits/tags/blobs, and this
> should not be considered new breakage; if you have a newer copy, you are
> encouraged to push again:
> dev/blueness.git
> dev/maksbotan.git
> dev/mgorny.git
> dev/qiaomuf.git
> dev/xmw.git
> proj/betagarden.git
> proj/catalyst.git (+tags)
> proj/devmanual.git
> proj/dotnet.git
> proj/elfix.git (+tags)
> proj/emacs-tools.git
> proj/gamerlay.git
> proj/hardened-dev.git
> proj/hardened-patchset.git
> proj/kde.git
> proj/lisp.git
> proj/openrc.git (+tags)
> proj/portage.git
> proj/ruby-overlay.git
> proj/sci.git
> proj/sunrise.git
> proj/webapp-config.git
> proj/x11.git
> user/gmt.git
> user/mv.git (+blobs)
> user/palmer.git
>
> Statistics:
> ---
>   354 repos total
> -  10 repos unrecoverable (all in /dev)
> = 344 repos recovered/available
>
> 9 repos that seem to empty
>26 repos with dangling commits/tags/blobs
> 2 repos recovered from external sources.
>
> Breakdown by path:
> --
> 193 proj/ repos
>  69 dev/  repos
>  91 user/ repos
>   1 other repo
>
> Post-mortem
> ---
> Hornbill went offline around: 2014-01-10 13:13 UTC
> Hornbill last started a backup of VCS: 2014-01-10 07:59:04 UTC
> Hornbill last completed a backup of VCS: 2014-01-10 08:20:54 UTC
>
> Between the backup starting, and the server going offline, we were able
> to confirm writes to the following Git repos:
> dev/fordfrog.git
> proj/kde.git
> gitolite-admin.git
>
> We believe that there were no writes to user/ repos, but are not 100%
> certain, as the logging was insufficient for this purpose.
>
> Hornbill went offline just over a week ago: Mid-afternoon on a Friday
> for the timezone where it's located. Due staff turnover and business
> changes at the previous sponsor, we were not able to contact anybody
> until regular office hours on Monday, January 13th.
>
> The server in question, while previously functioning, was not
> recoverable after a remote hands reboot on Monday afternoon (UTC).
> On Tuesday, more the sponsor was able to examine in it more depth, and
> it was not recoverable. More concealingly, it turned out to be one of
> the few remaining Gentoo infrastructure systems with IDE drives. The
> data was recovered, however it seemed to have a lot of corruption.
>
> It was noted that our backups were missing all of the dev/ repos, due to
> a system-wide rule to exclude /dev/ from backups (the rule should only
> be the real /dev, not any directory simply named "dev"). For this
> reason, we decided to try and get the data from the old server.
>
> Verification/recovery of the remaining data was also hampered by
> confirming that some of the Git repos in the backup were not entirely
> clean, containing legacy errors that turned out to be false positives
> from their CVS/SVN conversions, or dangling commits/blobs/tags.
>
> What could we do better next time:
> --
> - Have backups of all repos!
> - Compare the age of the backup immediately, a

Re: [gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Alec Warner]

On Fri, Jan 17, 2014 at 9:23 PM, Kent Fredric  wrote:

>
> On 18 January 2014 18:02, Robin H. Johnson  wrote:
>
>> - More people need to use the infra-status page to learn about the state
>>   of Gentoo services.
>>
>
>
> A service middle layer like fastly or cloudflare which could link to the
> infra page would be good here perhaps, so when an outage occurred ( at
> least on the web side ) appropriate links to infra could be given.
>

Cloudly stuff aside (most of infra is not super experienced or trusting of
cloud stuff) I think there was a lot of indecision during the outage.
Do we wait for the sponsor or restore from backup?
How good are the backups (turns out, they were decent?)
How much work is it to rebuild from them (turns out, one evening of Robin's
time + incidentals.)

Once we got the data back on the new machine, why did we post the all
clear? Then we knew there was corruption, but it took a long time to
disable git and http access. Some repos were missing, some were corrupt,
etc.

We don't have procedures for these sorts of things. I think we were
conservative in the changes we made. How do you disable a service like
gitolite? We deployed two fixes. One was to disable ssh for the 'git' user,
the second was to move the authorized keys files out of the way. We pursued
these avenues independently, and we did not check them into configuration
management, which I wish had happened. Later when we disabled the http part
(to make overlays throw 503's) that was checked in, which was nice.
Certainly I was afraid of breaking stuff for Robin, so I really tried to
avoid doing anything unless I was confident it would not impact him.


> And the infra status page is not exactly obvious. Its not listed on the
> "gentoo sites" list on the top right, and perhaps it aught to be.
>

I consider the page a great success in this story. I'm really happy about
it, and while you can always say 'hey we could have done better here' I
think we did pretty well.

-A


>
>
>
>
> --
> Kent
>
> perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
> for ( 9,8,0,7,1,6,5,4,3,2 );"
>
> http://kent-fredric.fox.geek.nz
>

Re: [gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Kent Fredric]

On 18 January 2014 18:02, Robin H. Johnson  wrote:

> - More people need to use the infra-status page to learn about the state
>   of Gentoo services.
>


A service middle layer like fastly or cloudflare which could link to the
infra page would be good here perhaps, so when an outage occurred ( at
least on the web side ) appropriate links to infra could be given.

And the infra status page is not exactly obvious. Its not listed on the
"gentoo sites" list on the top right, and perhaps it aught to be.



-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[gentoo-dev] overlays.gentoo.org restoration & post-mortem

[Robin H. Johnson]

overlays.gentoo.org service has been restored on a new system.  
Some statistics and a post-mortem follow.

Special thanks to antarus and a3li for all their interactions with our sponsor,
and managing most of the details. I just did the final data recovery and this
writeup.

Please resume using the service, and if you see something weird that you
think is different from before, please file a bug for Infrastructure.

In the process, the service moved to a new machine. The SSH keys have changed
as follows:
DSA: d6:71:99:1f:46:c9:42:95:e1:9d:be:8e:f7:76:51:b5
RSA: 92:b5:40:16:63:a3:61:9f:d7:63:64:ba:d5:51:41:b9
ECDSA: 96:f0:29:e6:d4:85:58:46:31:ba:0e:17:0b:8c:fa:d8

As this time, we will NOT be restoring Trac due to low demand. If you
still require an web-based SVN browser for old SVN repos, please contact
us at in...@gentoo.org.

If you have a dev/ repo under the list 'IMPORTANT' below, you MUST push
to the server again.

IMPORTANT: The following repos were damaged beyond repair, and were not
available in backups. You'll need to push again, I have reset the repos to
empty:
dev/anarchy.git
dev/dberkholz.git
dev/dev-zero.git
dev/dilfridge.git
dev/fordfrog.git
dev/graaff.git
dev/maekke.git
dev/mschiff.git
dev/quantumsummers.git
dev/zorry.git

FYI: The following repos appeared to be empty:
dev/b33fc0d3.git
dev/moult.git
dev/tomwij.git
user/blueicefield.git
user/disinbox.git
user/palatis.git
user/paragon.git
user/vmalov.git
user/xray.git

FYI: The following repos contained dangling commits/tags/blobs, and this
should not be considered new breakage; if you have a newer copy, you are
encouraged to push again:
dev/blueness.git
dev/maksbotan.git
dev/mgorny.git
dev/qiaomuf.git
dev/xmw.git
proj/betagarden.git
proj/catalyst.git (+tags)
proj/devmanual.git
proj/dotnet.git
proj/elfix.git (+tags)
proj/emacs-tools.git
proj/gamerlay.git
proj/hardened-dev.git
proj/hardened-patchset.git
proj/kde.git
proj/lisp.git
proj/openrc.git (+tags)
proj/portage.git
proj/ruby-overlay.git
proj/sci.git
proj/sunrise.git
proj/webapp-config.git
proj/x11.git
user/gmt.git
user/mv.git (+blobs)
user/palmer.git

Statistics:
---
  354 repos total
-  10 repos unrecoverable (all in /dev)
= 344 repos recovered/available

9 repos that seem to empty
   26 repos with dangling commits/tags/blobs
2 repos recovered from external sources.

Breakdown by path:
--
193 proj/ repos
 69 dev/  repos
 91 user/ repos
  1 other repo

Post-mortem
---
Hornbill went offline around: 2014-01-10 13:13 UTC
Hornbill last started a backup of VCS: 2014-01-10 07:59:04 UTC
Hornbill last completed a backup of VCS: 2014-01-10 08:20:54 UTC

Between the backup starting, and the server going offline, we were able
to confirm writes to the following Git repos:
dev/fordfrog.git
proj/kde.git
gitolite-admin.git

We believe that there were no writes to user/ repos, but are not 100%
certain, as the logging was insufficient for this purpose.

Hornbill went offline just over a week ago: Mid-afternoon on a Friday
for the timezone where it's located. Due staff turnover and business
changes at the previous sponsor, we were not able to contact anybody
until regular office hours on Monday, January 13th.

The server in question, while previously functioning, was not
recoverable after a remote hands reboot on Monday afternoon (UTC).
On Tuesday, more the sponsor was able to examine in it more depth, and
it was not recoverable. More concealingly, it turned out to be one of
the few remaining Gentoo infrastructure systems with IDE drives. The
data was recovered, however it seemed to have a lot of corruption.

It was noted that our backups were missing all of the dev/ repos, due to
a system-wide rule to exclude /dev/ from backups (the rule should only
be the real /dev, not any directory simply named "dev"). For this
reason, we decided to try and get the data from the old server.

Verification/recovery of the remaining data was also hampered by
confirming that some of the Git repos in the backup were not entirely
clean, containing legacy errors that turned out to be false positives
from their CVS/SVN conversions, or dangling commits/blobs/tags.

What could we do better next time:
--
- Have backups of all repos!
- Compare the age of the backup immediately, and consider going live
  with the backup. Only 5 hours of work would have been lost, and even
  then possibly only temporarily, due to the distributed nature of Git.
- More people need to use the infra-status page to learn about the state
  of Gentoo services.

Actions for Infra
-
- Include dev/ repos were not in the backup
- Set up Gitolite mirroring
- Review gitolite logging (needs to be easier to confirm when writes
  took place)

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85


signature.asc
Description: Digital signature

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[heroxbd]

Alex Xu  writes:

> This doesn't really help. Consider that deltas require both a *start*
> and *end*. It's not as simple as adding it all up, since you would need
> a 19-3d, then 3-1d, then 1-0d.

In this case, 19-18d, 18-16d, 16-0d, instead. The largest delta is
always used for the most recent. That's why we need 8 pieces of 2d
delta and 16 pieces of 1d delta.


pgpkuCRmXKr7o.pgp
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[William Hubbs]

On Fri, Jan 17, 2014 at 04:02:27PM +0100, Michał Górny wrote:
> Dnia 2014-01-17, o godz. 14:02:51
> gro...@gentoo.org napisał(a):
> 
> > Maybe, a good solution is to introduce a special arch, "noarch", for such 
> > packages (similar to what's done in the rpm world). Then, if a package is 
> > ~noarch, it is automatically considered ~arch for all arches. Similar for 
> > stable. The maintainer should be able to keyword ~noarch and to stabilize 
> > noarch. Comments?
> 
> If you want to play with such a major change, you should start a new
> thread instead of starting it in the middle of this spam-art.
> Otherwise, many people will miss it and complain loudly afterwards.

I am going to agree with mgorny on this; highjacking this thread with
this change is not appropriate; all we were doing in this thread is
trying to figure out a way to improve our stabilization policy.

Introducing a "noarch" keyword should be discussed in a completely
separate thread.

William


signature.asc
Description: Digital signature

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Alex Xu]

On 17/01/14 08:08 PM, hero...@gentoo.org wrote:
> Michał Górny  writes:
> 
>> However, it may be actually beneficial to provide other durations, like
>> weekly deltas. In my tests, the daily updates for this week summed up
>> to almost 50M while the weekly was barely 20M.
> 
> Is there a way to merge the deltas without the original squashfs?

Uh... what? How would that work?

> how fast is the delta generation?

Very.

> It could be done on the server side on the fly and cached.

Too much work.


> Or we provide a set of 16,8,4,2,1 day deltas, then 
> 
>  16d: 1 piece needed
>  8d: 2 needed
>  4d: 4 needed
>  2d: 8 needed
>  1d: 16 needed
> 
> The total of 31 pieces will cover a month (31 days) with at most 5
> deltas to be downloaded.
> 
> e.g. If the system is 19 days old, then we download a 1d, 2d, and 16d.
> 

This doesn't really help. Consider that deltas require both a *start*
and *end*. It's not as simple as adding it all up, since you would need
a 19-3d, then 3-1d, then 1-0d.




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Thomas D.]

Hi,

Michał Górny wrote:
> Now, does anyone have an old portage-YYZZ.tar.{bz2,xz} snapshot? I
> need the official one from our mirrors, preferably 3-4 months old.






-- 
Regards,
Thomas

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[heroxbd]

Michał Górny  writes:

> However, it may be actually beneficial to provide other durations, like
> weekly deltas. In my tests, the daily updates for this week summed up
> to almost 50M while the weekly was barely 20M.

Is there a way to merge the deltas without the original squashfs? how
fast is the delta generation? It could be done on the server side on the
fly and cached.

Or we provide a set of 16,8,4,2,1 day deltas, then 

 16d: 1 piece needed
 8d: 2 needed
 4d: 4 needed
 2d: 8 needed
 1d: 16 needed

The total of 31 pieces will cover a month (31 days) with at most 5
deltas to be downloaded.

e.g. If the system is 19 days old, then we download a 1d, 2d, and 16d.

Cheers,
Benda


pgpS54FnX5ALN.pgp
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Manuel Rüger]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 01/17/2014 06:08 PM, gro...@gentoo.org wrote:
> On Fri, 17 Jan 2014, Tom Wijsman wrote:
>> On Fri, 17 Jan 2014 16:31:54 +0100 Ulrich Mueller
>>  wrote:
 On Fri, 17 Jan 2014, grozin  wrote:
 Maybe, a good solution is to introduce a special arch,
 "noarch", for such packages (similar to what's done in the
 rpm world). Then, if a package is ~noarch, it is
 automatically considered ~arch for all arches. Similar for
 stable. The maintainer should be able to keyword ~noarch and
 to stabilize noarch. Comments?
>>> 
>>> How would you handle dependencies in such a scenario? All
>>> dependencies must be keyworded or stable on all architectures,
>>> before the package can be keyworded or stabilised on noarch?
>> 
>> Maybe we can let the package managers only perceive it as
>> keyworded or stable if all of its dependencies are keyworded or
>> stable on the architecture that the user runs. Then we can have
>> repoman just ignore checking dependencies' keywords when we
>> keyword or stabilize them.
> Very reasonable.
> 
> Andrey
> 


I think the idea itself is good, but we should not add this to
KEYWORDS directly, as it might cause some problems with older package
managers(?).

A new variable can be introduced, which will overwrite testing
keywords to stable keywords, if the var is set to "stable" and keeps
everything in KEYWORDS marked as testing otherwise.

If this var exists in an ebuild and there is a new stabilization bug,
the arch team can decide if they want to mark it stable for all
architectures (via setting the var to stable) or only for the
architecture they tested it for (if some dependencies are missing on
other architectures).
This practice ensures that at least one arch team member of any arch
tested it.

The use of the to-be-added variable could also be extended for
vulnerability fixing.
It's more important to users to deal with less vulnerabilities for a
long time than a working stable dependency tree. Because the latter
got easier with the autounmask feature in portage.

If the var is set by the maintainer to "security-fix-$bugid" and the
users add an option to their profile, it automatically sets the ebuild
to stable and prompts an info with the bugid.
Users who do not want to wait for stabilization or GLSA have a better
possibility to secure their system earlier.
The advantage in general is that quickly added fixes get a wider
testing. So stable users will also profit.


Cheers

Manuel
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJ8BAEBCgBmBQJS2cwvXxSAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4MDA1RERERkM0ODM2QkE4MEY3NzY0N0M1
OEZCQTM2QzhEOUQ2MzVDAAoJEFj7o2yNnWNcpuwP/27np/ekt9AWHJz/OC7BpDz8
gxrE0iuocczV6m5/CejSY8GEdd26xQRdyloZ/7f74W1I5jRB/WPbHUKa0dnglZba
AG/WRJRYOao6537t5p9n6knH3Bj0wIcZl90Jox80HOXsvk/eBwZaliE+jcA+6Mat
Dcl0FVgl/b1WJZaa0aEt5st8nnW5TJ5ZTuWBG6e6shH94qAFcr8VWLOTtY1xqvHf
U1GHlynM4h+nX7BnDlhPxPf2l9XPBZNQFOAvWfvE341uZcSUpkQYfYzpo2TKdYop
oPL6hfMsHq5uggB0OvVaf4CiuCKhV3re7GVexKJE9Moigrb0v/nWCy5ApvR0Ww/N
7wozyGcWUKc/oHW3CHGAYr4wbFjjI9LjB5IVEjmEAGmJ5bq7/RViM+5slj/s9kBe
Vioti6i2vYVs4awm/HrMvremVUJ03Deh8uwVnOaggyrggRnwbxEsRxdsMCmYNkKN
2xAVvnSi2YEMSwBWvClRJwFvvrZzzsWd+t2Y/e/VqAFNvZn0H0lqZAP1+z540nzU
I7f2ym88jedwRJq41q6TgI9XaHlTFXLMcb9Hu19Xv/faUu5jHxg+ZvQtIwC/2YRy
6La1PGO9uk0wHOgixHe/bPXmZ2ArTHB6pAzgiLMpQxuahJhbGXiTmjM8qBc8IKD2
t0VP0WhLWZahQtQ21vrW
=UumH
-END PGP SIGNATURE-

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Tom Wijsman]

On Fri, 17 Jan 2014 18:28:41 +
Ciaran McCreesh  wrote:

> On Fri, 17 Jan 2014 17:47:58 +0100
> Tom Wijsman  wrote:
> > Maybe we can let the package managers only perceive it as keyworded
> > or stable if all of its dependencies are keyworded or stable on the
> > architecture that the user runs. Then we can have repoman just
> > ignore checking dependencies' keywords when we keyword or stabilize
> > them.
> > 
> > Not sure how implementable this idea is though...
> 
> It's going to hurt for four reasons that I can think of right now.
> 
> Firstly, things you think are "obviously portable" sometimes aren't.

We could write a search for architecture dependent / specific code.

> Secondly, users already get confused by "stable use masks". This is
> going to be even worse: users aren't going to understand why a noarch
> package isn't available for them.

We can improve the output of the package manager to make this easier
to understand; one way is to clarify it, the other way is to just
replace it by the actual archictecture the user runs.

As a side note, I think we might want to name this anyarch... :)

> Thirdly, you have to decide how to deal with long chains and cycles in
> noarch dependencies.
>
> Fourthly, the interaction with || deps is an awful mess.

Yes, these are though problems to resolve; my mind came up with that
this idea needs recursion, hence the "not sure how implementable".

A cycle can be broken by stopping to continue to a certain dependency
if that dependency is of the parent reverse dependencies, capture the
set of packages as a cycle. Then check for each package in the cycle if
their dependencies satisfy each package; if so, all the packages in
the cycle are satisfied.

Of course, this doesn't take into account more complex cycles were two
cycles are connected to each other; but if we would have such thing,
I'd rather see that get fixed in the Portage tree as that sounds like a
needlessly complex set of ebuilds.

Long chains might or might exist and might or might not be problematic,
that's something we would need to test out when this is implemented.

|| deps can be done, just check them in the same order like before;
what I'm more scared of is the amount of anyarch/noarch there would be
added to the Portage tree, just a few can be easily done.

If it is going to be a large share of the Portage tree we'll want to
look for another design or just not change how this works at all. 

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

Dnia 2014-01-17, o godz. 23:31:32
Duncan <1i5t5.dun...@cox.net> napisał(a):

> Michał Górny posted on Fri, 17 Jan 2014 20:30:00 +0100 as excerpted:
> 
> > Dnia 2014-01-17, o godz. 19:19:14 Duncan <1i5t5.dun...@cox.net>
> > napisał(a):
> > 
> >> Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
> >> 
> >> > 96M  portage-20140108.sqfs
> 
> >> > For deltas [...]
> >> > 
> >> > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> 
> >> > applying it takes ~2.5 second on my 2 GHz Athlon64.
> >> 
> >> diffs are ~1/16 the full squashfs size[.] So people updating once a
> >> week [or] 10 days would see a bandwidth savings, provided the sync
> >> script was intelligent enough to apply updates serially.
> >> 
> >> The breakover point would be roughly an update every two weeks, or
> >> twice a month
> > 
> > However, it may be actually beneficial to provide other durations, like
> > weekly deltas. In my tests, the daily updates for this week summed up to
> > almost 50M while the weekly was barely 20M.
> 
> That's useful additional data.  Thanks.
> 
> And yes, a weekly delta would be quite useful, taking the breakover point 
> out to about a month or so.  Practically speaking, I'd guess most 
> gentooers update once a month or more, so that should cover the vast 
> majority.  Beyond a month, just downloading a new full squashfs makes as 
> much sense anyway, and as the cutover would be automated, users on the 
> borderline wouldn't have to worry about whether they should just do the 
> normal sync or download an entirely new tarball, as they now need to do, 
> if they even bother at all.  For those users, it'd be an even BIGGER win.

Well, I've got even a better idea and I'll try to provide direct deltas
from N days back to the newest snapshot. Since the tree changes a lot,
those deltas will be smaller to download ;).

Now, does anyone have an old portage-YYZZ.tar.{bz2,xz} snapshot? I
need the official one from our mirrors, preferably 3-4 months old.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

[gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Duncan]

Michał Górny posted on Fri, 17 Jan 2014 20:30:00 +0100 as excerpted:

> Dnia 2014-01-17, o godz. 19:19:14 Duncan <1i5t5.dun...@cox.net>
> napisał(a):
> 
>> Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
>> 
>> > 96Mportage-20140108.sqfs

>> > For deltas [...]
>> > 
>> > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw

>> > applying it takes ~2.5 second on my 2 GHz Athlon64.
>> 
>> diffs are ~1/16 the full squashfs size[.] So people updating once a
>> week [or] 10 days would see a bandwidth savings, provided the sync
>> script was intelligent enough to apply updates serially.
>> 
>> The breakover point would be roughly an update every two weeks, or
>> twice a month
> 
> However, it may be actually beneficial to provide other durations, like
> weekly deltas. In my tests, the daily updates for this week summed up to
> almost 50M while the weekly was barely 20M.

That's useful additional data.  Thanks.

And yes, a weekly delta would be quite useful, taking the breakover point 
out to about a month or so.  Practically speaking, I'd guess most 
gentooers update once a month or more, so that should cover the vast 
majority.  Beyond a month, just downloading a new full squashfs makes as 
much sense anyway, and as the cutover would be automated, users on the 
borderline wouldn't have to worry about whether they should just do the 
normal sync or download an entirely new tarball, as they now need to do, 
if they even bother at all.  For those users, it'd be an even BIGGER win.

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Maciej Mrozowski]

On Friday 17 of January 2014 13:06:22 gro...@gentoo.org wrote:

| dev-util/kdevelop-php-docs

While of course it doesn't invalidate your entire idea, this particular example
is a kdevelop plugin written in C++ that provides php API documentation 
integration.
This tells however that decision to "auto"-keyword/stabilize remaining arches
cannot be taken lightly and not by everyone.

regards
MM

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Markos Chandras]

On 01/17/2014 04:27 PM, Michał Górny wrote:
> Hello, all.
> 
> I'm using squashfs to hold my Gentoo repositories on all of my systems
> for some time. As you probably know, this allows me to save space while
> keeping portage fast. However, it makes updating the tree quite
> burdensome and time-consuming.
> 
> We're already hosting daily gx86 tarballs on our mirrors, and deltas
> made using diffball. Those can be used with Zac's emerge-delta-webrsync
> to get daily updates done with minimal network overhead. Sadly, it
> takes the whole process even more time consuming :).
> 
> Therefore, I'd like to suggest an alternative solution that could help
> out Gentoo users that use squashfs for gx86 and would like to be able
> to get daily updates fast and easy.
> 
> The idea is to host -- along with the tarballs -- daily squashfs images
> of gx86 in a chosen format. Additionally, the images would come with
> deltas made using xdelta3 or a similar tool. Those deltas -- with
> a slight download overhead -- would allow very fast updates
> of the squashfs.
> 
> Now some numbers. I did some tests 'converting' late gx86 daily
> tarballs to squashfs. I've used squashfs 4.2 with LZO compression
> since it's quite good and very fast.
> 
> 96M   portage-20140108.sqfs
> 96M   portage-20140109.sqfs
> 96M   portage-20140110.sqfs
> 96M   portage-20140111.sqfs
> 96M   portage-20140112.sqfs
> 96M   portage-20140113.sqfs
> 97M   portage-20140114.sqfs
> 97M   portage-20140115.sqfs
> 
> For deltas, I've used xdelta3 with max compression (-9) and djw
> secondary compression (it gave ~0.1M smaller files than fgk
> and ~0.5M gain than with no secondary compression).
> 
> 4,9M  portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3M  portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> 5,6M  portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
> 8,9M  portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
> 6,3M  portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
> 7,8M  portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
> 8,5M  portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> 
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing
> a compressed filesystem. What's important, however, is that applying
> it takes ~2.5 second on my 2 GHz Athlon64.
> 
> So, even with the extra download time, the update is much faster
> than recreating the squashfs. And unlike some types of unionfs,
> it doesn't come with extra runtime slowdown.
> 
> What do you think?
> 

+1 I like the idea

-- 
Regards,
Markos Chandras

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

Dnia 2014-01-17, o godz. 19:19:14
Duncan <1i5t5.dun...@cox.net> napisał(a):

> Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
> 
> > Now some numbers. I did some tests 'converting' late gx86 daily tarballs
> > to squashfs. I've used squashfs 4.2 with LZO compression since it's
> > quite good and very fast.
> > 
> > 96M portage-20140108.sqfs
> [...]
> > 97M portage-20140114.sqfs
> > 97M portage-20140115.sqfs
> > 
> > For deltas [...]
> > 
> > 4,9Mportage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> > 6,3Mportage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> [...]
> > 8,5Mportage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> > 
> > As you can see, the deltas are quite large compared to the actual
> > changes. However, we could have expected that since we're diffing a
> > compressed filesystem. What's important, however, is that applying it
> > takes ~2.5 second on my 2 GHz Athlon64.
> 
> And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs 
> size, perhaps a bit larger.  So people updating once a week or even about 
> every 10 days would see a bandwidth savings, provided the sync script was 
> intelligent enough to apply updates serially.
> 
> The breakover point would be roughly an update every two weeks, or twice 
> a month, at which point just downloading a new full squashfs would be 
> easier, at about the same bandwidth.

Yes, that's the initial goal. The update code would catch that
condition and perform full fetch instead.

However, it may be actually beneficial to provide other durations, like
weekly deltas. In my tests, the daily updates for this week summed up
to almost 50M while the weekly was barely 20M.

> > What do you think?
> 
> How does this, particularly the metadata cache, interact with overlays?  
> That's /my/ big question.

The same way as usual. Nothing special happens unless you override
eclasses via overlays. If you do that, you need unionfs to save
the cache updates -- but then, it has no point for you to use squashfs.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

[gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Duncan]

Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:

> Now some numbers. I did some tests 'converting' late gx86 daily tarballs
> to squashfs. I've used squashfs 4.2 with LZO compression since it's
> quite good and very fast.
> 
> 96M   portage-20140108.sqfs
[...]
> 97M   portage-20140114.sqfs
> 97M   portage-20140115.sqfs
> 
> For deltas [...]
> 
> 4,9M  portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3M  portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
[...]
> 8,5M  portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> 
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing a
> compressed filesystem. What's important, however, is that applying it
> takes ~2.5 second on my 2 GHz Athlon64.

And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs 
size, perhaps a bit larger.  So people updating once a week or even about 
every 10 days would see a bandwidth savings, provided the sync script was 
intelligent enough to apply updates serially.

The breakover point would be roughly an update every two weeks, or twice 
a month, at which point just downloading a new full squashfs would be 
easier, at about the same bandwidth.

> What do you think?

How does this, particularly the metadata cache, interact with overlays?  
That's /my/ big question.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

Dnia 2014-01-17, o godz. 10:18:58
Alec Warner  napisał(a):

> On Fri, Jan 17, 2014 at 8:27 AM, Michał Górny  wrote:
> 
> > Hello, all.
> >
> > I'm using squashfs to hold my Gentoo repositories on all of my systems
> > for some time. As you probably know, this allows me to save space while
> > keeping portage fast. However, it makes updating the tree quite
> > burdensome and time-consuming.
> >
> > We're already hosting daily gx86 tarballs on our mirrors, and deltas
> > made using diffball. Those can be used with Zac's emerge-delta-webrsync
> > to get daily updates done with minimal network overhead. Sadly, it
> > takes the whole process even more time consuming :).
> >
> > Therefore, I'd like to suggest an alternative solution that could help
> > out Gentoo users that use squashfs for gx86 and would like to be able
> > to get daily updates fast and easy.
> >
> > The idea is to host -- along with the tarballs -- daily squashfs images
> > of gx86 in a chosen format. Additionally, the images would come with
> > deltas made using xdelta3 or a similar tool. Those deltas -- with
> > a slight download overhead -- would allow very fast updates
> > of the squashfs.
> >
> > Now some numbers. I did some tests 'converting' late gx86 daily
> > tarballs to squashfs. I've used squashfs 4.2 with LZO compression
> > since it's quite good and very fast.
> >
> > 96M portage-20140108.sqfs
> > 96M portage-20140109.sqfs
> > 96M portage-20140110.sqfs
> > 96M portage-20140111.sqfs
> > 96M portage-20140112.sqfs
> > 96M portage-20140113.sqfs
> > 97M portage-20140114.sqfs
> > 97M portage-20140115.sqfs
> >
> > For deltas, I've used xdelta3 with max compression (-9) and djw
> > secondary compression (it gave ~0.1M smaller files than fgk
> > and ~0.5M gain than with no secondary compression).
> >
> > 4,9Mportage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> > 6,3Mportage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> > 5,6Mportage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
> > 8,9Mportage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
> > 6,3Mportage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
> > 7,8Mportage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
> > 8,5Mportage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> >
> > As you can see, the deltas are quite large compared to the actual
> > changes. However, we could have expected that since we're diffing
> > a compressed filesystem. What's important, however, is that applying
> > it takes ~2.5 second on my 2 GHz Athlon64.
> >
> 
> It wasn't clear to me, are these trees with metadata included?

Yes, full metadata with md5-cache. That's the same thing you get via
'emerge --sync'. And that's why the deltas are so big -- I recall three
big cache updates this week.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Ciaran McCreesh]

On Fri, 17 Jan 2014 17:47:58 +0100
Tom Wijsman  wrote:
> Maybe we can let the package managers only perceive it as keyworded or
> stable if all of its dependencies are keyworded or stable on the
> architecture that the user runs. Then we can have repoman just ignore
> checking dependencies' keywords when we keyword or stabilize them.
> 
> Not sure how implementable this idea is though...

It's going to hurt for four reasons that I can think of right now.

Firstly, things you think are "obviously portable" sometimes aren't.

Secondly, users already get confused by "stable use masks". This is
going to be even worse: users aren't going to understand why a noarch
package isn't available for them.

Thirdly, you have to decide how to deal with long chains and cycles in
noarch dependencies.

Fourthly, the interaction with || deps is an awful mess.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Alec Warner]

On Fri, Jan 17, 2014 at 8:27 AM, Michał Górny  wrote:

> Hello, all.
>
> I'm using squashfs to hold my Gentoo repositories on all of my systems
> for some time. As you probably know, this allows me to save space while
> keeping portage fast. However, it makes updating the tree quite
> burdensome and time-consuming.
>
> We're already hosting daily gx86 tarballs on our mirrors, and deltas
> made using diffball. Those can be used with Zac's emerge-delta-webrsync
> to get daily updates done with minimal network overhead. Sadly, it
> takes the whole process even more time consuming :).
>
> Therefore, I'd like to suggest an alternative solution that could help
> out Gentoo users that use squashfs for gx86 and would like to be able
> to get daily updates fast and easy.
>
> The idea is to host -- along with the tarballs -- daily squashfs images
> of gx86 in a chosen format. Additionally, the images would come with
> deltas made using xdelta3 or a similar tool. Those deltas -- with
> a slight download overhead -- would allow very fast updates
> of the squashfs.
>
> Now some numbers. I did some tests 'converting' late gx86 daily
> tarballs to squashfs. I've used squashfs 4.2 with LZO compression
> since it's quite good and very fast.
>
> 96M portage-20140108.sqfs
> 96M portage-20140109.sqfs
> 96M portage-20140110.sqfs
> 96M portage-20140111.sqfs
> 96M portage-20140112.sqfs
> 96M portage-20140113.sqfs
> 97M portage-20140114.sqfs
> 97M portage-20140115.sqfs
>
> For deltas, I've used xdelta3 with max compression (-9) and djw
> secondary compression (it gave ~0.1M smaller files than fgk
> and ~0.5M gain than with no secondary compression).
>
> 4,9Mportage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3Mportage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> 5,6Mportage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
> 8,9Mportage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
> 6,3Mportage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
> 7,8Mportage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
> 8,5Mportage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
>
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing
> a compressed filesystem. What's important, however, is that applying
> it takes ~2.5 second on my 2 GHz Athlon64.
>

It wasn't clear to me, are these trees with metadata included?

-A


>
> So, even with the extra download time, the update is much faster
> than recreating the squashfs. And unlike some types of unionfs,
> it doesn't come with extra runtime slowdown.
>
> What do you think?
>
> --
> Best regards,
> Michał Górny
>

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[viv...@gmail.com]

On 01/17/14 17:27, Michał Górny wrote:
> Hello, all.
>
> I'm using squashfs to hold my Gentoo repositories on all of my systems
> for some time. As you probably know, this allows me to save space while
> keeping portage fast. However, it makes updating the tree quite
> burdensome and time-consuming.
Me too and you have my total support (maybe I've even proposed this
before to te list)

[snip]

>
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing
> a compressed filesystem. What's important, however, is that applying
> it takes ~2.5 second on my 2 GHz Athlon64.

Have you tried to give an order (always the same) to the compressed files?
It could give an advantage, tough it may be limited to 2^16 files the
option is -sort 


thanks for it,
Francesco

noarch packages, was Re: [gentoo-dev] rfc: revisiting our stabilization policy

[grozin]

On Fri, 17 Jan 2014, Ulrich Mueller wrote:

On Fri, 17 Jan 2014, grozin  wrote:

Maybe, a good solution is to introduce a special arch, "noarch", for
such packages (similar to what's done in the rpm world). Then, if a
package is ~noarch, it is automatically considered ~arch for all
arches. Similar for stable. The maintainer should be able to keyword
~noarch and to stabilize noarch. Comments?


How would you handle dependencies in such a scenario? All dependencies
must be keyworded or stable on all architectures, before the package
can be keyworded or stabilised on noarch?

Many "pure data" packages don't depend on anything.

Pure TeX/LaTeX packages should be keyworded ~noarch only if a suitable 
binary TeX is keyworded for each arch. Hmm, this, probably, means that 
they can never be stabilized as noarch.


Pure python scripts (without library dependencies) should become ~noarch 
if some suitable python binary is keyworded for each arch. Similarly for 
perl, ruby. Python is installed on each Gentoo box anyway, so, in this 
case it is less problematic.


Andrey

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[grozin]

On Fri, 17 Jan 2014, Tom Wijsman wrote:

On Fri, 17 Jan 2014 16:31:54 +0100
Ulrich Mueller  wrote:

On Fri, 17 Jan 2014, grozin  wrote:

Maybe, a good solution is to introduce a special arch, "noarch", for
such packages (similar to what's done in the rpm world). Then, if a
package is ~noarch, it is automatically considered ~arch for all
arches. Similar for stable. The maintainer should be able to keyword
~noarch and to stabilize noarch. Comments?


How would you handle dependencies in such a scenario? All dependencies
must be keyworded or stable on all architectures, before the package
can be keyworded or stabilised on noarch?


Maybe we can let the package managers only perceive it as keyworded or
stable if all of its dependencies are keyworded or stable on the
architecture that the user runs. Then we can have repoman just ignore
checking dependencies' keywords when we keyword or stabilize them.

Very reasonable.

Andrey

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Peter Stuge]

Michał Górny wrote:
> > > What do you think?
> > 
> > Excellent. I think it could quickly become the prefered protage
> > storage format, although a loopback mount is needed.
> 
> You can use sys-fs/squashfuse :).
> 
> Though honestly I'd prefer if portage was able to take squashfs image
> path in repos.conf and mount it locally for build-time. This will have
> the extra advantage that we wouldn't have to worry about remounting it
> after sync.

Or read it directly without mounting.


//Peter


pgpVeLJOgAY6E.pgp
Description: PGP signature

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

Dnia 2014-01-17, o godz. 17:51:41
Peter Stuge  napisał(a):

> Michał Górny wrote:
> > What do you think?
> 
> Excellent. I think it could quickly become the prefered protage
> storage format, although a loopback mount is needed.

You can use sys-fs/squashfuse :).

Though honestly I'd prefer if portage was able to take squashfs image
path in repos.conf and mount it locally for build-time. This will have
the extra advantage that we wouldn't have to worry about remounting it
after sync.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Peter Stuge]

Michał Górny wrote:
> What do you think?

Excellent. I think it could quickly become the prefered protage
storage format, although a loopback mount is needed.


//Peter


pgpZaqbTCHide.pgp
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Tom Wijsman]

On Fri, 17 Jan 2014 16:31:54 +0100
Ulrich Mueller  wrote:

> > On Fri, 17 Jan 2014, grozin  wrote:
> 
> > Maybe, a good solution is to introduce a special arch, "noarch", for
> > such packages (similar to what's done in the rpm world). Then, if a
> > package is ~noarch, it is automatically considered ~arch for all
> > arches. Similar for stable. The maintainer should be able to keyword
> > ~noarch and to stabilize noarch. Comments?
> 
> How would you handle dependencies in such a scenario? All dependencies
> must be keyworded or stable on all architectures, before the package
> can be keyworded or stabilised on noarch?

Maybe we can let the package managers only perceive it as keyworded or
stable if all of its dependencies are keyworded or stable on the
architecture that the user runs. Then we can have repoman just ignore
checking dependencies' keywords when we keyword or stabilize them.

Not sure how implementable this idea is though...

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 17/01/14 11:27 AM, Michał Górny wrote:
> Hello, all.
> 
> I'm using squashfs to hold my Gentoo repositories on all of my
> systems for some time. As you probably know, this allows me to save
> space while keeping portage fast. However, it makes updating the
> tree quite burdensome and time-consuming.
> 
> We're already hosting daily gx86 tarballs on our mirrors, and
> deltas made using diffball. Those can be used with Zac's
> emerge-delta-webrsync to get daily updates done with minimal
> network overhead. Sadly, it takes the whole process even more time
> consuming :).
> 
> Therefore, I'd like to suggest an alternative solution that could
> help out Gentoo users that use squashfs for gx86 and would like to
> be able to get daily updates fast and easy.
> 
> The idea is to host -- along with the tarballs -- daily squashfs
> images of gx86 in a chosen format. Additionally, the images would
> come with deltas made using xdelta3 or a similar tool. Those deltas
> -- with a slight download overhead -- would allow very fast
> updates of the squashfs.
> 
> Now some numbers. I did some tests 'converting' late gx86 daily 
> tarballs to squashfs. I've used squashfs 4.2 with LZO compression 
> since it's quite good and very fast.
> 
> 96M   portage-20140108.sqfs 96M   portage-20140109.sqfs 96M
> portage-20140110.sqfs 96M portage-20140111.sqfs 96M
> portage-20140112.sqfs 96M portage-20140113.sqfs 97M
> portage-20140114.sqfs 97M portage-20140115.sqfs
> 
> For deltas, I've used xdelta3 with max compression (-9) and djw 
> secondary compression (it gave ~0.1M smaller files than fgk and
> ~0.5M gain than with no secondary compression).
> 
> 4,9M  portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw 6,3M
> portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw 5,6M
> portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw 8,9M
> portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw 6,3M
> portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw 7,8M
> portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw 8,5M
> portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> 
> As you can see, the deltas are quite large compared to the actual 
> changes. However, we could have expected that since we're diffing a
> compressed filesystem. What's important, however, is that applying 
> it takes ~2.5 second on my 2 GHz Athlon64.
> 
> So, even with the extra download time, the update is much faster 
> than recreating the squashfs. And unlike some types of unionfs, it
> doesn't come with extra runtime slowdown.
> 
> What do you think?
> 

PLEASE DO!  This sounds fantastic, and is something i've been
considering proposing for some time.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlLZWoMACgkQ2ugaI38ACPAb0gEAwFPWdtI5J8l9QH1YTrKe2Mbm
OwPL4wGg9ORaHv0FkVcA/iLsP/z3uz3sJoWciR5ZCZ73HblyDgH/flAhakNhl3NZ
=ool3
-END PGP SIGNATURE-

[gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

Hello, all.

I'm using squashfs to hold my Gentoo repositories on all of my systems
for some time. As you probably know, this allows me to save space while
keeping portage fast. However, it makes updating the tree quite
burdensome and time-consuming.

We're already hosting daily gx86 tarballs on our mirrors, and deltas
made using diffball. Those can be used with Zac's emerge-delta-webrsync
to get daily updates done with minimal network overhead. Sadly, it
takes the whole process even more time consuming :).

Therefore, I'd like to suggest an alternative solution that could help
out Gentoo users that use squashfs for gx86 and would like to be able
to get daily updates fast and easy.

The idea is to host -- along with the tarballs -- daily squashfs images
of gx86 in a chosen format. Additionally, the images would come with
deltas made using xdelta3 or a similar tool. Those deltas -- with
a slight download overhead -- would allow very fast updates
of the squashfs.

Now some numbers. I did some tests 'converting' late gx86 daily
tarballs to squashfs. I've used squashfs 4.2 with LZO compression
since it's quite good and very fast.

96M portage-20140108.sqfs
96M portage-20140109.sqfs
96M portage-20140110.sqfs
96M portage-20140111.sqfs
96M portage-20140112.sqfs
96M portage-20140113.sqfs
97M portage-20140114.sqfs
97M portage-20140115.sqfs

For deltas, I've used xdelta3 with max compression (-9) and djw
secondary compression (it gave ~0.1M smaller files than fgk
and ~0.5M gain than with no secondary compression).

4,9Mportage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
6,3Mportage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
5,6Mportage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
8,9Mportage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
6,3Mportage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
7,8Mportage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
8,5Mportage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw

As you can see, the deltas are quite large compared to the actual
changes. However, we could have expected that since we're diffing
a compressed filesystem. What's important, however, is that applying
it takes ~2.5 second on my 2 GHz Athlon64.

So, even with the extra download time, the update is much faster
than recreating the squashfs. And unlike some types of unionfs,
it doesn't come with extra runtime slowdown.

What do you think?

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Ulrich Mueller]

> On Fri, 17 Jan 2014, grozin  wrote:

> Maybe, a good solution is to introduce a special arch, "noarch", for
> such packages (similar to what's done in the rpm world). Then, if a
> package is ~noarch, it is automatically considered ~arch for all
> arches. Similar for stable. The maintainer should be able to keyword
> ~noarch and to stabilize noarch. Comments?

How would you handle dependencies in such a scenario? All dependencies
must be keyworded or stable on all architectures, before the package
can be keyworded or stabilised on noarch?

Ulrich


pgpF0dADhYThd.pgp
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Michał Górny]

Dnia 2014-01-17, o godz. 14:02:51
gro...@gentoo.org napisał(a):

> Maybe, a good solution is to introduce a special arch, "noarch", for such 
> packages (similar to what's done in the rpm world). Then, if a package is 
> ~noarch, it is automatically considered ~arch for all arches. Similar for 
> stable. The maintainer should be able to keyword ~noarch and to stabilize 
> noarch. Comments?

If you want to play with such a major change, you should start a new
thread instead of starting it in the middle of this spam-art.
Otherwise, many people will miss it and complain loudly afterwards.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] rfc: revisiting our stabilization policy

[Rich Freeman]

On Fri, Jan 17, 2014 at 2:58 AM, Matt Turner  wrote:
> On Thu, Jan 16, 2014 at 11:02 PM,   wrote:
>> Maybe, a good solution is to introduce a special arch, "noarch", for such
>> packages (similar to what's done in the rpm world). Then, if a package is
>> ~noarch, it is automatically considered ~arch for all arches. Similar for
>> stable. The maintainer should be able to keyword ~noarch and to stabilize
>> noarch. Comments?
>>
> There's been opposition to this in the past, but I'm in favor of
> giving this a shot.
>

I too think it is at least worth a try.  We can learn from this either
way.  Maybe start out leaving it up to maintainer discretion, and if
that becomes an issue we can try to formulate guidelines.

Rich