Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On Wed, 2013-01-02 at 17:05 -0800, Zac Medico wrote: This command seems to do the trick: $ ls -1 /usr/portage/profiles/updates/ | grep -Ev '(08|09|10|11|12|13)$' 1Q-2004 1Q-2005 1Q-2006 1Q-2007 2Q-2004 2Q-2005 2Q-2006 2Q-2007 3Q-2004 3Q-2005 3Q-2006 3Q-2007 4Q-2004 4Q-2005 4Q-2006 4Q-2007 OK, these old updates have now been cleaned from the tree. They have been tar'd and can be downloaded at: http://dev.gentoo.org/~dolsen/cleaned-updates-20130111.tbz2 If we are going to archive them to be available. It can be moved to there, otherwise I can remove it once we know it is not needed. Now, where to document this procedure for future reference? -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On 01/11/2013 01:10 AM, Brian Dolbec wrote: On Wed, 2013-01-02 at 17:05 -0800, Zac Medico wrote: This command seems to do the trick: $ ls -1 /usr/portage/profiles/updates/ | grep -Ev '(08|09|10|11|12|13)$' 1Q-2004 1Q-2005 1Q-2006 1Q-2007 2Q-2004 2Q-2005 2Q-2006 2Q-2007 3Q-2004 3Q-2005 3Q-2006 3Q-2007 4Q-2004 4Q-2005 4Q-2006 4Q-2007 OK, these old updates have now been cleaned from the tree. They have been tar'd and can be downloaded at: http://dev.gentoo.org/~dolsen/cleaned-updates-20130111.tbz2 If we are going to archive them to be available. It can be moved to there, otherwise I can remove it once we know it is not needed. Now, where to document this procedure for future reference? Maybe here: http://www.gentoo.org/proj/en/qa/treecleaners/policy.xml -- Thanks, Zac
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On Mon, 2012-12-10 at 08:59 +0100, Ulrich Mueller wrote: On Mon, 10 Dec 2012, Sergei Trofimovich wrote: gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time [long list omitted] old entries are done in different context (comparing to 2012): - some packages change names 2 or 3 times - slots have different meaning moreover: - if you set your PORTDIR to different directory you'll get all that full update. And will break the system. Old profile entries used to break eclass-manpages and latex-base (due to double renaming) It's worse: Bad entries in the old files may go unnoticed for a long time. But if such a file is updated for whatever reason, it will be reprocessed on users' systems, including any bad entries contained in it. Thus the reason for removal is simple: old entries are potentially buggy as nobody verifies them. I wouldn't even know how to verify them. Let's remove that cruft. We can be extra conservative and keep five years of backlog (i.e. everything from before 2008 would be removed now). Ulrich OK, that seems to be some very good reasons to tree-clean them. What's our next step? Tree-cleaners, does this fall into your department? Or should I prepare a list of files and/or updates to clean? -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On 01/02/2013 02:46 PM, Brian Dolbec wrote: On Mon, 2012-12-10 at 08:59 +0100, Ulrich Mueller wrote: On Mon, 10 Dec 2012, Sergei Trofimovich wrote: gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time [long list omitted] old entries are done in different context (comparing to 2012): - some packages change names 2 or 3 times - slots have different meaning moreover: - if you set your PORTDIR to different directory you'll get all that full update. And will break the system. Old profile entries used to break eclass-manpages and latex-base (due to double renaming) It's worse: Bad entries in the old files may go unnoticed for a long time. But if such a file is updated for whatever reason, it will be reprocessed on users' systems, including any bad entries contained in it. Thus the reason for removal is simple: old entries are potentially buggy as nobody verifies them. I wouldn't even know how to verify them. Let's remove that cruft. We can be extra conservative and keep five years of backlog (i.e. everything from before 2008 would be removed now). Ulrich OK, that seems to be some very good reasons to tree-clean them. What's our next step? It might be nice to document the removal policy in the developer handbook, devmanual, or something. Tree-cleaners, does this fall into your department? That seems fitting. Or should I prepare a list of files and/or updates to clean? This command seems to do the trick: $ ls -1 /usr/portage/profiles/updates/ | grep -Ev '(08|09|10|11|12|13)$' 1Q-2004 1Q-2005 1Q-2006 1Q-2007 2Q-2004 2Q-2005 2Q-2006 2Q-2007 3Q-2004 3Q-2005 3Q-2006 3Q-2007 4Q-2004 4Q-2005 4Q-2006 4Q-2007 -- Thanks, Zac
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On Mon, 10 Dec 2012, Sergei Trofimovich wrote: gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time [long list omitted] old entries are done in different context (comparing to 2012): - some packages change names 2 or 3 times - slots have different meaning moreover: - if you set your PORTDIR to different directory you'll get all that full update. And will break the system. Old profile entries used to break eclass-manpages and latex-base (due to double renaming) It's worse: Bad entries in the old files may go unnoticed for a long time. But if such a file is updated for whatever reason, it will be reprocessed on users' systems, including any bad entries contained in it. Thus the reason for removal is simple: old entries are potentially buggy as nobody verifies them. I wouldn't even know how to verify them. Let's remove that cruft. We can be extra conservative and keep five years of backlog (i.e. everything from before 2008 would be removed now). Ulrich
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On 12/10/2012 12:10 AM, Paweł Hajdan, Jr. wrote: I propose that we say, once a year, schedule a tree-cleaning of old updates files. These updates files could be added to a tarball made available for download. That way if they are needed to update a system older than what the main tree has been tree-cleaned to. They can then be manually downloaded, extracted to the normal location and then run the fixpackages command. I think that complicates the process. :-/ But maybe the advantages outweigh that. The main question here is what is a reasonable length of time to keep the updates actively in-tree? -- From my experience in the forums, I think any updates older than 4 years should be subject to tree-cleaning. Yeah, 4 years is ancient and would probably be non-trivial to update anyway. -- Most old systems that have been updated tend to be less than that, probably about 2 years. 2 years seem reasonable. For the records: We do have some Gentoo box serving as VirtualBox host here, installed in early 2010, not updated since then, with an uptime of 836 days right now. It is subject to upgrade, but there may come another year until that to happen (never change a running system). Although I do not expect the update to be trivial, keeping things like pkgmove for at least 4 years sounds reasonable. /haubi/
[gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
Starting from a question by Markos in #gentoo-portage about whether to remove entries in profiles/updates for tree-cleaned packages... I propose that we say, once a year, schedule a tree-cleaning of old updates files. These updates files could be added to a tarball made available for download. That way if they are needed to update a system older than what the main tree has been tree-cleaned to. They can then be manually downloaded, extracted to the normal location and then run the fixpackages command. The main question here is what is a reasonable length of time to keep the updates actively in-tree? -- From my experience in the forums, I think any updates older than 4 years should be subject to tree-cleaning. -- Most old systems that have been updated tend to be less than that, probably about 2 years. This should allow a reasonable amount of time for a user to update an old system. Thoughts? -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On 12/9/12 1:17 PM, Brian Dolbec wrote: Starting from a question by Markos in #gentoo-portage about whether to remove entries in profiles/updates for tree-cleaned packages... What's the advantage of doing that? I propose that we say, once a year, schedule a tree-cleaning of old updates files. These updates files could be added to a tarball made available for download. That way if they are needed to update a system older than what the main tree has been tree-cleaned to. They can then be manually downloaded, extracted to the normal location and then run the fixpackages command. I think that complicates the process. :-/ But maybe the advantages outweigh that. The main question here is what is a reasonable length of time to keep the updates actively in-tree? -- From my experience in the forums, I think any updates older than 4 years should be subject to tree-cleaning. Yeah, 4 years is ancient and would probably be non-trivial to update anyway. -- Most old systems that have been updated tend to be less than that, probably about 2 years. 2 years seem reasonable. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On Sun, 2012-12-09 at 15:10 -0800, Paweł Hajdan, Jr. wrote: On 12/9/12 1:17 PM, Brian Dolbec wrote: Starting from a question by Markos in #gentoo-portage about whether to remove entries in profiles/updates for tree-cleaned packages... What's the advantage of doing that? None, it actually could make it more difficult for a user to know why his old installed pkg isn't available. It was just what started the discussion about cleaning the old updates. Zac suggested this thread for opinions... ... [12:46] zmedico dol-sen: you should take a poll on the gentoo-dev ml to see how long people think we should keep them ... [12:47] zmedico yeah, seems like it's good to end-of-life them at some point I propose that we say, once a year, schedule a tree-cleaning of old updates files. These updates files could be added to a tarball made available for download. That way if they are needed to update a system older than what the main tree has been tree-cleaned to. They can then be manually downloaded, extracted to the normal location and then run the fixpackages command. I think that complicates the process. :-/ But maybe the advantages outweigh that. It does make updating an ancient system slightly more difficult. But that would be the least of the user's troubles compared to some of the pkg updates, tinderbox downloads and manual unpacks that have been needed to be done. But on the other hand how long should we keep that stale info in the tree? See below :) The main question here is what is a reasonable length of time to keep the updates actively in-tree? -- From my experience in the forums, I think any updates older than 4 years should be subject to tree-cleaning. Yeah, 4 years is ancient and would probably be non-trivial to update anyway. yup, they are -- Most old systems that have been updated tend to be less than that, probably about 2 years. 2 years seem reasonable. That works too. FYI... Currently there are updates files in profiles/updates/ dating back to 2004 -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
Brian Dolbec wrote: remove entries in profiles/updates for tree-cleaned packages... What's the advantage of doing that? None .. FYI... Currently there are updates files in profiles/updates/ dating back to 2004 Do they take up significant storage space or transfer time, compared to the rest of the tree? If not, I can't think of a reason to remove them. //Peter pgpwXGARYACGA.pgp Description: PGP signature
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On Mon, 2012-12-10 at 01:52 +0100, Peter Stuge wrote: Brian Dolbec wrote: remove entries in profiles/updates for tree-cleaned packages... What's the advantage of doing that? None .. FYI... Currently there are updates files in profiles/updates/ dating back to 2004 Do they take up significant storage space or transfer time, compared to the rest of the tree? If not, I can't think of a reason to remove them. //Peter No, once they are downloaded, they don't change ever after the quarterly rollover which starts a new updates file. Nor do they take up significant storage space. They probably take up a higher percentage of your fs's inodes than % diskspace. But they do take time to process and verify every time emerge initiates a fixpackages call after every sync. That will save time more than it will space. Also, (taking it a bit to extremes ;) ) why would a new user with a 2 month old system and up-to-date tree want 8 year old update cruft lingering about their filesystem...taking time to check they've been processed? Do you? Let's just do an annual tree-cleaning and be done with them. -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On Sun, 09 Dec 2012 21:15:37 -0800 Brian Dolbec dol...@gentoo.org wrote: No, once they are downloaded, they don't change ever after the quarterly rollover which starts a new updates file. gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time 4Q-2012 3Q-2012 1Q-2008 2Q-2012 4Q-2011 1Q-2012 3Q-2011 2Q-2011 1Q-2011 2Q-2005 3Q-2008 4Q-2010 1Q-2005 2Q-2008 3Q-2010 2Q-2010 1Q-2010 3Q-2004 4Q-2009 3Q-2009 2Q-2009 1Q-2009 4Q-2008 3Q-2005 3Q-2007 4Q-2007 2Q-2007 1Q-2007 4Q-2006 2Q-2006 3Q-2006 1Q-2006 4Q-2005 4Q-2004 2Q-2004 1Q-2004 old entries are done in different context (comparing to 2012): - some packages change names 2 or 3 times - slots have different meaning moreover: - if you set your PORTDIR to different directory you'll get all that full update. And will break the system. Old profile entries used to break eclass-manpages and latex-base (due to double renaming) Random example: Let's loot at why 2Q-2005 was changed in 2010: https://bugs.gentoo.org/show_bug.cgi?id=351760 Thus the reason for removal is simple: old entries are potentially buggy as nobody verifies them. -- Sergei signature.asc Description: PGP signature
Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files
On 12/09/2012 09:15 PM, Brian Dolbec wrote: No, once they are downloaded, they don't change ever after the quarterly rollover which starts a new updates file. Nor do they take up significant storage space. They probably take up a higher percentage of your fs's inodes than % diskspace. But they do take time to process and verify every time emerge initiates a fixpackages call after every sync. That will save time more than it will space. It skips the older files if their timestamp hasn't changed since it was recorded in /var/cache/edb/mtimedb. So, there's minimal overhead from keeping them, unless you lose /var/cache/edb/mtimedb somehow. -- Thanks, Zac
