Re: Proposal for new frozen repositories server
On 6 February 2011 12:29, Daniel Drake d...@laptop.org wrote: http://wiki.laptop.org/go/User:DanielDrake/NewMockProposal Thanks to everyone for the feedback. We decided not to take up the proposal because of the disadvantage that it is harder to roll back to earlier release candidates. However, we have split the monster repo into individual git repositories, and employed a few git tricks to make things less painful. New disks have been installed and the system software has been updated. It's now live, fully backwards compatible, and seems to be working. I've also documented the processes here: http://wiki.laptop.org/go/Frozen_repositories Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Proposal for new frozen repositories server
On Sun, 6 Feb 2011, Martin Langhoff wrote: On Sun, Feb 6, 2011 at 11:44 AM, Daniel Drake d...@laptop.org wrote: git has a tendency to hang during switches between branches git + many large compressed binary files == lots of unhappiness, pointless abuse of IO. You don't save any interesting disk space re metadata (fs metadata vs git metadata likely cancel out). And git's internal assumptions about what you're trying to do... yeah, LOTs of unhappiness. You have my vote on ditching it. And I cannot think of anything better than a good old filesystem for the task at hand. there was a discussion on the git mailing list a week or so ago about how to handle large files, and it appears that you can do a lot with the clean/smudge filters that can make it store this sort of file in the filesystem, but in a way that is transparent to users. now, when you do a checkout (including when you switch branches), you do need to make that checkout contain the correct files for that version. that will include copying the large binary versions for that version. David Lang ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Proposal for new frozen repositories server
On Mon, Feb 7, 2011 at 5:51 AM, da...@lang.hm wrote: there was a discussion on the git mailing list a week or so ago about how to handle large files, and it appears that you can do a lot with the clean/smudge filters that can make it store this sort of file in the filesystem, but in a way that is transparent to users. It's interesting. But none of that would help much with the usage pattern we have in this specific tool. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Proposal for new frozen repositories server
mock.laptop.org, our server for frozen packages i.e. a clone of the latest Fedora and OLPC RPMs frozen for each software release, is out of disk space and somewhat unloved, and I'd like to use the opportunity to make some changes to the system. In a sentence, the changes are: Repositories which are essentially unmodified by OLPC are moved out of revision control, and OLPC-local repositories are moved from being unrelated branches in a huge/old/long-living git repository into their own individual, short-lived git repositories. Full details here: http://wiki.laptop.org/go/User:DanielDrake/NewMockProposal Comments welcome. I'm looking to refine the proposal and implement it within the next few weeks Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Proposal for new frozen repositories server
On Sun, Feb 6, 2011 at 7:29 AM, Daniel Drake d...@laptop.org wrote: mock.laptop.org, our server for frozen packages i.e. a clone of the latest Fedora and OLPC RPMs frozen for each software release, is out of disk space and somewhat unloved, and I'd like to use the opportunity to make some changes to the system. In a sentence, the changes are: Repositories which are essentially unmodified by OLPC are moved out of revision control, and OLPC-local repositories are moved from being unrelated branches in a huge/old/long-living git repository into their own individual, short-lived git repositories. Full details here: http://wiki.laptop.org/go/User:DanielDrake/NewMockProposal Comments welcome. I'm looking to refine the proposal and implement it within the next few weeks Why not just add more disk space? Disk space is cheap. Or archive the older parts of the repo. The benefit to version controlling even the non-OLPC packages is that the repo contains a *complete* snapshot of all the bits that went into a particular build. This protects against upstream reorganizing their repos, or cleaning old packages, or changing their package manager, etc, etc. It makes builds 100% reproducible at any point in the future (or at least that was the point). Removing any packages from the repo would pretty much defeat the whole purpose. If disk space really is a problem, one alternative is to make release candidate builds on a branch, and only merge released builds to master. Then you can prune the branches with git to free up disk space, while still ensuring that you have all the bits related to released builds. But really -- why not more disk? [I understand there are parts of mock which are non-ideal, but the essential saving all the bits part isn't one of them. IMHO.] --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Proposal for new frozen repositories server
On 6 February 2011 15:55, C. Scott Ananian csc...@laptop.org wrote: The benefit to version controlling even the non-OLPC packages is that the repo contains a *complete* snapshot of all the bits that went into a particular build. This protects against upstream reorganizing their repos, or cleaning old packages, or changing their package manager, etc, etc. It makes builds 100% reproducible at any point in the future (or at least that was the point). Removing any packages from the repo would pretty much defeat the whole purpose. I agree. None of that changes under my proposal. No packages are removed from the system. If disk space really is a problem, one alternative is to make release candidate builds on a branch, and only merge released builds to master. Then you can prune the branches with git to free up disk space, while still ensuring that you have all the bits related to released builds. But really -- why not more disk? [I understand there are parts of mock which are non-ideal, but the essential saving all the bits part isn't one of them. IMHO.] git has a tendency to hang during switches between branches, meaning you have to kill it then clean up the mess. I tried to do prune and gc, left it for hours and nothing seemed to happen. The repo is big and ugly, and trying to google the issues I just see people saying that git shouldn't be used for binary trees and talking about its inefficiencies. I agree with the design principles of the existing mock and don't feel like they are being changed under my proposal -- just the storage system. Thanks a lot for jumping in! Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
