[gentoo-user] Portage Storage using SVN
Hi guys, I proposed this awhile back, and got shot down. At the time, the arguments for using SVN for portage storage were pretty shallow, and someone was able to easily shoot them down. I believe I have come up with better reasoning for using SVN. Someone may still shoot them down, but hey, it's worth a try. PROBLEM 1 Let's say openldap had a problem. So, we decide to mask the latest version of openldap, in an effort to roll back to the version that was working. Well, we find out that openldap still does not work. So, we finally determine that it is library W. So, now we mask library W, in an attempt to roll back to the version that was working. Oh no, now we find out that library W is used by 20 other packages, that require the latest version of library W in order to work. So, now we have to mask library W, and 20 packages in order to get our openldap system functional, assuming you cared about the 20 other broken packages, which may break other packages, which may break yet other packages. Wouldn't it be nice to just go emerge --revert-portage, which goes back to the last exported copy of the portage, that you had from subversion? Boy, would that ever be convenient. It would be simple enough to store a local history of portage tags that the user was using in the past. PROBLEM 2 Have you ever synced the portage, only to find out there's a broken package simply because you synced in the middle of a mirror being updated? Well, subversion operations are entirely atomic, meaning no need to worry about such silly things. Also, copies (tags) could be created for every single day, and special ones for milestones. PROBLEM 3 Don't sync more than once a day, or you may be temporarily banned? Well, with SVN being tagged only once a day, there would be no need to worry about this, seeing that POTENTIAL ISSUES Now, I'm not entirely sure of the performance implications of subversion for this purpose. So, that would definitely have to either be tested, or someone would have to talk with the subversion folks to know if it would be a problem for thousands of users to access subversion in readonly mode. It would certainly be annoying for a developer to go svn commit, and have to wait for half an hour because everyone else is updating their local copies. But, that could be solved by mirrors only getting updated once every day, at 12 midnight. How does one get SVN copied over to mirrors in an atomic way, or does the SVN location just get shut down during the mirror update? Any thoughts? -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Portage Storage using SVN
Hi, On Sun, 23 Jul 2006 02:42:43 -0600 Trenton Adams [EMAIL PROTECTED] wrote: I proposed this awhile back, and got shot down. At the time, the arguments for using SVN for portage storage were pretty shallow, and someone was able to easily shoot them down. I believe I have come up with better reasoning for using SVN. Someone may still shoot them down, but hey, it's worth a try. #1: You're aware that there's a CVS for portage, aren't you? I'm still not quite sure if you are suggesting using SVN for the portage mirrors and if you are suggesting that users also have a full SVN history on the clients, too? PROBLEM 1 [...] PROBLEM 2 [...] PROBLEM 3 [...] Well, are those really problems at all? I mean, isn't it easy to overcome them? Is it worth dedicating time and work into that svn thing? POTENTIAL ISSUES Now, I'm not entirely sure of the performance implications of subversion for this purpose. So, that would definitely have to either be tested, or someone would have to talk with the subversion folks to know if it would be a problem for thousands of users to access subversion in readonly mode. Well, of course! There's definately a reason to use rsync. It would certainly be annoying for a developer to go svn commit, and have to wait for half an hour because everyone else is updating their local copies. But, that could be solved by mirrors only getting updated once every day, at 12 midnight. Oh, yeah. Your midnight, my midnight? It would definately be annoying to make a small glitch and have to wait 24hrs until the fix for that gets promoted. The problem you mentioned that at some points there are slightly errorneous ebuilds in portage or minor inconsistencies can only be fixed by promoting updates fast. The solution you propose costs a lot of CPU power, even more storage on the mirrors and lacks some positive aspects that the current solution has. Take a look at e.g. the major BSDs ports and package systems. They certainly have similar problems. OK, looking at the BSDs, I like the feature that there are branches with the aim to build a package tree that is as consistent as possible. That would be a plus. But that would imply a lot of work and a change in ebuild maintainance. I don't see this coming soon for Gentoo. -hwh -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Portage Storage using SVN
On 7/23/06, Neil Bothwick [EMAIL PROTECTED] wrote: On Sun, 23 Jul 2006 02:42:43 -0600, Trenton Adams wrote: Let's say openldap had a problem. So, we decide to mask the latest version of openldap, in an effort to roll back to the version that was working. Well, we find out that openldap still does not work. So, we finally determine that it is library W. So, now we mask library W, in an attempt to roll back to the version that was working. Oh no, now we find out that library W is used by 20 other packages, that require the latest version of library W in order to work. So, now we have to mask library W, and 20 packages in order to get our openldap system functional, assuming you cared about the 20 other broken packages, which may break other packages, which may break yet other packages. Wouldn't it be nice to just go emerge --revert-portage, which goes back to the last exported copy of the portage, that you had from subversion? Boy, would that ever be convenient. It would be simple enough to store a local history of portage tags that the user was using in the past. Wouldn't that revert the whole of portage, not just the parts affecting ldap, W and the rest? All of a sudden emerge world wants to downgrade previously upgraded programs, some of which may have been updated for security reasons. This means the user has to go through the Changelogs, ebuild and package.mask to find out why they are being downgraded and whether to apply the downgrade. this would be far more work that package masking a few updates dependent on W, which could probably be scripted. Yes, it would revert the whole portage, and that would be desired, so as to be up quickly. It could potentially take hours to figure out which packages are being affected by a broken updated package. But, perhaps it would be better to write a utility that will downgrade a package, as well as all dependencies. Or does this already happen when you downgrade? -- Neil Bothwick BASIC: Bill's Attempt to Seize Industry Control -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Portage Storage using SVN
On 7/23/06, Hans-Werner Hilse [EMAIL PROTECTED] wrote: Hi, On Sun, 23 Jul 2006 02:42:43 -0600 Trenton Adams [EMAIL PROTECTED] wrote: I proposed this awhile back, and got shot down. At the time, the arguments for using SVN for portage storage were pretty shallow, and someone was able to easily shoot them down. I believe I have come up with better reasoning for using SVN. Someone may still shoot them down, but hey, it's worth a try. #1: You're aware that there's a CVS for portage, aren't you? I'm still not quite sure if you are suggesting using SVN for the portage mirrors and if you are suggesting that users also have a full SVN history on the clients, too? No, not a full history, just a tag history. PROBLEM 1 [...] PROBLEM 2 [...] PROBLEM 3 [...] Well, are those really problems at all? I mean, isn't it easy to overcome them? Is it worth dedicating time and work into that svn thing? I'm not sure, is it? Is there scripts already out there to overcome the problems suggested? If there are, I would sure appreciate knowing about them. :) It could be a lack of my understanding how the portage downgrade process works. But if you downgrade a package, will it downgrade all the packages depending on that version as well? POTENTIAL ISSUES Now, I'm not entirely sure of the performance implications of subversion for this purpose. So, that would definitely have to either be tested, or someone would have to talk with the subversion folks to know if it would be a problem for thousands of users to access subversion in readonly mode. Well, of course! There's definately a reason to use rsync. It would certainly be annoying for a developer to go svn commit, and have to wait for half an hour because everyone else is updating their local copies. But, that could be solved by mirrors only getting updated once every day, at 12 midnight. Oh, yeah. Your midnight, my midnight? It would definately be annoying to make a small glitch and have to wait 24hrs until the fix for that gets promoted. The problem you mentioned that at some points there are slightly errorneous ebuilds in portage or minor inconsistencies can only be fixed by promoting updates fast. That's true, and I suppose that's not quite as good as what exists right now. But does gentoo really have less than 24 hour bug fix turn arounds? -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Portage Storage using SVN
On Sunday 23 July 2006 10:42, Trenton Adams wrote: Hi guys, I proposed this awhile back, and got shot down. At the time, the arguments for using SVN for portage storage were pretty shallow, and someone was able to easily shoot them down. I believe I have come up with better reasoning for using SVN. Someone may still shoot them down, but hey, it's worth a try. PROBLEM 1 Let's say openldap had a problem. So, we decide to mask the latest version of openldap, in an effort to roll back to the version that was working. Well, we find out that openldap still does not work. So, we finally determine that it is library W. So, now we mask library W, in an attempt to roll back to the version that was working. Oh no, now we find out that library W is used by 20 other packages, that require the latest version of library W in order to work. So, now we have to mask library W, and 20 packages in order to get our openldap system functional, assuming you cared about the 20 other broken packages, which may break other packages, which may break yet other packages. no, you don't. You put a dependency for libraryW version XY-working into the openldap related ebuilds. Wouldn't it be nice to just go emerge --revert-portage, which goes back to the last exported copy of the portage, that you had from subversion? Boy, would that ever be convenient. It would be simple enough to store a local history of portage tags that the user was using in the past. why not do a normal emerge sync emerge -au world? the updated openldap package will pull in the right library W, and if other apps need library W+1, well that is where slots are used. Your problem is none. It is solved every day within the portage tree. No svn needed for that. PROBLEM 3 Don't sync more than once a day, or you may be temporarily banned? Well, with SVN being tagged only once a day, there would be no need to worry about this, seeing that you are banned because of the enormous waste of bandwidth if you sync every odd hours. I don't see where svn would resolve that. Even if all mirrors would only updated once a day (what a nightmare - a broken package may stay for 24h or more in the tree, instead being replaced as soon as the dev notices), there would be people syncing more than once in 24h. -- gentoo-user@gentoo.org mailing list
