Re: [Monotone-devel] cvs_import
On Wed, Jul 17, 2013 at 08:47:54PM +0200, Markus Wanner wrote: Hendrik, On 07/15/2013 07:21 PM, Hendrik Boom wrote: Just wondering what the current status of of monotone's CVS support; in particular, cvs_import, csv_pull. cvs_sync, cvs_takeover, and I've heard there's even a cvs_push. Standard monotone's cvs_import works fine for simple cases. I didn't do any work on the cvs_import branch and I don't think it's in a usable state. So standard monotone's cvs_import is different from what's on the cvs_iport branch, as described on http://wiki.monotone.ca/CvsImport/, even assuming that page is up to date. And there's also http://wiki.monotone.ca/Branches/nvm.cvsimport-branch-reconstruction/ which bears your name and says Status: Work in Progress Still close to completion. Do you know if that status is still valid? or if anyone else might have worked on it or be on the case? Is it still worth doing? You might want to check if cvs2svn can be of help. They have a nice git export function and their CVS sanitizer code is field proven. I'm not sure if you can get that into monotone, though. Is the conversion a one-time event, or can it keep up with further revisions on the cvs site withoug having to start over? There are all one-time conversion options, which need access to the RCS files on the CVS server. tailor may be an option, if you want a continuous mirror. THe main need for a continuous mirror I see would be during transition from CVS to monotone. There would likely be work in progress on the old system while the new one is being adopted. It certainly has a monotone plugin. I'm not sure what the status of cvs_sync is, but it's intended to provide continuous synchronization as well. tailor (http://darcs.net/RelatedSoftware/Tailor)looks interesting, thanks. -- hendrik Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
Daniel Carosone wrote: The latter seems to be of the most general utility, given current volumes and efforts of other projects (assuming the format contains what we need and isn't completely horrible). The git-fast-import format is quite good and well-documented [1]. Markus Wanner wrote: Note that git has the ability to represent more than two ancestors for a merged revision, which monotone does not. Not that it's an often used feature, but IIRC cvs2git is one of its users. (I vaguely remember discussing these issues with Michael Haggerty, author of cvs2svn. See that project's mailing list). cvs2svn/cvs2git etc. has a setting to limit merges to two parents. So this should not be a problem. hend...@topoi.pooq.com wrote: with all these cvs2* programs around, it might be useful to isolate their common core (the cvs part and the 2 part) from the * part. Incresed modularity. ANy chance that such factoring would spread to the others and overall maintenance costs? These programs (cvs2svn, cvs2git, cvs2bzr, and cvs2hg) are all part of the cvs2svn project [2] and share 90% of their code (namely the hard part, which is figuring out the CVS history). Michael [1] http://www.kernel.org/pub/software/scm/git/docs/git-fast-import.html [2] http://cvs2svn.tigris.org/ ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
Hi, hend...@topoi.pooq.com wrote: I'd like to look at this, possibly next January. If anyone else would, please do also. Cool! Which one was that again, git to monotone converter? The more the merrier, at least during design discovery stage. What languages are all these *2* programs written in? cvs2svn and its derivates are written in python. IN case I have to tear one apart, are any written in secure, statically typed languages (whose programs I find easiest to read and debug). The existing cvs_import and git_export commands are certainly written in C++. Something like git_import would be very welcome, IMO. Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
On Wed, Dec 23, 2009 at 09:01:04AM +0100, Markus Wanner wrote: Hi, hend...@topoi.pooq.com wrote: I'd like to look at this, possibly next January. If anyone else would, please do also. Cool! Which one was that again, git to monotone converter? DOn't care all that much. I have an existing CVS repository, still in active use, to convert to monotone. Ideally under the constraint that people still working on branches in CVS will be able to finish their work and check stuff in and have the updates converted as they trickle in -- so that new stuff can be started in monotone and old stuff finished in CVS. The more the merrier, at least during design discovery stage. What languages are all these *2* programs written in? cvs2svn and its derivates are written in python. python is easier to debug than C++. One option is to replace the svn-output stuff with uses of monotone automate. with all these cvs2* programs around, it might be useful to isolate their common core (the cvs part and the 2 part) from the * part. Incresed modularity. ANy chance that such factoring would spread to the others and overall maintenance costs? I also have a CVS updater (CVSup) that updates slave CVS repositories from a master repository (CVSup). It's written in Modula 3, another language that's easy to debug in and the advantage that it's statically typed. CVSup will interface with remote CVSs over the net, but lacks the analysis that cvs2svn does to figure out the structure. IN case I have to tear one apart, are any written in secure, statically typed languages (whose programs I find easiest to read and debug). The existing cvs_import and git_export commands are certainly written in C++. Something like git_import would be very welcome, IMO. Ideally mtn sync would detect the system used at the other end and use the appropriate conversion tool automatically. But this is probably just pie in the sky for now. Especially syncing with systems that don't maintain information monotone uses. -- hendrik ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
On Wed, Dec 23, 2009 at 03:22:12PM -0500, hend...@topoi.pooq.com wrote: I have an existing CVS repository, still in active use, to convert to monotone. Ideally under the constraint that people still working on branches in CVS will be able to finish their work and check stuff in and have the updates converted as they trickle in -- so that new stuff can be started in monotone and old stuff finished in CVS. I might add that the people in the project still have to be convinced that monotone is the way to go. At least one influential one is, but I don't think anyone is convinced of the conversion process at present (including me). That's why I'd like to take a good look at the problem. -- hendrik ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
Hi, hend...@topoi.pooq.com wrote: If we achieve interoperability with svn, then the cvs2svn code can be used followed by svn2mtn. ..or modify cvs2git (which is based on cvs2svn) to export to something monotone can parse, instead of git-fast-import format. ..or write a git2mtn that can parse git-fast-import format. Both of which are probably easier than writing svn2mtn. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
Am 22.12.2009 22:02, schrieb Markus Wanner: ..or modify cvs2git (which is based on cvs2svn) to export to something monotone can parse, instead of git-fast-import format. ..or write a git2mtn that can parse git-fast-import format. I had a git-fast-import parser that emitted monotone packets once that I wanted to use with a mercurial-to-git exporter, and dropped it in favor of building my own hg2mtn parser. The reason is that the git-fast-import format is quite specific to the design of git. For example, while git-fast-import (later on) gained rename tracking, it's still optional - as git doesn't care for it anyway. In the end, I had to resynthesize various pieces of data that existed in the original format and were lost in the hg-git step. I'm not sure if that applies for any svn2git translator, and if desired, I can look if I can find my (old) mtn-fast-import tree somewhere. Regards, Patrick ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
On Tue, Dec 22, 2009 at 10:02:24PM +0100, Markus Wanner wrote: If we achieve interoperability with svn, then the cvs2svn code can be used followed by svn2mtn. ..or modify cvs2git (which is based on cvs2svn) to export to something monotone can parse, instead of git-fast-import format. ..or write a git2mtn that can parse git-fast-import format. The latter seems to be of the most general utility, given current volumes and efforts of other projects (assuming the format contains what we need and isn't completely horrible). Any volunteers? -- Dan. pgpi1SsnNQCaL.pgp Description: PGP signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
Hi, Daniel Carosone wrote: The latter seems to be of the most general utility, given current volumes and efforts of other projects (assuming the format contains what we need and isn't completely horrible). It can hardly be any more horrible than CVS itself ;-) Note that git has the ability to represent more than two ancestors for a merged revision, which monotone does not. Not that it's an often used feature, but IIRC cvs2git is one of its users. (I vaguely remember discussing these issues with Michael Haggerty, author of cvs2svn. See that project's mailing list). Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
On Wed, Dec 23, 2009 at 08:07:29AM +1100, Daniel Carosone wrote: On Tue, Dec 22, 2009 at 10:02:24PM +0100, Markus Wanner wrote: If we achieve interoperability with svn, then the cvs2svn code can be used followed by svn2mtn. ..or modify cvs2git (which is based on cvs2svn) to export to something monotone can parse, instead of git-fast-import format. ..or write a git2mtn that can parse git-fast-import format. The latter seems to be of the most general utility, given current volumes and efforts of other projects (assuming the format contains what we need and isn't completely horrible). Another post indicated that git-fast-import format lacks some structure that had to be resynthesized in another *2* conversion. Perhaps that's not the best format, say I, who at present know nothing about any of these formats. Any volunteers? I'd like to look at this, possibly next January. If anyone else would, please do also. The more the merrier, at least during design discovery stage. What languages are all these *2* programs written in? IN case I have to tear one apart, are any written in secure, statically typed languages (whose programs I find easiest to read and debug). -- hendrik ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import failure
From time to time I check in to see if the state of cvs_import has improved at all by attempting to import the NetBSD src repository. Since it had been over a year since last I tried, I gave it a go over the weekend. After about 36 hours, it died (which is about what I recall it doing last time). 1) is there somewhere I should be opening some kind of bug on this? (Is there one open?) 2) I have vague recollections of having somewhat more success with nvm.cvs_branch_reconstruction - but that looks like it hasn't been touched in a year. Other thoughts? Unfortunately, my scrollback filled up, and I only got most of the command output: mtn: warning: skipping invalid path '\n.d,v' mtn: warning: skipping invalid path '\n.err,v' mtn: warning: skipping invalid path '\n.r,v'mtn: warning: skipping invalid path '\n.t,v' mtn: warning: skipping invalid path '\n1.d,v' mtn: warning: skipping invalid path '\n1.r,v' mtn: warning: skipping invalid path '\n1.t,v'mtn: warning: skipping invalid path '\n2.d,v' mtn: warning: skipping invalid path '\n2.r,v' mtn: warning: skipping invalid path '\n2.t,v' mtn: branches | versions mtn: 319 | 1362879 external/intel-fw-public/iwl4965/dist/iwlwifi-4965-1.uc mtn: branches | revisions | versions mtn: 319 | 18928 | 1362879 mtn: fatal: formatter failed on unix/ssh_agent_platform.cc:132: boost::too_few_args: for mat-string referred to more arguments than were passed mtn: fatal: boost::exception_detail::clone_implboost::exception_detail::error_info_injectorboost::io::too_few_args : boost::too_few_args: format-string referred to more arg uments than were passed mtn: this is almost certainly a bug in monotone.mtn: please send this error message, the output of 'mtn version --full', mtn: and a description of what you were doing to monotone-de...@nongnu.org.mtn: wrote debugging log to /home/riz/.monotone/dump mtn: if reporting a bug, please include this filecolossus:riz ~/netbsd2 mtn --version --full monotone 0.45 (base revision: a19f8b2017c81c3c6c8b2bb3247f865f6ed4e5a9)colossus:riz ~/netbsd2 cat ~/.monotone/dump Encountered an error while musing upon the following:paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violatedwarning: skipping invalid path '\n.d,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n.err,v'Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violatedwarning: skipping invalid path '\n.r,v'Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n.t,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n1.d,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n1.r,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n1.t,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n2.d,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n2.r,v' Encountered an error while musing upon the following: paths.cc:420: detected internal error, 'I(utf8_validate(utf8(data, origin::internal)) !has_bad_component_chars(data) data != . data != ..)' violated warning: skipping invalid path '\n2.t,v' Encountered an error while musing upon the following: unix/ssh_agent_platform.cc:132: detected system error, 'E(++deadcycles 8)' violated Current work set: 4 items - begin 'system_flavour' (in virtual void sanity::initialize(int, char**, const char*), at sanity.cc:103) NetBSD 5.0_STABLE NetBSD 5.0_STABLE (COLOSSUS) #1: Sat Nov 21 16:54:47 PST 2009
Re: [Monotone-devel] cvs_import failure
Am 23.11.09 17:30, schrieb Jeff Rizzo: From time to time I check in to see if the state of cvs_import has improved at all by attempting to import the NetBSD src repository. Since it had been over a year since last I tried, I gave it a go over the weekend. After about 36 hours, it died (which is about what I recall it doing last time). 1) is there somewhere I should be opening some kind of bug on this? (Is there one open?) 2) I have vague recollections of having somewhat more success with nvm.cvs_branch_reconstruction - but that looks like it hasn't been touched in a year. Other thoughts? Unfortunately, my scrollback filled up, and I only got most of the command output: Unfortunately nobody has been working on cvs_import for the last couple of months and I doubt someone will soon pick up the topic again. For most people a subversion exporter is even higher on the priority list than a working cvs import. There's certainly prior art for cvs import (f.e. I heard that the cvs2svn code is pretty well a couple of times), but unless its a high demand topic for somebody, there won't happen much in this area. I can only say that patches and code are more than welcome and that we'd love to support everybody improving the importer code, but thats about it. Thomas. -- GPG-Key 0x160D1092 | tommyd3...@jabber.ccc.de | http://thomaskeller.biz Please note that according to the EU law on data retention, information on every electronic information exchange might be retained for a period of six months or longer: http://www.vorratsdatenspeicherung.de/?lang=en signature.asc Description: OpenPGP digital signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import failure
On Mon, Nov 23, 2009 at 08:30:44AM -0800, Jeff Rizzo wrote: From time to time I check in to see if the state of cvs_import has improved at all by attempting to import the NetBSD src repository. Since it had been over a year since last I tried, I gave it a go over the weekend. After about 36 hours, it died (which is about what I recall it doing last time). These bad filenames have always been an issue; I've just removed them before importing, which allows it to go much further. It's been about the same amount of time since I last tried an import too, so I completely forget how far it does get after that. 1) is there somewhere I should be opening some kind of bug on this? (Is there one open?) 2) I have vague recollections of having somewhat more success with nvm.cvs_branch_reconstruction - but that looks like it hasn't been touched in a year. Yeah, my last test was with that branch too. -- Dan. pgp2EKsZT2fih.pgp Description: PGP signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import date bug
It seem there is a bug with cvs_import. The bug is the same reported in: http: //www.mail-archive.com/monotone-devel@nongnu.org/msg13338.html but the patch seems not working for me. I'm using a version of monotone that contains that fix but dates in a converted cvs repository always contain year 1970. ./mtn version monotone 0.45dev (base revision: e5dcbaac354a94d95e8c39ab7d6d9439dabfef40) uname -a Linux PCP-luigi 2.6.28-15- generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC 2009 i686 GNU/Linux That bug seems fixed by the following patch: # # old_revision [e5dcbaac354a94d95e8c39ab7d6d9439dabfef40] # # patch rcs_import.cc # from [e0868099bc25d42879979b40e410b356221cad9b] #to [6a571a6fd084342518cd7652b8920ad54e3df985] # --- rcs_import.cc e0868099bc25d42879979b40e410b356221cad9b +++ rcs_import.cc 6a571a6fd084342518cd7652b8920ad54e3df985 @@ -1371,7 +1371,7 @@ cluster_consumer::store_auxiliary_certs( branch_name(branchname, origin::user), utf8(cvs. changelog_interner.lookup(p.changelog), origin::internal), - date_t(p.time * 1000), + date_t((s64)p.time * 1000), cvs.author_interner.lookup(p.author)); } ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import
Markus Schiltknecht wrote: mtn: branches | versions mtn:0 | 101454 contrib/ncurses/menu/m_items.c,vterminate called after throwing an instance of 'Botan::Memory_Exhaustion' How much memory do you have on that machine? On my 64bit Linux, the FreeBSD import peaks at about 1.6 Gb of needed memory. The import doesn't run through, though. It hits another invariant later on. Whops, that doesn't even fit in the swap, right now... Well, I should have a box with more RAM HDD in the near future. -- Lapo Luchini [EMAIL PROTECTED] (OpenPGP X.509) www.lapo.it (Jabber, ICQ, MSN) ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import
Hi, Ralf S. Engelschall wrote: BTW, if you need a clean medium-sized CVS repository for testing which is both very strict in its branch naming and layout and also contains a reasonable amount of branches and sub-branches, revisions spreaded over 6 years and about 35000 commits, feel free to test with the OpenPKG CVS (rsync -rav rsync://rsync.openpkg.org/openpkg-cvs/ /tmp/openpkg-cvs/). The interesting module in this repository is named openpkg-src (the other modules you can ignore), containing all the OpenPKG package specifications on a dozend branches and sub-branches. When I tried the cvsimport-branch-reconstruction stuff some time ago it crashed on this repository module, so it might be interested to check again. In the meantime I used Tailor to import and track at least the HEAD of openpkg-src in a Monotone repository... FYI: the OpenPKG repository has some files with names containing invalid characters in it, thus cvs_import currently aborts on those. The very same applies to the NetBSD repository. AFAIU, the correct fix would be an adjustment of our tree_walker, to also fix all other problems with non-UTF-8 encoded filenames. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import
Hello Ralf, Ralf S. Engelschall wrote: BTW, if you need a clean medium-sized CVS repository for testing which is both very strict in its branch naming and layout and also contains a reasonable amount of branches and sub-branches, revisions spreaded over 6 years and about 35000 commits, feel free to test with the OpenPKG CVS (rsync -rav rsync://rsync.openpkg.org/openpkg-cvs/ /tmp/openpkg-cvs/). Thanks for the hint. I've downloaded and tested it. Unfortunately, it still fails on an invariant... :-( Known issue, I think there even is a a test case for it. For everybody interested in the inner workings of that beast, I've updated the wiki: http://www.venge.net/mtn-wiki/CvsImport. There are still some issues, i.e.: * non-UTF-8 filenames (i.e. as in the NetBSD repository). The tree_walker fails on those. * blobs (revisions) which have dependencies into two branches. In monotone that's theoretically possible, but not in CVS. (i.e. Postgres repo) * deleted files might erronously appear in tagged revisions. I'm working on those... Please let me know if you have an urgent desire for importing a specific CVS repository or if you'd like to test importing one yourself. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import
In message [EMAIL PROTECTED] on Wed, 07 Nov 2007 16:53:52 +0100, Markus Schiltknecht [EMAIL PROTECTED] said: markus Please let me know if you have an urgent desire for importing markus a specific CVS repository or if you'd like to test importing markus one yourself. I do... the OpenSSL one. Cheers, Richard -- Richard Levitte [EMAIL PROTECTED] http://richard.levitte.org/ When I became a man I put away childish things, including the fear of childishness and the desire to be very grown up. -- C.S. Lewis ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import
Hi, I've just pushed all of my cvs_import work in the branch n.v.m.cvsimport-branch-reconstruction (which certainly is a misnomer nowadays). Sorry for all the sudden traffic ;-) Four cvs related tests still fail, but I'm currently testing with bigger repositories. I'll give more details tomorrow, too tired now. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import
On Tue, Nov 06, 2007, Markus Schiltknecht wrote: I've just pushed all of my cvs_import work in the branch n.v.m.cvsimport-branch-reconstruction (which certainly is a misnomer nowadays). Sorry for all the sudden traffic ;-) Four cvs related tests still fail, but I'm currently testing with bigger repositories. [...] BTW, if you need a clean medium-sized CVS repository for testing which is both very strict in its branch naming and layout and also contains a reasonable amount of branches and sub-branches, revisions spreaded over 6 years and about 35000 commits, feel free to test with the OpenPKG CVS (rsync -rav rsync://rsync.openpkg.org/openpkg-cvs/ /tmp/openpkg-cvs/). The interesting module in this repository is named openpkg-src (the other modules you can ignore), containing all the OpenPKG package specifications on a dozend branches and sub-branches. When I tried the cvsimport-branch-reconstruction stuff some time ago it crashed on this repository module, so it might be interested to check again. In the meantime I used Tailor to import and track at least the HEAD of openpkg-src in a Monotone repository... Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import feature sponsored
On Sat, Sep 08, 2007 at 08:49:29PM +0200, Markus Schiltknecht wrote: Dear monotone hackers, I'm happy to announce that I've recently signed a contract with MQSoftware, who sponsors me for finishing my cvs_import rewrite. I wish to thank Kelly F. Hickel for making this possible, and I'm looking forward to contribute to their and monotone's success. How cool! Looking forward to seeing what you come up with. -- Nathaniel -- Details are all that matters; God dwells there, and you never get to see Him if you don't struggle to get them right. -- Stephen Jay Gould ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import feature sponsored
Dear monotone hackers, I'm happy to announce that I've recently signed a contract with MQSoftware, who sponsors me for finishing my cvs_import rewrite. I wish to thank Kelly F. Hickel for making this possible, and I'm looking forward to contribute to their and monotone's success. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import prepared revisions
Hi, I'm wondering why cvs_import (original and mine) feeds the revisions from the cluster_consumer to a vector of prepared revisions. And afterwards cluster_consumer::store_revisions() sends them all at once to the database. Why do we split tasks like that? Why can't we just skip the list of prepared_revisions and store them directly into the database in cluster_consumer::consume_cluster()? Assuming write caching is enabled, I would suppose feeding them to the database in consume_cluster() would even be faster, because that would not split between CPU/RAM intensive and disk I/O intensive tasks but do them in parallel. Can I remove the prepared_revision stuff safely or do I miss something? Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import benchmark
On Thu, Oct 12, 2006 at 08:31:40AM +0200, Lapo Luchini wrote: % du -hs cvsroot.ypsilon 1.5Mcvsroot.ypsilon % python2.4 benchmark.py -m mtn26=mtn-0.26.exe -m mtn30=mtn-0.30.exe -m mtn061011=mtn-061011.exe -b 'cvsimp=CVSImport(EmptyRepo(), ypsilon, /home/lapo/monotone-trunk/benchmark/cvsroot.ypsilon)' --cache /home/lapo/monotone-trunk/benchmark/cache scratch results3 % for n in results3/*/stats.csv; do echo $n; cat $n; done results3/cvsimp-mtn26-time/stats.csv cvs_import-system-time,0.51,0.53,0.59 cvs_import-user-time,104.54,105.25,103.37 cvs_import-wall-time,130.21,127.43,127.83 results3/cvsimp-mtn30-time/stats.csv cvs_import-system-time,0.46,0.48,0.49 cvs_import-user-time,97.07,98.1,97.09 cvs_import-wall-time,119.28,119.21,119.68 results3/cvsimp-mtn061011-time/stats.csv cvs_import-system-time,0.79,0.54,0.81 cvs_import-user-time,196.57,196.5,192.68 cvs_import-wall-time,236.94,234.7,230.96 So... 6% improvement between 0.26 and 0.30, and a 100% worsening from 0.30 and yesterday's trunk! 0_0 Weird! I wonder what can have done that. Maybe 'heights' branch is landed and was not faster as expected? No, it hasn't landed yet... I'll definitely take a closer look at it this evening, now gotta go to work ;-) On unix, obviously, we could quickly peek into the problem by adding some of the profiling switches, but I dunno what you can do on windows. -- Nathaniel -- - Don't let your informants burn anything. - Don't grow old. - Be good grad students. -- advice of Murray B. Emeneau on the occasion of his 100th birthday ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import and timezones
On Oct 2, 2006, at 4:18 AM, Christof Petig wrote: Tony Tung schrieb: Is there some secret incantation to tell monotone to treat CVS timestamps as GMT? Or am I totally off base? TZ=UTC monotone ... ? yes this sucks. There is no way to tell whether CVS uses GMT or localtime (besides checking the installation (I remember a debconf entry and a cvs version dependancy)) Christof I already tried that, but got the same results. Thanks, Tony ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import and timezones
On Oct 2, 2006, at 9:46 AM, Tony Tung wrote: On Oct 2, 2006, at 4:18 AM, Christof Petig wrote: Tony Tung schrieb: Is there some secret incantation to tell monotone to treat CVS timestamps as GMT? Or am I totally off base? TZ=UTC monotone ... ? yes this sucks. There is no way to tell whether CVS uses GMT or localtime (besides checking the installation (I remember a debconf entry and a cvs version dependancy)) Christof I already tried that, but got the same results. Thanks, Tony It seems tzset() is never called. Without that, the environment variable TZ is never processed. I put it in cvs_commit::cvs_commit (rcs_file const r, string const rcs_version, file_id const ident, cvs_history cvs) just to test it, and it behaves correctly. Since we don't want to actually do that every single time cvs_commit (...) is called, where would be a good place to call tzset()? Thanks, Tony ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
On Fri, Dec 16, 2005 at 09:14:35AM +0100, Christof Petig wrote:
Problem: cvs_import would need to know how to reach the imported
repository for correct certificates (at the moment cvssync stores the
revision information per repository).
Off the cuff thought: CVS repositories don't have unique names anyway.
Would it make more sense for cvssync to store a nickname for the repo,
and then at cvs_{push,pull} time one can figure out the mapping from
nickname to an actual access method?
This mapping could be cached in a db var, similar to the default
netsync server stuff.
The nickname could have a simple default, so that if one only ever
uses a single cvs server for a branch family (as is by far the more
common case), then one never needs to specify a nickname, or even know
that the nickname exists.
-- Nathaniel
--
Sentience can be such a burden.
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
On Thu, 2005-12-15 at 15:53 -0800, Nathaniel Smith wrote: The goal of the last cvs_import rewrite was basically, use cvs2svn's algorithm to get things right :-). The main limitations I'm aware of with current cvs_import are: -- it doesn't do any branch reconstruction. This would be _really_ helpful; at the moment, this makes its branch handling almost completely useless, because in monotone going forward you won't be able to usefully merge between branches that aren't linked up. The difficulty is that this is necessarily a heuristic operation, and there are states a CVS repo can be in that are simply not possible to meaningfully translate. -- no incremental re-importing. In that case, I don't really understand how monotone's cvs_import works. After reading rcs_import.cc I got the impression, that the treewalker first collects all files and parses them. Then somehow revisions and branches are created (import_cvs_branch() or what function was that?). I found it very hard to read the code and I still don't really understand what's going on. Could somebody quickly outline the algorithm used and the differences to cvs2svn? Right now we just do things in memory, and it seems to go okay. We'd have to look at the argument for how saving stuff somewhere persistent would be a significant win, I guess. I was thinking about helping re-import. A re-import could use information like what rcs-version of a file is in which monotone revision - or so. Still a vague idea... Intriguing idea! How do you work out what the merge looks like? ..the next non-overlapping cvs revision would be the merged revision in monotone, I guess. Even if that wasn't really the case, it's not any worse than the fixed up CVS repository. Maybe we could add hooks or have an interactive cvs_import variant where you could define where (in CVS) the merge really happened? Or more generally: give the user the possibility to edit the CVS - monotone revision mapping (at least in case of CVS inconsistencies). My goals with this are: * gain speed in subsequent imports Do you have profiling data showing where the bottlenecks are? Sorry, no. But only having to parse changed files in CVSROOT and only importing new revisions seems like a sure gain to me. And that could be possible when storing results from cvs_import. (Right now I'm pretty sure the bottleneck is the changeset sanity checking, just like for initial pull. Re-imports don't have to deal with that; maybe if we added re-import support it would already be plenty fast.) That's what I meant with 'subsequent imports' * (correct) branch support This would be _really_ good to have, and even an urgent need, as mentioned above. It might even be the first thing you should look at/try, as a way of familiarizing yourself with the issues involved in CVS importing... Yes, I'll try. But so far I didn't understand the cvs_import.cc code enough to find similarities in the algorithm to cvs2svn :-( Anyway, thanks for your help. I'll read the code again... Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
Nathaniel Smith schrieb: On the other hand, I don't see why cvs_import and cvssync couldn't record what they've done in a compatible format, so that you can do a cvs_import and then use the resulting repo with cvssync, for instance... cvs_import would have to generate certificates which indicate the corresponding file revisions for a revision. That's all what's missing. Problem: cvs_import would need to know how to reach the imported repository for correct certificates (at the moment cvssync stores the revision information per repository). Christof signature.asc Description: OpenPGP digital signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
Markus Schiltknecht schrieb: Right now we just do things in memory, and it seems to go okay. We'd have to look at the argument for how saving stuff somewhere persistent would be a significant win, I guess. I was thinking about helping re-import. A re-import could use information like what rcs-version of a file is in which monotone revision - or so. Still a vague idea... Perhaps re-import is not needed once cvs_import can store the information needed for cvssync. Cvssync is efficient when it comes to synchronize two repositories (small number of revisions, recent changes). Intriguing idea! How do you work out what the merge looks like? ..the next non-overlapping cvs revision would be the merged revision in monotone, I guess. Even if that wasn't really the case, it's not any worse than the fixed up CVS repository. Maybe we could add hooks or have an interactive cvs_import variant where you could define where (in CVS) the merge really happened? Or more generally: give the user the possibility to edit the CVS - monotone revision mapping (at least in case of CVS inconsistencies). That's exactly what cvssync does. Storing this information in an efficient way is difficult. But since cvs_import creates the revision (or more specifically the manifest) it can add a hidden file (.cvsrevisions) which is delta encoded as usual and efficiently stored. ... It occured to me that cvs_pull (cvssync's import) could work that way ... This would make the storage more efficient and robust. But it creates a notable difference between cvs_pull (which can change/add files) and cvs_push (which can only change certificates). Great idea, I will follow this way and change cvssync (will take some time to finish) * (correct) branch support This would be _really_ good to have, and even an urgent need, as mentioned above. It might even be the first thing you should look at/try, as a way of familiarizing yourself with the issues involved in CVS importing... Yes, I'll try. But so far I didn't understand the cvs_import.cc code enough to find similarities in the algorithm to cvs2svn :-( Anyway, thanks for your help. I'll read the code again... [with CVS manifest I mean a mapfile,rcs revision at a specific changeset] Of course I thought about branch rooting, too. My current plan is to calculate the base cvs manifest before the first branch commit (by removing .N.1 where appropriate IIRC) and look for a monotone revision matching this CVS manifest (which is easier if the CVS manifest is stored as a file, so you can look up the revision which contains this file). Take this monotone revision as the parent of the first revision to create on this subbranch. If you don't find a matching revision (e.g. because someone hand-edited the repository or did strange branching magic) simply let the branch float (no parent revision). Problems arise from the fact that branch tags are moveable (so you might not/no longer be able to calculate a name for the branch). Christof signature.asc Description: OpenPGP digital signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
In message [EMAIL PROTECTED] on Thu, 15 Dec 2005 15:53:10 -0800, Nathaniel Smith [EMAIL PROTECTED] said: njs On Wed, Dec 14, 2005 at 12:06:37PM +0100, Markus Schiltknecht wrote: njs I have been looking at cvsps, which can extract patchsets from cvs njs repositories. Unfortunately, it fails in case of inconsistent CVS njs repositories (i.e. the PostgreSQL cvs repo). njs njs Yeah, the rumors are basically that the only actually correct tools njs for cvs importing are: njs -- cvs2svn njs -- monotone's cvs_import (but it doesn't link up branches) njs -- Canonical's internal, secret tool Hmm, does anyone have an opinion on git-cvs (the cvs importing tool for git)? Since git has (or had?) some similarities with monotone, it might be something to look at more closely... Either way, inconsistent CVS repositories is a real PITA, and there's a lot of different thoughts on how to handle them. I know no tool that handles those perfectly. Cheers, Richard - Please consider sponsoring my work on free software. See http://www.free.lp.se/sponsoring.html for details. -- Richard Levitte [EMAIL PROTECTED] http://richard.levitte.org/ When I became a man I put away childish things, including the fear of childishness and the desire to be very grown up. -- C.S. Lewis ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
At 2005.12.16.04.53, in [EMAIL PROTECTED], Richard Levitte - VMS Whacker [EMAIL PROTECTED] wrote: In message [EMAIL PROTECTED] on Thu, 15 Dec 2005 15:53:10 -0800, Nathaniel Smith [EMAIL PROTECTED] said: njs On Wed, Dec 14, 2005 at 12:06:37PM +0100, Markus Schiltknecht wrote: njs I have been looking at cvsps, which can extract patchsets from cvs njs repositories. Unfortunately, it fails in case of inconsistent CVS njs repositories (i.e. the PostgreSQL cvs repo). njs njs Yeah, the rumors are basically that the only actually correct tools njs for cvs importing are: njs -- cvs2svn njs -- monotone's cvs_import (but it doesn't link up branches) njs -- Canonical's internal, secret tool Hmm, does anyone have an opinion on git-cvs (the cvs importing tool for git)? Since git has (or had?) some similarities with monotone, it might be something to look at more closely... Git's cvs-import uses cvsps to do the hard work. I've used it to import several repositories, and have had a lot of success with it. I also tried the cvssync branch found git was 100% successful with the imports while cvssync had quite a few issues. Though that was a while ago, so cvssync may have gotten better. Dave -- *** David Langenberg Network Based Services The University of Chicago Success always occurs in private, and failure in full view. *** ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
On Fri, Dec 16, 2005 at 10:39:43AM +0100, Christof Petig wrote: That's exactly what cvssync does. Storing this information in an efficient way is difficult. But since cvs_import creates the revision (or more specifically the manifest) it can add a hidden file (.cvsrevisions) which is delta encoded as usual and efficiently stored. Why hidden? In fact, why not store it as CVS/Entries? It might not be quite as efficient, but it might be more useful to the end user. -- Dan. pgpKAb7bNcVAX.pgp Description: PGP signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
[CCing the list] Markus Schiltknecht schrieb: What do you think? A totally different approach would be to enhance the cvssync branch to replace cvs_import. It can import local and remote repositories alike. And you can reimport/commit changes from/to the server. My basic feeling is that (performance issues put aside (which is a very brave assumption)) being able to import remote repositories as well is a bonus of this branch. It has basic side-branch support but I ran out of time to finish it. You might want to check for the shortcomings of the current cvssync code on the webpage http://venge.net/monotone/wiki/BranchStatuses . Be my guest to ask more questions. As the author of the branch I'm somewhat biased of course. Christof signature.asc Description: OpenPGP digital signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
On Thu, Dec 15, 2005 at 11:40:21AM +0100, Christof Petig wrote: (performance issues put aside (which is a very brave assumption)) I'm not sure they can be put aside in this case; cvssync talks to cvs, and lets cvs do the actual reconstruction in this case, right? The problem is that using cvs to extract all file texts from a cvs repo is O(n^2), while cvs_import, by reading the files directly, can be O(n). There's a fair amount of real-world evidence that this difference is critical, for real-world large repositories. (For instance, my understanding is that cvs2svn used to also be O(n^2) due to a bug, so when Danny Berlin was converting the gcc repository to svn, he actually created a Frankensteinien combination of monotone's cvs_import code and cvs2svn and used that.) On the other hand, I don't see why cvs_import and cvssync couldn't record what they've done in a compatible format, so that you can do a cvs_import and then use the resulting repo with cvssync, for instance... -- Nathaniel -- So let us espouse a less contested notion of truth and falsehood, even if it is philosophically debatable (if we listen to philosophers, we must debate everything, and there would be no end to the discussion). -- Serendipities, Umberto Eco ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import rewrite
On Thu, Dec 15, 2005 at 03:53:10PM -0800, Nathaniel Smith wrote: I.e. you don't absolutely need to sort by date. Overlapping commits in CVS would better be handled as two heads which later got merged again in monotone. Intriguing idea! How do you work out what the merge looks like? Indeed it is, it sounds very good. I like this idea. A lot. The merge, almost by definition, would be a matter of merging manifests for unrelated changes to different files. If the commits overlap in CVS on the same file, it's almost certain to have created a conflict for the CVS users, and they'd then become separate commits around that sequence point. Someone's CVS repository is sure to prove me wrong, of course. -- Dan. pgpEDBiNbC1Hx.pgp Description: PGP signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import rewrite
Hello monotone hackers, what's up with the cvs_import rewrite branches? Anybody still working on such a thing? I have been looking at cvsps, which can extract patchsets from cvs repositories. Unfortunately, it fails in case of inconsistent CVS repositories (i.e. the PostgreSQL cvs repo). Then I gave cvs2svn a try, which complained about a symbol occuring twice in some files, but that was easy to fix. Then it converted the whole inconsistent CVS repository into a subversion one. AFAICT getting most inconsistencies right and handling branches correctly. cvs2svn uses a multiple passes of processing the cvs repository. Some store their results in database files (BerkeleyDB, I think it was). The only drawback I saw was it's inability to do subsequent conversions. So you have to convert once, then you must use subversion. From the output and a quick glance at the source (in python) it does something like that: * parse the cvsroot and its rcs files * check consistency of files * make up revisions from parsed rcs files * check consistency of revisions * sort cvs revisions * create a cvs revisions - svn revisions mapping * import the single revisions one after another into a subversion repository. Anyway, I thought about reimplementing cvs_import. I would use the algorithms of cvs2svn to get a more or less consistent view of the cvs repository. Due to the nature of monotone, it should be easy to improve the algorithm to be able to handle subsequent imports. If I'm heading for such a rewrite, what should I be aware of? Would it be wise to store results from different processing passes in the normal monotone db? This would help subsequent imports a lot, of course. On the other hand you then have 'non-monotone-data' in your database, which you probably want to delete some day. What could be different to cvs2svn? I.e. you don't absolutely need to sort by date. Overlapping commits in CVS would better be handled as two heads which later got merged again in monotone. My goals with this are: * gain speed in subsequent imports * (correct) branch support What do you think? Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] cvs_import/sync questions
Emile Snyder schrieb: Hi all, Is it worth looking at cvs bugs on the main branch, or is the cvssync branch going to supersede it? Is cvssync close to merging? Is cvssync stable enough that I should try to figure out bugs on it, or is it still in flux and I should just let Christof do his thing? I have some things still in my TODO list for cvs_sync: - fix the current failure of t_cvspull_separate (should be easy) - fix the failure of t_cvspull_samedirname (should be easy) - make sure that I'm ignoring self modifying files (e.g. $Id$) well enough on commit - make the history reconstruction algo smarter so it does not reconstruct the full history but constricts to the last revision (for now, branches might cause more complex setups). This should speed up cvs_* on deep projects (without --since) by some orders of magnitude. [This would fix the first bug as well] - make cvs_push collect some intermediate changelogs when it commits a merge point (merge of X and Y is not too informative if you don't have the monotone database) far future ## - make cvs_push work with two children (select which path to take) - side branch support - cvs push into an empty repository? - import of a CVS working directory (so you don't have to contact the CVS server if you want to commit via monotone) Of course any help is appreciated (please coordinate! ;-) ). Perhaps a working tailor module (inter VCS syncing scripts) is the way to go (it already has darcs, svn, cvs, ~monotone~). I did not evaluate it (any reports welcome) or compared it to my cvs_push/pull. Christof signature.asc Description: OpenPGP digital signature ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] cvs_import/sync questions
Hi all, Is it worth looking at cvs bugs on the main branch, or is the cvssync branch going to supersede it? Is cvssync close to merging? Is cvssync stable enough that I should try to figure out bugs on it, or is it still in flux and I should just let Christof do his thing? thanks, -emile +-- Bother, said Pooh, Eeyore, ready two photon torpedoes and lock phasers on the Heffalump, Piglet, meet me in transporter room three. -- Robert Billing +-- signature.asc Description: This is a digitally signed message part ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
