Re: [fossil-users] fast-import crash (mark not declared)
On Fri, Jun 08, 2018 at 05:06:23PM +0300, Petr Ovtchenkov wrote: > During attempt to export fossil's repo (tcl, http://core.tcl.tk/tcl) to git I > face with > fast-import (fossil export --git --export-marks ../tcl-fossil/fossil.marks > ../tcl.fossil | git > fast-import --export-marks=../tcl-fossil/git.marks) crash: Hm. That repo has timewraps. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On Fri, 8 Jun 2018 10:27:51 -0400 Richard Hipp wrote: > On 6/8/18, Petr Ovtchenkov wrote: > > Hello, > > > > During attempt to export fossil's repo (tcl, http://core.tcl.tk/tcl) to git > > I face with > > fast-import (fossil export --git --export-marks ../tcl-fossil/fossil.marks > > ../tcl.fossil | git > > fast-import --export-marks=../tcl-fossil/git.marks) crash: > > What version of Fossil are you using? (What is the output of "fossil > version"?) # fossil version This is fossil version 2.6 [9718f3b078] 2018-05-04 12:56:42 UTC > Could it be that this problem was fixed by the check-in > shown in the links below? > > https://www.fossil-scm.org/fossil/info/c0a3e9ff6fbe3b48 > https://www.fossil-scm.org/fossil/timeline?c=c0a3e9ff6f=11 > The # ../fossil-repo/fossil version This is fossil version 2.6 [1eee6c70aa] 2018-06-08 09:41:20 UTC give me ../fossil-repo/fossil export --git --export-marks ../tcl-fossil/fossil.marks ../tcl.fossil | git fast-import --export-marks=../tcl-fossil/git.marks warning: Not updating refs/heads/ios (new tip 26fd9f86bc43fd2a2c195a70fd76a4502a81fb4d does not contain 29e5955df0857820f2cb89b4adf3af6efdd1e3fe) ... error: multiple updates for ref 'refs/tags/bg-tip-282' not allowed. /usr/libexec/git-core/git-fast-import statistics: - Alloc'd objects: 17 Total objects: 1296 (178198 duplicates ) blobs :0 ( 86813 duplicates 0 deltas of 0 attempts) trees : 757 ( 68483 duplicates619 deltas of 757 attempts) commits: 270 ( 22902 duplicates 0 deltas of 0 attempts) tags : 269 ( 0 duplicates 0 deltas of 0 attempts) Total branches: 702 ( 7115 loads ) marks:1048576 (109985 unique) atoms: 2615 Memory total: 10391 KiB pools: 2423 KiB objects: 7968 KiB - pack_report: getpagesize()= 4096 pack_report: core.packedGitWindowSize = 1073741824 pack_report: core.packedGitLimit = 35184372088832 pack_report: pack_used_ctr= 45324 pack_report: pack_mmap_calls =136 pack_report: pack_open_windows= 4 / 4 pack_report: pack_mapped = 506387095 / 506387095 - i.e. success! Thanks! Just note for followers: don't forget to do git gc --aggressive --prune=all Before: du -hs . 1.6G . After: du -hs . 103M . Original fossil's repo is 212M Jun 8 18:01 ../tcl.fossil -- - ptr ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On 6/8/18, Petr Ovtchenkov wrote: > Hello, > > During attempt to export fossil's repo (tcl, http://core.tcl.tk/tcl) to git > I face with > fast-import (fossil export --git --export-marks ../tcl-fossil/fossil.marks > ../tcl.fossil | git > fast-import --export-marks=../tcl-fossil/git.marks) crash: What version of Fossil are you using? (What is the output of "fossil version"?) Could it be that this problem was fixed by the check-in shown in the links below? https://www.fossil-scm.org/fossil/info/c0a3e9ff6fbe3b48 https://www.fossil-scm.org/fossil/timeline?c=c0a3e9ff6f=11 -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On Fri, 17 Feb 2017 00:19:08 -0600 Artur Shepilkowrote: > To summarize the findings: > - Sqlite Fossil repo has a number of special cases that do not export > directly, resulting in "git fast-import" crash. > - To accomplish the export, one needs to apply the following fixes to > the __local__ clone of the Sqlite Fossil repo: > fossil set autosync off > fossil reparent 590d4ac1ee0db824 eb7a544fe49d1626bacecfe53ddc03fe082e3243 > fossil amend 655991ec8a --date "2010-09-28 19:16:47" > fossil amend 1ef0dc9328 --date "2010-09-29 13:31:00" > > Then initiate the fossil export process anew. > Please take a note of NOT pushing any of these changes into the Sqlite > remote repo. Conformed: fossil set autosync off fossil reparent 590d4ac1ee0db824 eb7a544fe49d1626bacecfe53ddc03fe082e3243 fossil amend 655991ec8a --date "2010-09-28 19:16:47" fossil amend 1ef0dc9328 --date "2010-09-29 13:31:00" cd ../sqlite-git && fossil export --git \ --export-marks ../sqlite-fossil/fossil.marks ../sqlite.fossil | \ git fast-import --import-marks=../sqlite-fossil/git.marks --export-marks=../sqlite-fossil/git.marks Give error: multiple updates for ref 'refs/tags/experimental' not allowed. git-fast-import statistics: - Alloc'd objects: 115000 Total objects:39137 ( 74446 duplicates ) blobs :0 ( 49034 duplicates 0 deltas of 0 attempts) trees :27819 ( 18184 duplicates 25217 deltas of 25356 attempts) commits:11125 ( 7228 duplicates 0 deltas of 0 attempts) tags : 193 ( 0 duplicates 0 deltas of 0 attempts) Total branches: 656 ( 1220 loads ) marks:1048576 ( 67387 unique) atoms: 1879 Memory total: 7876 KiB pools: 2485 KiB objects: 5390 KiB - pack_report: getpagesize()= 4096 pack_report: core.packedGitWindowSize = 1073741824 pack_report: core.packedGitLimit = 8589934592 pack_report: pack_used_ctr= 64171 pack_report: pack_mmap_calls = 1817 pack_report: pack_open_windows= 2 / 2 pack_report: pack_mapped = 923989449 / 923989449 - i.e. success. Don't forget to run git gc --aggressive --prune=all Before: du -hs . 884M . After: du -hs . 33M . Thanks! -- - ptr ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Hi, On 2017-02-17 at 07:37 -0500 Richard Hippwrote: >On 2/17/17, Rafal Bisingier wrote: >> >> Out of curiosity: what will happen, when commited parent have timestamp >> far away in future? > >You could move the defective commit onto a branch (a branch named >"mistake" is a popular choice). Or you could change the date on the >defective commit. Thank you for quick answer. Again. ;-) -- Greetings Rafal Bisingier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On 2/17/17, Rafal Bisingierwrote: > > Out of curiosity: what will happen, when commited parent have timestamp > far away in future? You could move the defective commit onto a branch (a branch named "mistake" is a popular choice). Or you could change the date on the defective commit. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Hi, On 2017-02-17 at 07:03 -0500 Richard Hippwrote: >On 2/17/17, Richard Hipp wrote: >> On 2/17/17, Rafal Bisingier wrote: >>> >>> Shouldn't Fossil at least warn user trying to commit a child wit >>> timestamp earlier than parent? >> >> I think it does that, now. > >Confirmed: That fix went in 7 years ago: >https://www.fossil-scm.org/fossil/info/8fdac87b688c3ea0 It's even harder than I proposed - abort instead of warning. Thank you for confirmation. Out of curiosity: what will happen, when commited parent have timestamp far away in future? Do we have to skew our clock to commit a child? Or is there some mechanism to get around it without breaking next commit? -- Greetings Rafal Bisingier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On 2/17/17, Richard Hippwrote: > On 2/17/17, Rafal Bisingier wrote: >> >> Shouldn't Fossil at least warn user trying to commit a child wit >> timestamp earlier than parent? > > I think it does that, now. Confirmed: That fix went in 7 years ago: https://www.fossil-scm.org/fossil/info/8fdac87b688c3ea0 -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On Fri, 17 Feb 2017 00:04:29 -0600 Artur Shepilkowrote: > ... > I just realized that the check-in date may be altered via "fossil > amend --date" or from UI via check-in's edit submenu. > This will add the tag "date=" and update the check-ins MTIME. > > Looks like both timeline and export indeed pick up the updated MTIME. > > I looked up the sqlite's cases that crashed export (655991ec8a781d67, > 1ef0dc9328f47506) and both seem to have a corrected date tag, yet the > corresponding event.MTIME is not updated, i.e. MTIME=OTIME. That's why > this had no expected effect. I guess, it's on purpose as you > mentioned, for testing. If I understand correctly, objects graph depends upon wall time, right? If so, this looks as serious defect in design. -- WBR, - ptr ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Hi On Fri, 17 Feb 2017 at 06:07 -0500 Richard Hippwrote: > On 2/17/17, Rafal Bisingier wrote: > > > > Shouldn't Fossil at least warn user trying to commit a child wit > > timestamp earlier than parent? > > I think it does that, now. The timewarps all happened a while ago. Oh, then sorry for the noise... ;-) -- Greetings Rafal Bisingier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On 2/17/17, Rafal Bisingierwrote: > > Shouldn't Fossil at least warn user trying to commit a child wit > timestamp earlier than parent? I think it does that, now. The timewarps all happened a while ago. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Hi On Thu, 16 Feb 2017 at 14:15 -0800 Ross Berteigwrote: > Sure, we'd all love it if everyone kept their clocks in sync, but in the real > world skew is simply going to happen. In the absence of a central server to > provide the authoritative time stamp, a DVCS like both Fossil and Git has no > real choice other than to trust the time provided by each local machine as > authoritative enough to document each commit. > > If commits from each user of a project end up lined up in order on the > timeline, that is just a happy side effect of people keeping their clocks > synchronized. > > Having a parent committed after a child does look odd, but I think it has to > be simply accepted as part of the record, warts and all, of the work on the > project. Shouldn't Fossil at least warn user trying to commit a child wit timestamp earlier than parent? I suppose that in the moment of commit fossil is able to check this two timestamps (system time and ts of parent checkin) and compare it? I'd even say that in case of local time misconfiguration (when commiting child will end up with earlier timestamp than it's parent) it's possible to semi-automagically "fix" this by setting "computed timestamp" a the new commit timestamp. or better by simply executing what "fossil amend --date" do. -- Greetings Rafal Bisingier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
To summarize the findings: - Sqlite Fossil repo has a number of special cases that do not export directly, resulting in "git fast-import" crash. - To accomplish the export, one needs to apply the following fixes to the __local__ clone of the Sqlite Fossil repo: fossil set autosync off fossil reparent 590d4ac1ee0db824 eb7a544fe49d1626bacecfe53ddc03fe082e3243 fossil amend 655991ec8a --date "2010-09-28 19:16:47" fossil amend 1ef0dc9328 --date "2010-09-29 13:31:00" Then initiate the fossil export process anew. Please take a note of NOT pushing any of these changes into the Sqlite remote repo. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Richard, Thank you for the details. I just realized that the check-in date may be altered via "fossil amend --date" or from UI via check-in's edit submenu. This will add the tag "date=" and update the check-ins MTIME. Looks like both timeline and export indeed pick up the updated MTIME. I looked up the sqlite's cases that crashed export (655991ec8a781d67, 1ef0dc9328f47506) and both seem to have a corrected date tag, yet the corresponding event.MTIME is not updated, i.e. MTIME=OTIME. That's why this had no expected effect. I guess, it's on purpose as you mentioned, for testing. No changes made to the remote repo. When these are updated locally, the export completes successfully. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On 2/16/17, Artur Shepilkowrote: > Is this some undocumented Fossil feature which is supposed to allow > overriding of the commit timestamp? Yes. You can change the displayed date of any check-in using a date tag. This feature is designed to fix check-ins that were made from a system with a mis-set system clock. The date tag is used to resolve "time-warps" such as https://www.sqlite.org/src/timeline?c=3f30f00a - (Please do not attempt to resolve that particular timewarp - it is one we use for testing.) The original check-in date is preserved and the new date is shown as an edit. For example in https://www.sqlite.org/src/info/e37137376a2b2306 you can see both the original check-in date and the revised check-in date set by the tag at https://www.sqlite.org/src/info/e864e83c133bfd55 > Nevertheless, in either case the date tag values are ignored in the > timeline for both commits; the original timestamp is being used. > > If date tag is indeed a way to tune the timeline, perhaps this should > be factored in both the timeline output and the selection of export > recordset. The revised date (according to the most recent and current date tag) is the date that is in the EVENT table in the MTIME field. That the EVENT.MTIME field is what drives the timeline display. And it should also be driving the git export, I suppose. The original date of the check-in is shown in the EVENT.OMTIME field > > > > On Thu, Feb 16, 2017 at 4:54 PM, Artur Shepilko wrote: >> Indeed, in export.c the commits are lined up chronologically. >> >> There's a way to try to catch such inconsistencies by tracking when >> the "from :mark" gets freshly incremented (that is the parent has not >> been exported yet): >> src/export.c::~581 >> int next_mark = unused_mark; >> zMark = mark_name_from_rid(pid, _mark); >> if ( unused_mark == next_mark+1 ){ >> /* Handle a "future" parent commit */ >> } >> printf("from %s\n", zMark); >> >> Still, this will need a restructuring of the export flow. >> This may also become a run-away problem, once we start chasing the >> parent commit-chain through potential merges etc. > ___ > fossil-users mailing list > fossil-users@lists.fossil-scm.org > http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users > -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Hmm, another look reveals some surprises: The "future" parent commit (https://www.sqlite.org/src/info/655991ec8a781d67) also has a "date" tag added, apparently in attempt to rectify the timeline: date=2010-09-28 19:16:47 added by [5fd7e19e] on 2010-09-28 19:19:21 Similar situation is with the other "future" commit I mentioned. Is this some undocumented Fossil feature which is supposed to allow overriding of the commit timestamp? Nevertheless, in either case the date tag values are ignored in the timeline for both commits; the original timestamp is being used. If date tag is indeed a way to tune the timeline, perhaps this should be factored in both the timeline output and the selection of export recordset. On Thu, Feb 16, 2017 at 4:54 PM, Artur Shepilkowrote: > Indeed, in export.c the commits are lined up chronologically. > > There's a way to try to catch such inconsistencies by tracking when > the "from :mark" gets freshly incremented (that is the parent has not > been exported yet): > src/export.c::~581 > int next_mark = unused_mark; > zMark = mark_name_from_rid(pid, _mark); > if ( unused_mark == next_mark+1 ){ > /* Handle a "future" parent commit */ > } > printf("from %s\n", zMark); > > Still, this will need a restructuring of the export flow. > This may also become a run-away problem, once we start chasing the > parent commit-chain through potential merges etc. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Indeed, in export.c the commits are lined up chronologically. There's a way to try to catch such inconsistencies by tracking when the "from :mark" gets freshly incremented (that is the parent has not been exported yet): src/export.c::~581 int next_mark = unused_mark; zMark = mark_name_from_rid(pid, _mark); if ( unused_mark == next_mark+1 ){ /* Handle a "future" parent commit */ } printf("from %s\n", zMark); Still, this will need a restructuring of the export flow. This may also become a run-away problem, once we start chasing the parent commit-chain through potential merges etc. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On Thu, Feb 16, 2017 at 02:15:25PM -0800, Ross Berteig wrote: > I'd argue the problem here is with Git. Actually, git is not at fault here. We do the export chronologically as it is much easier to do than computing a topologically sorted output. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
On 2/16/2017 1:11 PM, Artur Shepilko wrote: This fix seems to allow the git fast-import to proceed past this ... until it finds another inconsistency, unrelated to the one above. This time it's with commit: https://www.sqlite.org/src/info/3f30f00a384d2358 Again, looking at the timeline https://www.sqlite.org/src/timeline?r=trunk=2010-09-28 There's a unusual timeline zig-zag on 2010-09-29, where a "future" commit (https://www.sqlite.org/src/info/655991ec8a781d67) is a parent to a "past" commit above. Apparently, there was some time skew which crept in. Same happened with the another subsequent "future" commit (https://www.sqlite.org/src/info/1ef0dc9328f47506, parent to "past" https://www.sqlite.org/src/info/f34dc54d46d05adf) . The timestamps for these "future" parent commits are supposed to be prior to timestamps of their children commits. Not sure what's a right way to fix the skewed timestamps, no handy command for this. Prehaps through "fossil sqlite3" direct surgery, but one needs to know how not to kill the patient in the process... Fossil hid some skeletons in the Sqlite's closet, Git just found some. Let's bring them to life :) I'd argue the problem here is with Git. Sure, we'd all love it if everyone kept their clocks in sync, but in the real world skew is simply going to happen. In the absence of a central server to provide the authoritative time stamp, a DVCS like both Fossil and Git has no real choice other than to trust the time provided by each local machine as authoritative enough to document each commit. If commits from each user of a project end up lined up in order on the timeline, that is just a happy side effect of people keeping their clocks synchronized. Having a parent committed after a child does look odd, but I think it has to be simply accepted as part of the record, warts and all, of the work on the project. I know that Git users like to be able to polish those warts off. Fossil doesn't care. And shouldn't care. The timestamp of each checkin is stored in the D card of the checkin's manifest, so it is not possible to edit it directly without changing the SHA1 hash of the checkin. -- Ross Berteig r...@cheshireeng.com Cheshire Engineering Corp. http://www.CheshireEng.com/ +1 626 303 1602 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
Looking closer at the timeline and manifests shows that the inconsistency is with https://www.sqlite.org/src/info/590d4ac1ee0db824 which lists as its parent a tag's manifest (https://www.sqlite.org/src/info/f228c7ca0682c370). Normally the link should be to tag's target object, which in this case is https://www.sqlite.org/src/info/eb7a544fe49d1626. Now, this could be fixed with "fossil reparent" command (the manifest won't be changed): fossil reparent 590d4ac1ee0db824 eb7a544fe49d1626bacecfe53ddc03fe082e3243 This fix seems to allow the git fast-import to proceed past this ... until it finds another inconsistency, unrelated to the one above. This time it's with commit: https://www.sqlite.org/src/info/3f30f00a384d2358 Again, looking at the timeline https://www.sqlite.org/src/timeline?r=trunk=2010-09-28 There's a unusual timeline zig-zag on 2010-09-29, where a "future" commit (https://www.sqlite.org/src/info/655991ec8a781d67) is a parent to a "past" commit above. Apparently, there was some time skew which crept in. Same happened with the another subsequent "future" commit (https://www.sqlite.org/src/info/1ef0dc9328f47506, parent to "past" https://www.sqlite.org/src/info/f34dc54d46d05adf) . The timestamps for these "future" parent commits are supposed to be prior to timestamps of their children commits. Not sure what's a right way to fix the skewed timestamps, no handy command for this. Prehaps through "fossil sqlite3" direct surgery, but one needs to know how not to kill the patient in the process... Fossil hid some skeletons in the Sqlite's closet, Git just found some. Let's bring them to life :) ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fast-import crash (mark not declared)
> commit refs/heads/shunning_error > mark :56263 > committer drh 1257360677 + > data 24 > * from :56264 This is a curious case: looks like the branch:"shunning_error" starts off a tag (https://www.sqlite.org/src/info/f228c7ca0682c370f8c43) That is its parent is a commit edit/amend event, unlike normal branches which are children of other commits. The from mark (56264) seems to be the "next-value" of the mark sequence, which export.c::mark_name_from_rid() returns in case when look-up for the checkin id (parent in this case) finds nothing. Obviously from-mark is expected to be previously declared. This case seems like a detached/free-standing branch. From Git fast-import format description: "Omitting the from command in the first commit of a new branch will cause fast-import to create that commit with no ancestor. This tends to be desired only for the initial commit of a project. If the frontend creates all files from scratch when making a new branch, a merge command may be used instead of from to start the commit with an empty tree." Apparently this is a result of some unusual fix-up (https://www.sqlite.org/src/info/eb7a544fe49d1626) I thought with Fossil the repository tree stems from the root/initial commit and technically all branches have a common ancestor commit (root), so there're no detached branches. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users