Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Bryan Drewery]

On 12/4/2015 10:49 AM, Ulrich Spörlein wrote:
> 2015-11-08 12:06 GMT+01:00 Ulrich Spörlein :
>> 2015-11-08 11:32 GMT+01:00 Ulrich Spörlein :
>>> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein :
>
 Uli,

 One of the biggest concerns I've heard from folks using FreeBSD's git 
 mirror
 is that the hashes can change.

 I have a question about this.   Is it possible to keep track of what the
 "official" git mirror (on github) is doing and keep that as a log.  Then
 that log can be used to replay commits when there is a divergence problem.

 What I'm basically saying is that let's take this small example:

 importer is working fine @rev 1
 imports 1
 imports 10001
 imports 10002
 something happens to importer to give indeterminate shas.
 imports 10003 - sha is "unstable" sha3
 imports 10004 - sha is "unstable" sha4
 imports 10005 - sha is "unstable" sha5
 imports 10006 - sha is "unstable" sha6
 importer is fixed


 At this point normally we'd rewind the importer to 10002 and then force
 update the affected branches.

 My question is... can the imports of 10003, 10004, 10005 and 10006 be put
 into the importer such that any "mirror site" that re-does the import using
 the most up to date importer will get the same shas.

 That would allow to proceed with 10007, etc without force pushing.

 This should be possible based on querying "git" for the meta data 
 associated
 with sha3..sha6 and then forcing those commits to have the same meta data.

 This would eliminate the concern about shas in the mirror changing that 
 I've
 heard.
>>>
>>> The goal of the conversion is that everyone can re-do the conversion
>>> in their basement and come up with the same history and checksums.
>>> This was not the case when I first started, as there was some
>>> non-deterministic hash structure being used in svn2git. This was fixed
>>> in the code and then all converter runs produced the very same
>>> results.
>>>
>>> The scenario that we have right now, is that one of the merge commits
>>> done about two weeks ago is being handled different by svn2git w/ svn
>>> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's
>>> behavior changed to cause this. I'm afraid I also swapped out all my
>>> knowledge about svn2git internals and will have to redo this all from
>>> scratch :/
>>>
>>> Your suggestion could only work, if we hard-code this svn revision
>>> special handling into svn2git, either in the code or by providing more
>>> mappings and rules to the process. svn2git should run hermetic and not
>>> poke at github's commits to see how things were handled in the past.
>>> It has to be self-sufficient and must not depend on github.
>>>
>>> This would also only work, if the "breakage" window was very small,
>>> but it is already about two weeks long and will surely increase till I
>>> find the proper fix.
>>>
>>> So, to take a stand here: this sort of kludge is unlikely to ever
>>> happen. Git commit hashes *might* change in the future. I really don't
>>> see how this is a big deal anyway.  It happened once and I'm trying to
>>> have it never happen again. But why are people afraid of this
>>> happening? Every "official" git commit is tagged with a SVN revision
>>> and the contents of those revisions are obviously correct (just not
>>> the ancestry and the commit objects, possibly). So it would be easy to
>>> write a script that replays VendorA's git history and swaps out the
>>> new official commits for the old official commits. There would be no
>>> merge conflicts.
>>>
>>> I can see how this would be annoying if you have 100 developers and
>>> dozens of branches that are far from mainline FreeBSD. But I'm sure
>>> these companies that depend on git will come forward and donate some
>>> of their developer manpower to help me with keeping the converter
>>> stable/deterministic. Right? Right? :) :)
>>>
>>> Cheers,
>>> Uli
>>
>> Quick update: doc is so far unaffected by svn 1.9, but for ports, the
>> drift happened as of Jul 18, so you'd need to special case a lot of
>> commits.
>>
>> Here's the same commit, and the difference between 1.8 and 1.9:
>>
>> % git cat-file commit 803795d
>> tree 7fc83aba022834da5c218114b09ad4640735bcc0
>> parent c96fb0418e545a569b5975b4d878a30a948c29d5
>> author olgeni  1437203525 +
>> committer olgeni  1437203525 +
>>
>> Upgrade to version 0.4.1.
>> % git cat-file commit 61ca43b
>> tree 7fc83aba022834da5c218114b09ad4640735bcc0
>> parent c96fb0418e545a569b5975b4d878a30a948c29d5
>> author olgeni  1437203529 +
>> committer olgeni  1437203529 +
>>
>> Upgrade to version 0.4.1.
>>
>>
>> In case you don't see it, there's a 4s difference in the timestamps
>> for authoring and committing. 

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

2015-11-08 12:06 GMT+01:00 Ulrich Spörlein :
> 2015-11-08 11:32 GMT+01:00 Ulrich Spörlein :
>> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein :

>>> Uli,
>>>
>>> One of the biggest concerns I've heard from folks using FreeBSD's git mirror
>>> is that the hashes can change.
>>>
>>> I have a question about this.   Is it possible to keep track of what the
>>> "official" git mirror (on github) is doing and keep that as a log.  Then
>>> that log can be used to replay commits when there is a divergence problem.
>>>
>>> What I'm basically saying is that let's take this small example:
>>>
>>> importer is working fine @rev 1
>>> imports 1
>>> imports 10001
>>> imports 10002
>>> something happens to importer to give indeterminate shas.
>>> imports 10003 - sha is "unstable" sha3
>>> imports 10004 - sha is "unstable" sha4
>>> imports 10005 - sha is "unstable" sha5
>>> imports 10006 - sha is "unstable" sha6
>>> importer is fixed
>>>
>>>
>>> At this point normally we'd rewind the importer to 10002 and then force
>>> update the affected branches.
>>>
>>> My question is... can the imports of 10003, 10004, 10005 and 10006 be put
>>> into the importer such that any "mirror site" that re-does the import using
>>> the most up to date importer will get the same shas.
>>>
>>> That would allow to proceed with 10007, etc without force pushing.
>>>
>>> This should be possible based on querying "git" for the meta data associated
>>> with sha3..sha6 and then forcing those commits to have the same meta data.
>>>
>>> This would eliminate the concern about shas in the mirror changing that I've
>>> heard.
>>
>> The goal of the conversion is that everyone can re-do the conversion
>> in their basement and come up with the same history and checksums.
>> This was not the case when I first started, as there was some
>> non-deterministic hash structure being used in svn2git. This was fixed
>> in the code and then all converter runs produced the very same
>> results.
>>
>> The scenario that we have right now, is that one of the merge commits
>> done about two weeks ago is being handled different by svn2git w/ svn
>> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's
>> behavior changed to cause this. I'm afraid I also swapped out all my
>> knowledge about svn2git internals and will have to redo this all from
>> scratch :/
>>
>> Your suggestion could only work, if we hard-code this svn revision
>> special handling into svn2git, either in the code or by providing more
>> mappings and rules to the process. svn2git should run hermetic and not
>> poke at github's commits to see how things were handled in the past.
>> It has to be self-sufficient and must not depend on github.
>>
>> This would also only work, if the "breakage" window was very small,
>> but it is already about two weeks long and will surely increase till I
>> find the proper fix.
>>
>> So, to take a stand here: this sort of kludge is unlikely to ever
>> happen. Git commit hashes *might* change in the future. I really don't
>> see how this is a big deal anyway.  It happened once and I'm trying to
>> have it never happen again. But why are people afraid of this
>> happening? Every "official" git commit is tagged with a SVN revision
>> and the contents of those revisions are obviously correct (just not
>> the ancestry and the commit objects, possibly). So it would be easy to
>> write a script that replays VendorA's git history and swaps out the
>> new official commits for the old official commits. There would be no
>> merge conflicts.
>>
>> I can see how this would be annoying if you have 100 developers and
>> dozens of branches that are far from mainline FreeBSD. But I'm sure
>> these companies that depend on git will come forward and donate some
>> of their developer manpower to help me with keeping the converter
>> stable/deterministic. Right? Right? :) :)
>>
>> Cheers,
>> Uli
>
> Quick update: doc is so far unaffected by svn 1.9, but for ports, the
> drift happened as of Jul 18, so you'd need to special case a lot of
> commits.
>
> Here's the same commit, and the difference between 1.8 and 1.9:
>
> % git cat-file commit 803795d
> tree 7fc83aba022834da5c218114b09ad4640735bcc0
> parent c96fb0418e545a569b5975b4d878a30a948c29d5
> author olgeni  1437203525 +
> committer olgeni  1437203525 +
>
> Upgrade to version 0.4.1.
> % git cat-file commit 61ca43b
> tree 7fc83aba022834da5c218114b09ad4640735bcc0
> parent c96fb0418e545a569b5975b4d878a30a948c29d5
> author olgeni  1437203529 +
> committer olgeni  1437203529 +
>
> Upgrade to version 0.4.1.
>
>
> In case you don't see it, there's a 4s difference in the timestamps
> for authoring and committing. Here's the original:
>
> % svn log -vc392405 svn://svn.freebsd.org/ports
> 
> r392405 | olgeni | 

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[NGie Cooper]

Hi Ulrich,
This might be one of the reasons why the git converter was broken
recently: https://reviews.reviewboard.org/r/7617/diff (it seems that
svn is now reporting "nonexistent" for new files instead of "Revision
0"). It broke rbt with svn 1.9 :(...
Thanks!
-NGie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Alfred Perlstein]

Lars,

Try to remove .git/gc.log then re-run fetch.

If that doesn't work then move ".git/refs/remotes/origin/HEAD" to backup 
location outside of your .git directory and try again.


-Alfred

On 11/11/15 4:03 AM, Eggert, Lars wrote:

Hi,

I just got this error when fetching from remote; related?

[elars@laurel: ~/src] git fetch --all
Fetching origin
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Fetching upstream
remote: Counting objects: 557, done.
remote: Compressing objects: 100% (543/543), done.
remote: Total 557 (delta 213), reused 2 (delta 2), pack-reused 0
Receiving objects: 100% (557/557), 1.15 MiB | 433.00 KiB/s, done.
Resolving deltas: 100% (213/213), completed with 2 local objects.
 From github.com:/freebsd/freebsd
b4eb11a..3eb0ea4  master -> upstream/master
f147893..9c319c0  stable/10  -> upstream/stable/10
e901edd..b3c9fd2  stable/8   -> upstream/stable/8
81ab2b1..2fc7a9a  stable/9   -> upstream/stable/9
c2c933c..cc76737  svn_head   -> upstream/svn_head
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

fatal: bad object refs/remotes/origin/HEAD
error: failed to run repack

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

fatal: bad object refs/remotes/origin/HEAD
error: failed to run repack

Lars


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Eggert, Lars]

Hi,

I just got this error when fetching from remote; related?

[elars@laurel: ~/src] git fetch --all
Fetching origin
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Fetching upstream
remote: Counting objects: 557, done.
remote: Compressing objects: 100% (543/543), done.
remote: Total 557 (delta 213), reused 2 (delta 2), pack-reused 0
Receiving objects: 100% (557/557), 1.15 MiB | 433.00 KiB/s, done.
Resolving deltas: 100% (213/213), completed with 2 local objects.
From github.com:/freebsd/freebsd
   b4eb11a..3eb0ea4  master -> upstream/master
   f147893..9c319c0  stable/10  -> upstream/stable/10
   e901edd..b3c9fd2  stable/8   -> upstream/stable/8
   81ab2b1..2fc7a9a  stable/9   -> upstream/stable/9
   c2c933c..cc76737  svn_head   -> upstream/svn_head
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

fatal: bad object refs/remotes/origin/HEAD
error: failed to run repack

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

fatal: bad object refs/remotes/origin/HEAD
error: failed to run repack

Lars


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

2015-11-08 21:05 GMT+01:00 Oliver Pinter :
> Hi Uli!
>
> I can not find your original svn2git repo in gitorius
> (https://gitorious.org/ is down) , could you please the source code
> somewhere to git-repo? For example github.com/freebsd/svn2git?

You mean http://svn.freebsd.org/base/user/uqs/git_conv/
:)

This doesn't have the v1.9 fixes though. I only notices this week that
gitorious is down, but we also have
https://github.com/freebsd/svn2git/commits/master for a while now.
I'll push updated code there.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

2015-11-08 11:32 GMT+01:00 Ulrich Spörlein :
> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein :
>>>
>> Uli,
>>
>> One of the biggest concerns I've heard from folks using FreeBSD's git mirror
>> is that the hashes can change.
>>
>> I have a question about this.   Is it possible to keep track of what the
>> "official" git mirror (on github) is doing and keep that as a log.  Then
>> that log can be used to replay commits when there is a divergence problem.
>>
>> What I'm basically saying is that let's take this small example:
>>
>> importer is working fine @rev 1
>> imports 1
>> imports 10001
>> imports 10002
>> something happens to importer to give indeterminate shas.
>> imports 10003 - sha is "unstable" sha3
>> imports 10004 - sha is "unstable" sha4
>> imports 10005 - sha is "unstable" sha5
>> imports 10006 - sha is "unstable" sha6
>> importer is fixed
>>
>>
>> At this point normally we'd rewind the importer to 10002 and then force
>> update the affected branches.
>>
>> My question is... can the imports of 10003, 10004, 10005 and 10006 be put
>> into the importer such that any "mirror site" that re-does the import using
>> the most up to date importer will get the same shas.
>>
>> That would allow to proceed with 10007, etc without force pushing.
>>
>> This should be possible based on querying "git" for the meta data associated
>> with sha3..sha6 and then forcing those commits to have the same meta data.
>>
>> This would eliminate the concern about shas in the mirror changing that I've
>> heard.
>
> The goal of the conversion is that everyone can re-do the conversion
> in their basement and come up with the same history and checksums.
> This was not the case when I first started, as there was some
> non-deterministic hash structure being used in svn2git. This was fixed
> in the code and then all converter runs produced the very same
> results.
>
> The scenario that we have right now, is that one of the merge commits
> done about two weeks ago is being handled different by svn2git w/ svn
> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's
> behavior changed to cause this. I'm afraid I also swapped out all my
> knowledge about svn2git internals and will have to redo this all from
> scratch :/
>
> Your suggestion could only work, if we hard-code this svn revision
> special handling into svn2git, either in the code or by providing more
> mappings and rules to the process. svn2git should run hermetic and not
> poke at github's commits to see how things were handled in the past.
> It has to be self-sufficient and must not depend on github.
>
> This would also only work, if the "breakage" window was very small,
> but it is already about two weeks long and will surely increase till I
> find the proper fix.
>
> So, to take a stand here: this sort of kludge is unlikely to ever
> happen. Git commit hashes *might* change in the future. I really don't
> see how this is a big deal anyway.  It happened once and I'm trying to
> have it never happen again. But why are people afraid of this
> happening? Every "official" git commit is tagged with a SVN revision
> and the contents of those revisions are obviously correct (just not
> the ancestry and the commit objects, possibly). So it would be easy to
> write a script that replays VendorA's git history and swaps out the
> new official commits for the old official commits. There would be no
> merge conflicts.
>
> I can see how this would be annoying if you have 100 developers and
> dozens of branches that are far from mainline FreeBSD. But I'm sure
> these companies that depend on git will come forward and donate some
> of their developer manpower to help me with keeping the converter
> stable/deterministic. Right? Right? :) :)
>
> Cheers,
> Uli

Quick update: doc is so far unaffected by svn 1.9, but for ports, the
drift happened as of Jul 18, so you'd need to special case a lot of
commits.

Here's the same commit, and the difference between 1.8 and 1.9:

% git cat-file commit 803795d
tree 7fc83aba022834da5c218114b09ad4640735bcc0
parent c96fb0418e545a569b5975b4d878a30a948c29d5
author olgeni  1437203525 +
committer olgeni  1437203525 +

Upgrade to version 0.4.1.
% git cat-file commit 61ca43b
tree 7fc83aba022834da5c218114b09ad4640735bcc0
parent c96fb0418e545a569b5975b4d878a30a948c29d5
author olgeni  1437203529 +
committer olgeni  1437203529 +

Upgrade to version 0.4.1.


In case you don't see it, there's a 4s difference in the timestamps
for authoring and committing. Here's the original:

% svn log -vc392405 svn://svn.freebsd.org/ports

r392405 | olgeni | 2015-07-18 09:12:05 +0200 (Sat, 18 Jul 2015) | 2 lines
Changed paths:
   M /head/www/elixir-maru/Makefile
   M /head/www/elixir-maru/distinfo

Upgrade to version 0.4.1.

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[David Chisnall]

On 8 Nov 2015, at 02:32, Ulrich Spörlein  wrote:
> 
> 
> Git commit hashes *might* change in the future. I really don't
> see how this is a big deal anyway.  It happened once and I'm trying to
> have it never happen again. But why are people afraid of this
> happening? Every "official" git commit is tagged with a SVN revision
> and the contents of those revisions are obviously correct (just not
> the ancestry and the commit objects, possibly). So it would be easy to
> write a script that replays VendorA's git history and swaps out the
> new official commits for the old official commits. There would be no
> merge conflicts.

Git commit hashes must not change, or we completely destroy the utility of the 
git mirror for all downstream users.  You can no longer do a merge from 
upstream git if the hashes in your local clone do not match the hashes 
downstream.  Your answer of expecting every downstream user of FreeBSD’s git 
repo (GitHub tracks over 600 of them, there are likely more that are using 
private clones) to write a script to fix the mess for themselves is completely 
unacceptable.

If there has been a window in which incorrect hashes were generated, this can 
be fixed by moving that to a branch, doing the correct thing in master, and 
then using git-imerge in rebase-with-history mode.  After the point of the fix, 
there will be a single set of commits, but before that there will be two 
options as parents, one for each version of the export.

Please remember that a guarantee of not changing the hashes of the history was 
one of the conditions that Core had for promoting this to an official FreeBSD 
service.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Oliver Pinter]

On Sun, Nov 8, 2015 at 12:06 PM, Ulrich Spörlein  wrote:
> 2015-11-08 11:32 GMT+01:00 Ulrich Spörlein :
>> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein :

>>> Uli,
>>>
>>> One of the biggest concerns I've heard from folks using FreeBSD's git mirror
>>> is that the hashes can change.
>>>
>>> I have a question about this.   Is it possible to keep track of what the
>>> "official" git mirror (on github) is doing and keep that as a log.  Then
>>> that log can be used to replay commits when there is a divergence problem.
>>>
>>> What I'm basically saying is that let's take this small example:
>>>
>>> importer is working fine @rev 1
>>> imports 1
>>> imports 10001
>>> imports 10002
>>> something happens to importer to give indeterminate shas.
>>> imports 10003 - sha is "unstable" sha3
>>> imports 10004 - sha is "unstable" sha4
>>> imports 10005 - sha is "unstable" sha5
>>> imports 10006 - sha is "unstable" sha6
>>> importer is fixed
>>>
>>>
>>> At this point normally we'd rewind the importer to 10002 and then force
>>> update the affected branches.
>>>
>>> My question is... can the imports of 10003, 10004, 10005 and 10006 be put
>>> into the importer such that any "mirror site" that re-does the import using
>>> the most up to date importer will get the same shas.
>>>
>>> That would allow to proceed with 10007, etc without force pushing.
>>>
>>> This should be possible based on querying "git" for the meta data associated
>>> with sha3..sha6 and then forcing those commits to have the same meta data.
>>>
>>> This would eliminate the concern about shas in the mirror changing that I've
>>> heard.
>>
>> The goal of the conversion is that everyone can re-do the conversion
>> in their basement and come up with the same history and checksums.
>> This was not the case when I first started, as there was some
>> non-deterministic hash structure being used in svn2git. This was fixed
>> in the code and then all converter runs produced the very same
>> results.
>>
>> The scenario that we have right now, is that one of the merge commits
>> done about two weeks ago is being handled different by svn2git w/ svn
>> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's
>> behavior changed to cause this. I'm afraid I also swapped out all my
>> knowledge about svn2git internals and will have to redo this all from
>> scratch :/
>>
>> Your suggestion could only work, if we hard-code this svn revision
>> special handling into svn2git, either in the code or by providing more
>> mappings and rules to the process. svn2git should run hermetic and not
>> poke at github's commits to see how things were handled in the past.
>> It has to be self-sufficient and must not depend on github.
>>
>> This would also only work, if the "breakage" window was very small,
>> but it is already about two weeks long and will surely increase till I
>> find the proper fix.
>>
>> So, to take a stand here: this sort of kludge is unlikely to ever
>> happen. Git commit hashes *might* change in the future. I really don't
>> see how this is a big deal anyway.  It happened once and I'm trying to
>> have it never happen again. But why are people afraid of this
>> happening? Every "official" git commit is tagged with a SVN revision
>> and the contents of those revisions are obviously correct (just not
>> the ancestry and the commit objects, possibly). So it would be easy to
>> write a script that replays VendorA's git history and swaps out the
>> new official commits for the old official commits. There would be no
>> merge conflicts.
>>
>> I can see how this would be annoying if you have 100 developers and
>> dozens of branches that are far from mainline FreeBSD. But I'm sure
>> these companies that depend on git will come forward and donate some
>> of their developer manpower to help me with keeping the converter
>> stable/deterministic. Right? Right? :) :)
>>
>> Cheers,
>> Uli
>


Hi Uli!

I can not find your original svn2git repo in gitorius
(https://gitorious.org/ is down) , could you please the source code
somewhere to git-repo? For example github.com/freebsd/svn2git?

> Quick update: doc is so far unaffected by svn 1.9, but for ports, the
> drift happened as of Jul 18, so you'd need to special case a lot of
> commits.
>
> Here's the same commit, and the difference between 1.8 and 1.9:
>
> % git cat-file commit 803795d
> tree 7fc83aba022834da5c218114b09ad4640735bcc0
> parent c96fb0418e545a569b5975b4d878a30a948c29d5
> author olgeni  1437203525 +
> committer olgeni  1437203525 +
>
> Upgrade to version 0.4.1.
> % git cat-file commit 61ca43b
> tree 7fc83aba022834da5c218114b09ad4640735bcc0
> parent c96fb0418e545a569b5975b4d878a30a948c29d5
> author olgeni  1437203529 +
> committer olgeni  1437203529 +
>
> Upgrade to version 0.4.1.
>
>
> In case you don't see it, there's a 4s difference in the timestamps
> 

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

2015-11-08 2:51 GMT+01:00 Alfred Perlstein :
>>
> Uli,
>
> One of the biggest concerns I've heard from folks using FreeBSD's git mirror
> is that the hashes can change.
>
> I have a question about this.   Is it possible to keep track of what the
> "official" git mirror (on github) is doing and keep that as a log.  Then
> that log can be used to replay commits when there is a divergence problem.
>
> What I'm basically saying is that let's take this small example:
>
> importer is working fine @rev 1
> imports 1
> imports 10001
> imports 10002
> something happens to importer to give indeterminate shas.
> imports 10003 - sha is "unstable" sha3
> imports 10004 - sha is "unstable" sha4
> imports 10005 - sha is "unstable" sha5
> imports 10006 - sha is "unstable" sha6
> importer is fixed
>
>
> At this point normally we'd rewind the importer to 10002 and then force
> update the affected branches.
>
> My question is... can the imports of 10003, 10004, 10005 and 10006 be put
> into the importer such that any "mirror site" that re-does the import using
> the most up to date importer will get the same shas.
>
> That would allow to proceed with 10007, etc without force pushing.
>
> This should be possible based on querying "git" for the meta data associated
> with sha3..sha6 and then forcing those commits to have the same meta data.
>
> This would eliminate the concern about shas in the mirror changing that I've
> heard.

The goal of the conversion is that everyone can re-do the conversion
in their basement and come up with the same history and checksums.
This was not the case when I first started, as there was some
non-deterministic hash structure being used in svn2git. This was fixed
in the code and then all converter runs produced the very same
results.

The scenario that we have right now, is that one of the merge commits
done about two weeks ago is being handled different by svn2git w/ svn
v1.8 vs. svn v1.9 and I haven't investigated yet how the API's
behavior changed to cause this. I'm afraid I also swapped out all my
knowledge about svn2git internals and will have to redo this all from
scratch :/

Your suggestion could only work, if we hard-code this svn revision
special handling into svn2git, either in the code or by providing more
mappings and rules to the process. svn2git should run hermetic and not
poke at github's commits to see how things were handled in the past.
It has to be self-sufficient and must not depend on github.

This would also only work, if the "breakage" window was very small,
but it is already about two weeks long and will surely increase till I
find the proper fix.

So, to take a stand here: this sort of kludge is unlikely to ever
happen. Git commit hashes *might* change in the future. I really don't
see how this is a big deal anyway.  It happened once and I'm trying to
have it never happen again. But why are people afraid of this
happening? Every "official" git commit is tagged with a SVN revision
and the contents of those revisions are obviously correct (just not
the ancestry and the commit objects, possibly). So it would be easy to
write a script that replays VendorA's git history and swaps out the
new official commits for the old official commits. There would be no
merge conflicts.

I can see how this would be annoying if you have 100 developers and
dozens of branches that are far from mainline FreeBSD. But I'm sure
these companies that depend on git will come forward and donate some
of their developer manpower to help me with keeping the converter
stable/deterministic. Right? Right? :) :)

Cheers,
Uli
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Shane Ambler]

On 08/11/2015 21:36, Ulrich Spörlein wrote:


Here's the same commit, and the difference between 1.8 and 1.9:

% git cat-file commit 803795d
tree 7fc83aba022834da5c218114b09ad4640735bcc0
parent c96fb0418e545a569b5975b4d878a30a948c29d5
author olgeni  1437203525 +
committer olgeni  1437203525 +

Upgrade to version 0.4.1.
% git cat-file commit 61ca43b
tree 7fc83aba022834da5c218114b09ad4640735bcc0
parent c96fb0418e545a569b5975b4d878a30a948c29d5
author olgeni  1437203529 +
committer olgeni  1437203529 +

Upgrade to version 0.4.1.


In case you don't see it, there's a 4s difference in the timestamps
for authoring and committing. Here's the original:

% svn log -vc392405 svn://svn.freebsd.org/ports

r392405 | olgeni | 2015-07-18 09:12:05 +0200 (Sat, 18 Jul 2015) | 2 lines
Changed paths:
M /head/www/elixir-maru/Makefile
M /head/www/elixir-maru/distinfo

Upgrade to version 0.4.1.



So yeah, svn 1.9 returned a timestamp that was off by 4s. WTF?



4 seconds??
There have been 4 leap seconds added this century.
Did 1.9 add timestamp corrections relating to leap seconds?

Did the developer not use leapsecs when the svn server does?


--
FreeBSD - the place to B...Software Developing

Shane Ambler

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[NGie Cooper]

> On Nov 8, 2015, at 20:51, Shane Ambler  wrote:

...

> 4 seconds??
> There have been 4 leap seconds added this century.
> Did 1.9 add timestamp corrections relating to leap seconds?
> 
> Did the developer not use leapsecs when the svn server does?

Alternatively, was this impacted by a change in recent change to timekeeping 
(ntpd, etc)?
Thanks,
-NGie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Alfred Perlstein]

On 11/5/15 6:46 AM, Ulrich Spörlein wrote:

2015-11-04 18:57 GMT+01:00 Ulrich Spörlein :

The recent SVN update on the cluster broke svn2git in certain circumstances.

To fix this and make sure no content was dropped, the converter has
been stopped and we're working on bringing a fixed version online, as
well as vetting the correctness of the published git repositories.

ETA is currently unknown, expect an update to this thread within 24h.
Sorry for the inconvenience!

Uli, on behalf of git-admin

An independent run of the converter produces a different git
repository starting at the commit following this one:
https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52

This is from 9d ago and will likely require a force push to github
that will necessitate people to rebase or merge there work (a
fast-forward merge will fail).

This is the preliminary inspection and a third verification run is
currently underway. Expect another update within 24h.


Uli,

One of the biggest concerns I've heard from folks using FreeBSD's git 
mirror is that the hashes can change.


I have a question about this.   Is it possible to keep track of what the 
"official" git mirror (on github) is doing and keep that as a log.  Then 
that log can be used to replay commits when there is a divergence problem.


What I'm basically saying is that let's take this small example:

importer is working fine @rev 1
imports 1
imports 10001
imports 10002
something happens to importer to give indeterminate shas.
imports 10003 - sha is "unstable" sha3
imports 10004 - sha is "unstable" sha4
imports 10005 - sha is "unstable" sha5
imports 10006 - sha is "unstable" sha6
importer is fixed


At this point normally we'd rewind the importer to 10002 and then force 
update the affected branches.


My question is... can the imports of 10003, 10004, 10005 and 10006 be 
put into the importer such that any "mirror site" that re-does the 
import using the most up to date importer will get the same shas.


That would allow to proceed with 10007, etc without force pushing.

This should be possible based on querying "git" for the meta data 
associated with sha3..sha6 and then forcing those commits to have the 
same meta data.


This would eliminate the concern about shas in the mirror changing that 
I've heard.


-Alfred








___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

2015-11-05 15:46 GMT+01:00 Ulrich Spörlein :
> 2015-11-04 18:57 GMT+01:00 Ulrich Spörlein :
>> The recent SVN update on the cluster broke svn2git in certain circumstances.
>>
>> To fix this and make sure no content was dropped, the converter has
>> been stopped and we're working on bringing a fixed version online, as
>> well as vetting the correctness of the published git repositories.
>>
>> ETA is currently unknown, expect an update to this thread within 24h.
>> Sorry for the inconvenience!
>>
>> Uli, on behalf of git-admin
>
> An independent run of the converter produces a different git
> repository starting at the commit following this one:
> https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52
>
> This is from 9d ago and will likely require a force push to github
> that will necessitate people to rebase or merge there work (a
> fast-forward merge will fail).
>
> This is the preliminary inspection and a third verification run is
> currently underway. Expect another update within 24h.
>
> Uli

We've rolled back to SVN 1.8 and have resumed normal operations. We're
still working on getting svn2git to work nice with SVN 1.9 and produce
the same repository checksums.

Sorry for the inconvenience!
Uli
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

2015-11-04 18:57 GMT+01:00 Ulrich Spörlein :
> The recent SVN update on the cluster broke svn2git in certain circumstances.
>
> To fix this and make sure no content was dropped, the converter has
> been stopped and we're working on bringing a fixed version online, as
> well as vetting the correctness of the published git repositories.
>
> ETA is currently unknown, expect an update to this thread within 24h.
> Sorry for the inconvenience!
>
> Uli, on behalf of git-admin

An independent run of the converter produces a different git
repository starting at the commit following this one:
https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52

This is from 9d ago and will likely require a force push to github
that will necessitate people to rebase or merge there work (a
fast-forward merge will fail).

This is the preliminary inspection and a third verification run is
currently underway. Expect another update within 24h.

Uli
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

FYI: SVN to GIT converter currently broken, github is falling behind

[Ulrich Spörlein]

The recent SVN update on the cluster broke svn2git in certain circumstances.

To fix this and make sure no content was dropped, the converter has
been stopped and we're working on bringing a fixed version online, as
well as vetting the correctness of the published git repositories.

ETA is currently unknown, expect an update to this thread within 24h.
Sorry for the inconvenience!

Uli, on behalf of git-admin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"