Re: Migrating to git

[Coty Sutherland]

On Wed, Jan 24, 2018 at 4:57 PM, Christopher Schultz
 wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Coty,
>
> On 1/24/18 1:34 PM, Coty Sutherland wrote:
>> I think this is the right thread to add this :)
>>
>> Given that we will be able to accept PRs on Github after this
>> migration, do we want to add a Travis (or other CI) configuration
>> to our repository so that it will do a test build (and maybe test
>> one connector) when someone proposes a pull? Doing that will ensure
>> we get all of the new code accepted by PRs tested without manual
>> intervention or waiting for the ASF build infrastructure to do it.
>> You can make passing the tests a condition to accept the PR too I
>> believe.
>
> Could this "feature" be used to DOS some host(s)? If every PR causes a
> checkout/compile/test cycle, it could get "expensive".

Possibly, but my thoughts were to use travis (https://travis-ci.org/)
which has it's own hosts and such that we won't need to worry about.
Travis also allows you to limit concurrent builds and even limit it to
only building PRs. Were you worried about the potential for DoSing ASF
infra? I just pushed a test .travls.yml in my tomcat fork
(https://github.com/csutherl/tomcat/blob/trunk/.travis.yml) so I could
play with it a bit.

> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlppAVQdHGNocmlzQGNo
> cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFjBPg/+LywMRfUig0DsOkAt
> StgtY8dvvBJJyf9H9pQM8cqnjWx/nfKhvGhaaHgCd8ftfynM3k8I9X4iImKlRRhN
> xKCRbXyIr7QsCkbUDZqzVoTQeYE3A0OHSgo5kKlauI0+nrFPaoJW0fSAx9Gd4npV
> cV4PPsKokYiBbFl8/7q9taZ/FP0wB5QWf3hN1QEUFVyhlKzKhouB+dxmnzZzgBFS
> NRBFViOygJvYV299Jcg8UlyFC4HCfLy+L7uFsS5pf7MO29TG71xEMVrdNiMihz+j
> TxWG2n6AAK1fu0HS4GTIcan4zQUDtDoQB9VJD1TwOYQATgJQlNQbvaZvMk/+n6Gu
> WieMrGyLbJfix7asgv4qkQ80TyCTdQ1M3Ab4afrP0YGVzqIXRKFoyHQgiTZXgs6k
> de5Y6uFOgDna13up596BFKL94QkRyirAPXasVbB4QSuynmGiGCpAf7S+KHXb/T5W
> Ro2Zs1bHj1vTFB46NqGNJCjXC21ZmvPoujil7evohNc0vEVFNFLqsqcta2A30D6n
> +Yp/tbmDzY0xAl1Rp7BEUKVndzIWofvbQ08A41qWLMWTG1n16FjUQJg3xO3YHcDZ
> Oy0gWDnYvQlQnFKZs4bRCLle36dkAcZ2GzCwBELbxKKwiXk+mLS+Fz05YGG2BMYt
> 7GOhLV/GMhrz2Rs0MEzWgxlKhes=
> =ZVjJ
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Coty,

On 1/24/18 1:34 PM, Coty Sutherland wrote:
> I think this is the right thread to add this :)
> 
> Given that we will be able to accept PRs on Github after this 
> migration, do we want to add a Travis (or other CI) configuration
> to our repository so that it will do a test build (and maybe test
> one connector) when someone proposes a pull? Doing that will ensure
> we get all of the new code accepted by PRs tested without manual
> intervention or waiting for the ASF build infrastructure to do it.
> You can make passing the tests a condition to accept the PR too I
> believe.

Could this "feature" be used to DOS some host(s)? If every PR causes a
checkout/compile/test cycle, it could get "expensive".

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlppAVQdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFjBPg/+LywMRfUig0DsOkAt
StgtY8dvvBJJyf9H9pQM8cqnjWx/nfKhvGhaaHgCd8ftfynM3k8I9X4iImKlRRhN
xKCRbXyIr7QsCkbUDZqzVoTQeYE3A0OHSgo5kKlauI0+nrFPaoJW0fSAx9Gd4npV
cV4PPsKokYiBbFl8/7q9taZ/FP0wB5QWf3hN1QEUFVyhlKzKhouB+dxmnzZzgBFS
NRBFViOygJvYV299Jcg8UlyFC4HCfLy+L7uFsS5pf7MO29TG71xEMVrdNiMihz+j
TxWG2n6AAK1fu0HS4GTIcan4zQUDtDoQB9VJD1TwOYQATgJQlNQbvaZvMk/+n6Gu
WieMrGyLbJfix7asgv4qkQ80TyCTdQ1M3Ab4afrP0YGVzqIXRKFoyHQgiTZXgs6k
de5Y6uFOgDna13up596BFKL94QkRyirAPXasVbB4QSuynmGiGCpAf7S+KHXb/T5W
Ro2Zs1bHj1vTFB46NqGNJCjXC21ZmvPoujil7evohNc0vEVFNFLqsqcta2A30D6n
+Yp/tbmDzY0xAl1Rp7BEUKVndzIWofvbQ08A41qWLMWTG1n16FjUQJg3xO3YHcDZ
Oy0gWDnYvQlQnFKZs4bRCLle36dkAcZ2GzCwBELbxKKwiXk+mLS+Fz05YGG2BMYt
7GOhLV/GMhrz2Rs0MEzWgxlKhes=
=ZVjJ
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mark Thomas]

On 24/01/18 18:34, Coty Sutherland wrote:
> I think this is the right thread to add this :)
> 
> Given that we will be able to accept PRs on Github after this
> migration, do we want to add a Travis (or other CI) configuration to
> our repository so that it will do a test build (and maybe test one
> connector) when someone proposes a pull? Doing that will ensure we get
> all of the new code accepted by PRs tested without manual intervention
> or waiting for the ASF build infrastructure to do it. You can make
> passing the tests a condition to accept the PR too I believe.

+1 to more CI. I lean towards reporting results rather than hard
requirements but I suspect we'll tweak that sort of detail as we go along.

Mark



> 
> On Wed, Dec 13, 2017 at 8:52 AM, Rémy Maucherat  wrote:
>> On Wed, Dec 13, 2017 at 10:23 AM, Mark Thomas  wrote:
>>
>>> The plan is to switch to "github as master' which is actually a dual
>>> master system where commits can be made directly to either github or the
>>> ASF. I don't think it matter which we use. My (not recent) experience is
>>> that github is faster.
>>>
>>> Nice setup.
>>
>> Rémy
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Coty Sutherland]

I think this is the right thread to add this :)

Given that we will be able to accept PRs on Github after this
migration, do we want to add a Travis (or other CI) configuration to
our repository so that it will do a test build (and maybe test one
connector) when someone proposes a pull? Doing that will ensure we get
all of the new code accepted by PRs tested without manual intervention
or waiting for the ASF build infrastructure to do it. You can make
passing the tests a condition to accept the PR too I believe.

On Wed, Dec 13, 2017 at 8:52 AM, Rémy Maucherat  wrote:
> On Wed, Dec 13, 2017 at 10:23 AM, Mark Thomas  wrote:
>
>> The plan is to switch to "github as master' which is actually a dual
>> master system where commits can be made directly to either github or the
>> ASF. I don't think it matter which we use. My (not recent) experience is
>> that github is faster.
>>
>> Nice setup.
>
> Rémy

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Rémy Maucherat]

On Wed, Dec 13, 2017 at 10:23 AM, Mark Thomas  wrote:

> The plan is to switch to "github as master' which is actually a dual
> master system where commits can be made directly to either github or the
> ASF. I don't think it matter which we use. My (not recent) experience is
> that github is faster.
>
> Nice setup.

Rémy

Re: Migrating to git

[Mark Thomas]

On 12/12/17 13:41, Konstantin Kolinko wrote:
>>
>> The first draft of this is up.
>>
>> https://cwiki.apache.org/confluence/display/TOMCAT/Git+migration
>>
>> Please add any issues either directly to that page or this thread.
> 
> 1. I suspect that existing pull requests / issues at GitHub will be lost.
> 
> Maybe they will be kept for tomcat.git repository, but lost for all others.
> 
> At least we need to be prepared for the loss.

Added to wiki with possible solution.

> 2. Maybe we need to further remove Tomcat from ReviewBoard.
> 
> I think that it is somehow linked to the svn repository.
> 
> A link from Tomcat site to it was removed in
> http://svn.apache.org/viewvc?view=revision=1799409
> 
> https://reviews.apache.org/groups/tomcat/

Added to wiki with possible solution.

> 3. Document how to deal with single git repository locally.
> 
> I guess that this is managed with "git worktree" command, but have
> never used it and need some guidance.

Added to the wiki. 'git worktree' is certainly one way of handling this.
I was leaning towards multiple clones myself but I hadn't really thought
about it much. Some research and discussion around options would be useful.

> 4. Update "building" documentation in Tomcat X.Y.
> Publish it? (Or wait for the next release, as usual.)
> 
> Update repository documentation on the main site,
> http://tomcat.apache.org/svn.html

Added to wiki.

> A comment on migration script.
> 
>> # Add Tomcat 8.5x
>> git remote add tomcat85 https://github.com/apache/tomcat85.git
> 
> I think the final recipe should use ASF repository url here, instead
> of github one.

The plan is to switch to "github as master' which is actually a dual
master system where commits can be made directly to either github or the
ASF. I don't think it matter which we use. My (not recent) experience is
that github is faster.

>> # Make svn read only
>> # Turn off the svn / git mirror
>> # Switch to 'github as master'
> 
> One need to check that recent svn commits have been successfully
> synced to git first, before stopping the mirroring. (With Tomcat 6.0.x
> I sometimes observed a delay of ~30 minutes.)

ACK. I'll add that to the list. (I can force an update if necessary.)

Thanks for the thorough review.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Konstantin Kolinko]

>
> The first draft of this is up.
>
> https://cwiki.apache.org/confluence/display/TOMCAT/Git+migration
>
> Please add any issues either directly to that page or this thread.

1. I suspect that existing pull requests / issues at GitHub will be lost.

Maybe they will be kept for tomcat.git repository, but lost for all others.

At least we need to be prepared for the loss.


2. Maybe we need to further remove Tomcat from ReviewBoard.

I think that it is somehow linked to the svn repository.

A link from Tomcat site to it was removed in
http://svn.apache.org/viewvc?view=revision=1799409

https://reviews.apache.org/groups/tomcat/


3. Document how to deal with single git repository locally.

I guess that this is managed with "git worktree" command, but have
never used it and need some guidance.


4. Update "building" documentation in Tomcat X.Y.
Publish it? (Or wait for the next release, as usual.)

Update repository documentation on the main site,
http://tomcat.apache.org/svn.html




A comment on migration script.

> # Add Tomcat 8.5x
> git remote add tomcat85 https://github.com/apache/tomcat85.git

I think the final recipe should use ASF repository url here, instead
of github one.

> # Make svn read only
> # Turn off the svn / git mirror
> # Switch to 'github as master'

One need to check that recent svn commits have been successfully
synced to git first, before stopping the mirroring. (With Tomcat 6.0.x
I sometimes observed a delay of ~30 minutes.)


Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mark Thomas]

On 12/12/17 06:48, Rémy Maucherat wrote:
> On Mon, Dec 11, 2017 at 10:28 PM, Mark Thomas  wrote:
> 
>> On 11/12/17 16:24, Mark Thomas wrote:
>>> On 11/12/17 15:27, Rémy Maucherat wrote:
>>
>> 
>>
 Do we get a "demo" repo to see the result ?
>>>
>>> Because the plan above starts with the existing git mirror when we do
>>> decide to migrate it will be directly on the live repo. However, I'm
>>> happy to pull together a (read-only) demo git repo of what it would look
>>> like based on a point in time snapshot. I should be able to do that this
>>> week.
>>
>> If all goes to plan, the current apache/tomcat git repo would look like
>> this:
>> https://github.com/markt-asf/tomcat
>>
> 
> Ok, so that looks good. Thanks.

Great.

>>> I also need to get that list of potential issues and solutions written
>> up.
>>
>> I'll work on filling in the issues section tomorrow.

The first draft of this is up.

https://cwiki.apache.org/confluence/display/TOMCAT/Git+migration

Please add any issues either directly to that page or this thread. My
plan is update that page over the next few days/weeks, then start a
thread per issue, reach a consensus on each issue and then call a vote
to decide whether to go ahead or not. Given the time of year, my guess
is the vote will be early next year.

Suggestions to improve then plan also welcome.

Mark


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Rémy Maucherat]

On Mon, Dec 11, 2017 at 10:28 PM, Mark Thomas  wrote:

> On 11/12/17 16:24, Mark Thomas wrote:
> > On 11/12/17 15:27, Rémy Maucherat wrote:
>
> 
>
> >> Do we get a "demo" repo to see the result ?
> >
> > Because the plan above starts with the existing git mirror when we do
> > decide to migrate it will be directly on the live repo. However, I'm
> > happy to pull together a (read-only) demo git repo of what it would look
> > like based on a point in time snapshot. I should be able to do that this
> > week.
>
> If all goes to plan, the current apache/tomcat git repo would look like
> this:
> https://github.com/markt-asf/tomcat
>

Ok, so that looks good. Thanks.


>
> It won't be a fork so the UI behaviour will be slightly different but
> that should be enough to give you the idea.
>
> The migration steps are documented here:
> https://cwiki.apache.org/confluence/display/TOMCAT/Git+migration
>
> The good news is that I can run though this process in less than an hour
> so if:
> - we are happy with this; and
> - we are happy with the issues and proposed solutions; and
> - we agree to migrate to git
>
> it can happen pretty quickly and minimal impact / downtime for development.
>
> > I also need to get that list of potential issues and solutions written
> up.
>
> I'll work on filling in the issues section tomorrow.
>

Rémy

Re: Migrating to git

[Mark Thomas]

On 11/12/17 16:24, Mark Thomas wrote:
> On 11/12/17 15:27, Rémy Maucherat wrote:



>> Do we get a "demo" repo to see the result ?
> 
> Because the plan above starts with the existing git mirror when we do
> decide to migrate it will be directly on the live repo. However, I'm
> happy to pull together a (read-only) demo git repo of what it would look
> like based on a point in time snapshot. I should be able to do that this
> week.

If all goes to plan, the current apache/tomcat git repo would look like
this:
https://github.com/markt-asf/tomcat

It won't be a fork so the UI behaviour will be slightly different but
that should be enough to give you the idea.

The migration steps are documented here:
https://cwiki.apache.org/confluence/display/TOMCAT/Git+migration

The good news is that I can run though this process in less than an hour
so if:
- we are happy with this; and
- we are happy with the issues and proposed solutions; and
- we agree to migrate to git

it can happen pretty quickly and minimal impact / downtime for development.

> I also need to get that list of potential issues and solutions written up.

I'll work on filling in the issues section tomorrow.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mark Thomas]

On 11/12/17 15:27, Rémy Maucherat wrote:
> On Wed, Dec 6, 2017 at 2:54 PM, Mark Thomas  wrote:
> 
>> On 06/12/17 11:30, Mark Thomas wrote:
>>> On 06/12/17 10:34, Konstantin Kolinko wrote:
>>
>> 
>>
 If the goal is to create a single Git repository from our several ones,
 my suggestion is to create a repository and pull in the branches from
 the existing 7/8/8.5/9 git mirrors.
>>
>> 
>>
>>> Sounds good. And it can be tested without impacting on any of the
>>> current public repos. I'll try it out and report back.
>>
>> I've tested this with trunk and 8.5.x. and it works well. It is also
>> very quick. Much better than my original plan. We would be able to add
>> the older branches to the existing git mirror of trunk. That would make
>> the process something like:
>>
>> - make svn read-only for trunk, 8.5.x, 8.0.x and 7.0.x
>> - disable svn->git mirroring
>> - make git for trunk read/write (we could switch to github as the master
>>   at the same time)
>> - merge in branches for 8.5.x, 8.0.x and 7.0.x
>> - clean-up tags
>> - update CI systems, web pages etc for new locations
>> - move trunk, 8.5.x, 8.0.x and 7.0.x to the archive
>>
>> Before we do this, there are a number of points we should think about.
>> You listed a couple below. There are others on some older threads on
>> this topic.
>>
>> I suggest we write up (I'm happy to start it) a list of potential issues
>> and solutions on the wiki and make sure we are happy with those
>> solutions before we migrate.
>>
> Do we get a "demo" repo to see the result ?

Because the plan above starts with the existing git mirror when we do
decide to migrate it will be directly on the live repo. However, I'm
happy to pull together a (read-only) demo git repo of what it would look
like based on a point in time snapshot. I should be able to do that this
week.

I also need to get that list of potential issues and solutions written up.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Rémy Maucherat]

On Wed, Dec 6, 2017 at 2:54 PM, Mark Thomas  wrote:

> On 06/12/17 11:30, Mark Thomas wrote:
> > On 06/12/17 10:34, Konstantin Kolinko wrote:
>
> 
>
> >> If the goal is to create a single Git repository from our several ones,
> >> my suggestion is to create a repository and pull in the branches from
> >> the existing 7/8/8.5/9 git mirrors.
>
> 
>
> > Sounds good. And it can be tested without impacting on any of the
> > current public repos. I'll try it out and report back.
>
> I've tested this with trunk and 8.5.x. and it works well. It is also
> very quick. Much better than my original plan. We would be able to add
> the older branches to the existing git mirror of trunk. That would make
> the process something like:
>
> - make svn read-only for trunk, 8.5.x, 8.0.x and 7.0.x
> - disable svn->git mirroring
> - make git for trunk read/write (we could switch to github as the master
>   at the same time)
> - merge in branches for 8.5.x, 8.0.x and 7.0.x
> - clean-up tags
> - update CI systems, web pages etc for new locations
> - move trunk, 8.5.x, 8.0.x and 7.0.x to the archive
>
> Before we do this, there are a number of points we should think about.
> You listed a couple below. There are others on some older threads on
> this topic.
>
> I suggest we write up (I'm happy to start it) a list of potential issues
> and solutions on the wiki and make sure we are happy with those
> solutions before we migrate.
>
> Do we get a "demo" repo to see the result ?

Rémy

Re: Migrating to git

[Mark Thomas]

On 06/12/17 11:30, Mark Thomas wrote:
> On 06/12/17 10:34, Konstantin Kolinko wrote:



>> If the goal is to create a single Git repository from our several ones,
>> my suggestion is to create a repository and pull in the branches from
>> the existing 7/8/8.5/9 git mirrors.



> Sounds good. And it can be tested without impacting on any of the
> current public repos. I'll try it out and report back.

I've tested this with trunk and 8.5.x. and it works well. It is also
very quick. Much better than my original plan. We would be able to add
the older branches to the existing git mirror of trunk. That would make
the process something like:

- make svn read-only for trunk, 8.5.x, 8.0.x and 7.0.x
- disable svn->git mirroring
- make git for trunk read/write (we could switch to github as the master
  at the same time)
- merge in branches for 8.5.x, 8.0.x and 7.0.x
- clean-up tags
- update CI systems, web pages etc for new locations
- move trunk, 8.5.x, 8.0.x and 7.0.x to the archive

Before we do this, there are a number of points we should think about.
You listed a couple below. There are others on some older threads on
this topic.

I suggest we write up (I'm happy to start it) a list of potential issues
and solutions on the wiki and make sure we are happy with those
solutions before we migrate.

Mark

> 
>> Though:
>> 1. Current Git mirrors do not reflect edits to svn:log messages (done
>> to correct typos, empty messages, or to add CVE numbers)
>>
>> If we want to keep the corrected log messages, we will need to repeat
>> svn->git migration,
>> but this will invalidate the current tomcat[nn] git repositories.
> 
> We need to decide which way we want to go for that. Given that we won't
> be able to edit log messages going forwards, I'm leaning towards going
> with git repos in there current state but I don't feel particularly
> strongly either way.
> 
>> 2. Some git operations do not work with empty log messages.
>> A good time to fill in all such messages in svn repository is before
>> doing re-migration.
>>
>> (I know of "git rebase" having such problem. Generally git has a
>> command-line flag to tolerate such revisions, but when rebase is done
>> with a GUI there is no such checkbox)
> 
> My thoughts are broadly the same as for point 1 above.
> 
>> 3. I wonder what will be the size of such combined repository.
> 
> I'll let you know...
> 
>> BTW,
>> there exist such tool for repository mirroring, but I have not
>> investigated further
>> (I just saw it listed at
>> https://github.com/google/guava/wiki/FriendsOfGuava as the tool used
>> to mirror guava repository)
>> https://github.com/google/MOE
> 
> Are you suggesting we try and keep svn and git running in parallel? That
> would require support from infra I'm fairly sure we wouldn't get.
> 
>>> Plan B
>>> ==
>>>
>>> Pick a different component (native, jk) and migrate that first.
>>>
>>>
>>> If we do want to migrate there will be lots of details to work out such
>>> as how to migrate the "view differences" feature of the migration page
>>> but I'm sure we'll be able to work something out.
>>
>> tomcat-native uses svn:externals link to Tomcat source tree
>> and has a step in release script that checks that this svn:external is
>> up-to-date.
>>
>> All that needs a replacement.
> 
> Agreed.
> 
>> mod_jk is OK, does not have such dependency, if I remember correctly.
> 
> I don't recall one.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mark Thomas]

On 06/12/17 10:34, Konstantin Kolinko wrote:
> 2017-12-05 23:03 GMT+03:00 Mark Thomas :
>> Hi all,
>>
>> I've been doing some experiments to see how we might migrate from our
>> current svn structure to git. It appears that git svn is able to follow
>> directory moves so, with that in mind, I'd like to propose the following
>> outline plan:
>>
>> Plan A
>> ==
>>
>> 0. Run plan past infra
>> 1. Restructure svn
>> 2. Get infra to re-mirror new structure
>> 3. Validate mirror
>> 4. Pick a date to switch
>>
>> This assumes that we do want to switch to git. My sense from the most
>> recent discussion was that we did.
> 
> I do not want such migration, and am ready to vote -1 here.
> I postpone my vote to discuss the technical issues (in case such
> migration ever happens).
> 
> 
>> To expand on the point one, the restructuring would look like:
>>
>> /trunk  -> no change
>> /tags/TOMCAT_9* -> tags/tc9.0.x/
>> /tc8.5.x/trunk  -> branches/tc8.5.x
>> /tc8.5.x/tags/* -> tags/tc8.5.x/
>> /tc8.0.x/trunk  -> branches/tc8.0.x
>> /tc8.0.x/tags/* -> tags/tc8.0.x/
>> /tc7.0.x/trunk  -> branches/tc7.0.x
>> /tc7.0.x/tags/* -> tags/tc7.0.x/
>>
>> and then migrate /trunk, /tags and /branches to git, leaving the rest in
>> place. Most will stay there. Some components may move to git in the future.
> 
> Why?

Because having tried to perform an svn2git migration with the existing
structure takes far too long (when I tried this a few months ago I
stopped after around a week of processing) and ends up including a whole
bunch of stuff that didn't need to be there.

Moving things around first was an attempt to reduce the work required by
the migration process.

> When one does a git -> svn migrartion, the migration generally goes up
> the history of the directory,
> finds the root revision and starts replicating from there.  All those
> renames will be reflected in history along with the old paths. The new
> paths do not matter.
> Those names do not matter for Git.

In my simple local testing, the moving around prior to migration did
seem to have the desired effect. The actual Tomcat repository is much
more complex. The end result may not be exactly the same.

> They may matter for future move of svn tree to "archived" directory
> (the general rule is that the svn tree is moved after committers switch to 
> Git).
> 
> 
> If the goal is to create a single Git repository from our several ones,
> my suggestion is to create a repository and pull in the branches from
> the existing 7/8/8.5/9 git mirrors.

I have no strong preference for process. It is the end result I am
interested in.

> Suppose that the repository is configured with
> origin = URL of this new repository
> origin-tomcat70 = URL of tomcat70 repository
> (https://github.com/apache/tomcat70 or better the original one:
> https://git.apache.org/tomcat70.git)
> origin-tomcat80 = URL of tomcat80 repository
> origin-tomcat85 = URL of tomcat85 repository
> origin-tomcat90 = URL of tomcat90 repository
> 
> You will get
> refs/remote/origin-tomcat70/...
> refs/remote/origin-tomcat80/...
> refs/remote/origin-tomcat85/...
> refs/remote/origin-tomcat90/...
> 
> Now, create local names for those branches (in refs/heads),
> with "git branch" [--force] or with "git update-ref".
> 
> (A question is what those local names will be.
> TBD.
> My guess that the "trunk" branches will become
>  refs/heads/master
>  refs/heads/tomcat90
>  refs/heads/tomcat85
>  etc.)
> 
> 
> The benefit is that those refs will have the same sha1 as the refs in
> the the original https://git.apache.org/tomcat[nn].git repositories,
> allowing to deprecate tomcat[nn] repositories without loosing their history.
> 
> It also allows us to get the single repository and continue with
> svn->git mirroring,
> 
> The structure of repositories
> svn --[svn-git]--> tomcat[nn].git repositories --[the mapping
> described here]--> single tomcat.git repository.

Sounds good. And it can be tested without impacting on any of the
current public repos. I'll try it out and report back.

> Though:
> 1. Current Git mirrors do not reflect edits to svn:log messages (done
> to correct typos, empty messages, or to add CVE numbers)
> 
> If we want to keep the corrected log messages, we will need to repeat
> svn->git migration,
> but this will invalidate the current tomcat[nn] git repositories.

We need to decide which way we want to go for that. Given that we won't
be able to edit log messages going forwards, I'm leaning towards going
with git repos in there current state but I don't feel particularly
strongly either way.

> 2. Some git operations do not work with empty log messages.
> A good time to fill in all such messages in svn repository is before
> doing re-migration.
> 
> (I know of "git rebase" having such problem. Generally git has a
> command-line flag to tolerate such revisions, but when rebase is done
> with a GUI there is no such checkbox)

My thoughts are broadly the same as for point 1 above.

> 

Re: Migrating to git

[Konstantin Kolinko]

2017-12-05 23:03 GMT+03:00 Mark Thomas :
> Hi all,
>
> I've been doing some experiments to see how we might migrate from our
> current svn structure to git. It appears that git svn is able to follow
> directory moves so, with that in mind, I'd like to propose the following
> outline plan:
>
> Plan A
> ==
>
> 0. Run plan past infra
> 1. Restructure svn
> 2. Get infra to re-mirror new structure
> 3. Validate mirror
> 4. Pick a date to switch
>
> This assumes that we do want to switch to git. My sense from the most
> recent discussion was that we did.

I do not want such migration, and am ready to vote -1 here.
I postpone my vote to discuss the technical issues (in case such
migration ever happens).


> To expand on the point one, the restructuring would look like:
>
> /trunk  -> no change
> /tags/TOMCAT_9* -> tags/tc9.0.x/
> /tc8.5.x/trunk  -> branches/tc8.5.x
> /tc8.5.x/tags/* -> tags/tc8.5.x/
> /tc8.0.x/trunk  -> branches/tc8.0.x
> /tc8.0.x/tags/* -> tags/tc8.0.x/
> /tc7.0.x/trunk  -> branches/tc7.0.x
> /tc7.0.x/tags/* -> tags/tc7.0.x/
>
> and then migrate /trunk, /tags and /branches to git, leaving the rest in
> place. Most will stay there. Some components may move to git in the future.

Why?

When one does a git -> svn migrartion, the migration generally goes up
the history of the directory,
finds the root revision and starts replicating from there.  All those
renames will be reflected in history along with the old paths. The new
paths do not matter.
Those names do not matter for Git.

They may matter for future move of svn tree to "archived" directory
(the general rule is that the svn tree is moved after committers switch to Git).


If the goal is to create a single Git repository from our several ones,
my suggestion is to create a repository and pull in the branches from
the existing 7/8/8.5/9 git mirrors.

Suppose that the repository is configured with
origin = URL of this new repository
origin-tomcat70 = URL of tomcat70 repository
(https://github.com/apache/tomcat70 or better the original one:
https://git.apache.org/tomcat70.git)
origin-tomcat80 = URL of tomcat80 repository
origin-tomcat85 = URL of tomcat85 repository
origin-tomcat90 = URL of tomcat90 repository

You will get
refs/remote/origin-tomcat70/...
refs/remote/origin-tomcat80/...
refs/remote/origin-tomcat85/...
refs/remote/origin-tomcat90/...

Now, create local names for those branches (in refs/heads),
with "git branch" [--force] or with "git update-ref".

(A question is what those local names will be.
TBD.
My guess that the "trunk" branches will become
 refs/heads/master
 refs/heads/tomcat90
 refs/heads/tomcat85
 etc.)


The benefit is that those refs will have the same sha1 as the refs in
the the original https://git.apache.org/tomcat[nn].git repositories,
allowing to deprecate tomcat[nn] repositories without loosing their history.

It also allows us to get the single repository and continue with
svn->git mirroring,

The structure of repositories
svn --[svn-git]--> tomcat[nn].git repositories --[the mapping
described here]--> single tomcat.git repository.


Though:
1. Current Git mirrors do not reflect edits to svn:log messages (done
to correct typos, empty messages, or to add CVE numbers)

If we want to keep the corrected log messages, we will need to repeat
svn->git migration,
but this will invalidate the current tomcat[nn] git repositories.


2. Some git operations do not work with empty log messages.
A good time to fill in all such messages in svn repository is before
doing re-migration.

(I know of "git rebase" having such problem. Generally git has a
command-line flag to tolerate such revisions, but when rebase is done
with a GUI there is no such checkbox)


3. I wonder what will be the size of such combined repository.


BTW,
there exist such tool for repository mirroring, but I have not
investigated further
(I just saw it listed at
https://github.com/google/guava/wiki/FriendsOfGuava as the tool used
to mirror guava repository)
https://github.com/google/MOE


> Plan B
> ==
>
> Pick a different component (native, jk) and migrate that first.
>
>
> If we do want to migrate there will be lots of details to work out such
> as how to migrate the "view differences" feature of the migration page
> but I'm sure we'll be able to work something out.

tomcat-native uses svn:externals link to Tomcat source tree
and has a step in release script that checks that this svn:external is
up-to-date.

All that needs a replacement.

mod_jk is OK, does not have such dependency, if I remember correctly.

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mark Thomas]

On 06/12/17 02:05, Huxing Zhang wrote:
> Hi, 
> 
> I am +1 on migrating to git. I think this will help people easier to get 
> involved.
> 
> One quick question to Plan A:
> After migration to git, will the previous mirrors, e.g. 
> https://github.com/apache/tomcat70, be removed?

Yes.

> We maintain an internal repository with selective tomcat versions, which 
> depend on these mirrors to merge the latest releases.

I make use of some mirrors at $work too that would need some changes
after the migration.

Mark


> 
> --
> Mark Thomas <ma...@apache.org>
> 2017 Dec 6 (Wed) 04:03
> Tomcat Developers List <dev@tomcat.apache.org>
> Migrating to git
> 
> 
> Hi all,
> 
> I've been doing some experiments to see how we might migrate from our
> current svn structure to git. It appears that git svn is able to follow
> directory moves so, with that in mind, I'd like to propose the following
> outline plan:
> 
> Plan A
> ==
> 
> 0. Run plan past infra
> 1. Restructure svn
> 2. Get infra to re-mirror new structure
> 3. Validate mirror
> 4. Pick a date to switch
> 
> This assumes that we do want to switch to git. My sense from the most
> recent discussion was that we did.
> 
> To expand on the point one, the restructuring would look like:
> 
> /trunk  -> no change
> /tags/TOMCAT_9* -> tags/tc9.0.x/
> /tc8.5.x/trunk  -> branches/tc8.5.x
> /tc8.5.x/tags/* -> tags/tc8.5.x/
> /tc8.0.x/trunk  -> branches/tc8.0.x
> /tc8.0.x/tags/* -> tags/tc8.0.x/
> /tc7.0.x/trunk  -> branches/tc7.0.x
> /tc7.0.x/tags/* -> tags/tc7.0.x/
> 
> and then migrate /trunk, /tags and /branches to git, leaving the rest in
> place. Most will stay there. Some components may move to git in the future.
> 
> Plan B
> ==
> 
> Pick a different component (native, jk) and migrate that first.
> 
> 
> If we do want to migrate there will be lots of details to work out such
> as how to migrate the "view differences" feature of the migration page
> but I'm sure we'll be able to work something out.
> 
> Thoughts, comments etc.?
> 
> Mark
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mohamed Abd ALLAH]

+1 for git.
+1 for Branches instead of tags.

On Tue, Dec 5, 2017 at 9:05 PM, Huxing Zhang <huxing@alibaba-inc.com>
wrote:

> Hi,
>
> I am +1 on migrating to git. I think this will help people easier to get
> involved.
>
> One quick question to Plan A:
> After migration to git, will the previous mirrors, e.g.
> https://github.com/apache/tomcat70, be removed?
>
> We maintain an internal repository with selective tomcat versions, which
> depend on these mirrors to merge the latest releases.
>
> --
> Mark Thomas <ma...@apache.org>
> 2017 Dec 6 (Wed) 04:03
> Tomcat Developers List <dev@tomcat.apache.org>
> Migrating to git
>
>
> Hi all,
>
> I've been doing some experiments to see how we might migrate from our
> current svn structure to git. It appears that git svn is able to follow
> directory moves so, with that in mind, I'd like to propose the following
> outline plan:
>
> Plan A
> ==
>
> 0. Run plan past infra
> 1. Restructure svn
> 2. Get infra to re-mirror new structure
> 3. Validate mirror
> 4. Pick a date to switch
>
> This assumes that we do want to switch to git. My sense from the most
> recent discussion was that we did.
>
> To expand on the point one, the restructuring would look like:
>
> /trunk  -> no change
> /tags/TOMCAT_9* -> tags/tc9.0.x/
> /tc8.5.x/trunk  -> branches/tc8.5.x
> /tc8.5.x/tags/* -> tags/tc8.5.x/
> /tc8.0.x/trunk  -> branches/tc8.0.x
> /tc8.0.x/tags/* -> tags/tc8.0.x/
> /tc7.0.x/trunk  -> branches/tc7.0.x
> /tc7.0.x/tags/* -> tags/tc7.0.x/
>
> and then migrate /trunk, /tags and /branches to git, leaving the rest in
> place. Most will stay there. Some components may move to git in the future.
>
> Plan B
> ==
>
> Pick a different component (native, jk) and migrate that first.
>
>
> If we do want to migrate there will be lots of details to work out such
> as how to migrate the "view differences" feature of the migration page
> but I'm sure we'll be able to work something out.
>
> Thoughts, comments etc.?
>
> Mark
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org




-- 
Regards

Re: Migrating to git

[Huxing Zhang]

Hi, 

I am +1 on migrating to git. I think this will help people easier to get 
involved.

One quick question to Plan A:
After migration to git, will the previous mirrors, e.g. 
https://github.com/apache/tomcat70, be removed?
 
We maintain an internal repository with selective tomcat versions, which depend 
on these mirrors to merge the latest releases.

--
Mark Thomas <ma...@apache.org>
2017 Dec 6 (Wed) 04:03
Tomcat Developers List <dev@tomcat.apache.org>
Migrating to git


Hi all,

I've been doing some experiments to see how we might migrate from our
current svn structure to git. It appears that git svn is able to follow
directory moves so, with that in mind, I'd like to propose the following
outline plan:

Plan A
==

0. Run plan past infra
1. Restructure svn
2. Get infra to re-mirror new structure
3. Validate mirror
4. Pick a date to switch

This assumes that we do want to switch to git. My sense from the most
recent discussion was that we did.

To expand on the point one, the restructuring would look like:

/trunk  -> no change
/tags/TOMCAT_9* -> tags/tc9.0.x/
/tc8.5.x/trunk  -> branches/tc8.5.x
/tc8.5.x/tags/* -> tags/tc8.5.x/
/tc8.0.x/trunk  -> branches/tc8.0.x
/tc8.0.x/tags/* -> tags/tc8.0.x/
/tc7.0.x/trunk  -> branches/tc7.0.x
/tc7.0.x/tags/* -> tags/tc7.0.x/

and then migrate /trunk, /tags and /branches to git, leaving the rest in
place. Most will stay there. Some components may move to git in the future.

Plan B
==

Pick a different component (native, jk) and migrate that first.


If we do want to migrate there will be lots of details to work out such
as how to migrate the "view differences" feature of the migration page
but I'm sure we'll be able to work something out.

Thoughts, comments etc.?

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Mark Thomas]

On 05/12/17 20:43, Emmanuel Bourg wrote:
> Le 05/12/2017 à 21:03, Mark Thomas a écrit :
> 
>> and then migrate /trunk, /tags and /branches to git, leaving the rest in
>> place. Most will stay there. Some components may move to git in the future.
> 
> The goal is to put all Tomcat versions in the same Git repository right?

Right.

> Would that make the svn tags available as git tags or git branches?

tags.

> Is svn2git able to aggregate several tags directories (tags/tc7.0.x/*,
> tags/tc8.0.x/, etc) ?

Based on my simplified local testing, yes.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Coty Sutherland]

On Tue, Dec 5, 2017 at 3:43 PM, Emmanuel Bourg  wrote:
> Le 05/12/2017 à 21:03, Mark Thomas a écrit :
>
>> and then migrate /trunk, /tags and /branches to git, leaving the rest in
>> place. Most will stay there. Some components may move to git in the future.
>
> The goal is to put all Tomcat versions in the same Git repository right?
> Would that make the svn tags available as git tags or git branches? Is
> svn2git able to aggregate several tags directories (tags/tc7.0.x/*,
> tags/tc8.0.x/, etc) ? If not the tags could probably be merged in the
> same directory before the migration.

I'm +1 for this, but had the same questions. I think taking this
opportunity to consolidate the repos into one (especially on github)
if possible would help with new contributors given questions I've seen
on freenode.

> Emmanuel Bourg
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Migrating to git

[Emmanuel Bourg]

Le 05/12/2017 à 21:03, Mark Thomas a écrit :

> and then migrate /trunk, /tags and /branches to git, leaving the rest in
> place. Most will stay there. Some components may move to git in the future.

The goal is to put all Tomcat versions in the same Git repository right?
Would that make the svn tags available as git tags or git branches? Is
svn2git able to aggregate several tags directories (tags/tc7.0.x/*,
tags/tc8.0.x/, etc) ? If not the tags could probably be merged in the
same directory before the migration.

Emmanuel Bourg

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Migrating to git

[Mark Thomas]

Hi all,

I've been doing some experiments to see how we might migrate from our
current svn structure to git. It appears that git svn is able to follow
directory moves so, with that in mind, I'd like to propose the following
outline plan:

Plan A
==

0. Run plan past infra
1. Restructure svn
2. Get infra to re-mirror new structure
3. Validate mirror
4. Pick a date to switch

This assumes that we do want to switch to git. My sense from the most
recent discussion was that we did.

To expand on the point one, the restructuring would look like:

/trunk  -> no change
/tags/TOMCAT_9* -> tags/tc9.0.x/
/tc8.5.x/trunk  -> branches/tc8.5.x
/tc8.5.x/tags/* -> tags/tc8.5.x/
/tc8.0.x/trunk  -> branches/tc8.0.x
/tc8.0.x/tags/* -> tags/tc8.0.x/
/tc7.0.x/trunk  -> branches/tc7.0.x
/tc7.0.x/tags/* -> tags/tc7.0.x/

and then migrate /trunk, /tags and /branches to git, leaving the rest in
place. Most will stay there. Some components may move to git in the future.

Plan B
==

Pick a different component (native, jk) and migrate that first.


If we do want to migrate there will be lots of details to work out such
as how to migrate the "view differences" feature of the migration page
but I'm sure we'll be able to work something out.

Thoughts, comments etc.?

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org