0. Introduction
===
The fist incarnation of onegit.sh, despite all the tuning effort, was
still taking 36 hours to run. That was within the 'we should be able
to do the conversion over a week-end' criteria, but it was still
painfully long.
So, plan B, I tried to use git fast-export/import instead of git
filter-branch. That plan proved successful and now the conversion
itself takes about 30 _minutes_ (add another 15 for a final git gc of
the resulting core.git and a couple of hours to upload it all)
The core of plan B is lo_git_rewrite, a small C program that massage
the data stream between git fast-export and git-fast-import. It is
available in the dev-tools git repo.
1. Usage Notes
If you want to try it for yourself, here are few things to know:
1.1 Pre-requisites
1.1.1 Platform
This has only been tested on Linux. Other platform may work, but use
at your own risk.
1.1.2 Git
This has been tested with git 1.7.3.4. Any recent git should work, but
lo_git_rewrite makes a lot of implicit assumptions about the data
stream provided by git fast-export, so any version of git that alter
that flow, even in a way compatible with git fast-import
specifications may cause trouble.
1.1.3 source git repos
You need to have a 'source' bootstrap tree, including
clone/translation. make sure that master is checked-out and that you
are up-to-date and clean.
1.1.4 dev-tools
You need to clone the contrib/dev-tools repo and run make in
dev-tools/lo_git_rewrite
1.1.5 temp space
Most of the work is done in a temporary directory. you need 5+GB of
space there (I don't know for sure the exact amount but 5GB should be
enough)
1.1.6 target repos
The ongit.sh script will create a target repository, with clone/*
populated with the remaining separate git repos (help, translation,s,
dictionaries and binfilter).
The core repo is initially not properly compacted, and since you
porbably want to build it, you need enough space... as a rule of
thumb, count the same amount of space you would normally reserve for a
regular bootstrap buid.
1.2 Running
Assuming that your source bootstrap repo is at /lo/libo, that the
target will be /lo/core, that the dev-tools repo is at /lo/dev-tools
and that your temporary workspace is /fast, then run:
cd /lo
time ./dev-tools/onegit/onegit.sh -f -g /lo/libo -n core -t
/fast/gittemp 21 | tee onegit.log
while it is running you can look at /lo/onegit.msgs. it contains a
high-level log of what is going on.
Note that in onegit.msgs lines should start with === any line that
start with *** indicate that something went wrong.
Note: the onegit.sh has been tuned to work optimally on a Intel Xeon
X3360 @ 2.83GHz (quad-core), with 8GB of memory and pretty good
disks.
For optimal result on a different machine you may need to tweak the
number of batch that ran in // and their composition. (see section
2.2.2 for gotcha).
1.3 Known issue
The onegit.sh script, as a final step try to apply a set of patches,
to fix issue related to the migration. unfortunately, since master is
a moving target these patches may fail to apply.
At this stage the conversion is done and core is usable. you can try
to fix the patches that failed to apply (and apply the rest of the
patches)
the patches are in dev-tools/onegit/patches/*
1.4 Testing
once all the patches are applied. you can start using the 'core' repo
as if it was bootstrap.
2. Reviewer Notes
==
Reviews are of course welcomed
In order to help with the review, here are a few pointers.
2.1 Review of lo_git_rewrite
lo_git_rewrite is a fairly small C program that sit between git
fast-export and git fast-import.
Its goal to to fix the trailing spaces, tab issues and to optionally
exclude or filter-out a specific module and/or filter-out files with a
specific extension.
2.1.1 arguments
lo_git_rewrite understand to following command line argument. (note
the syntax is --foo bar and NOT --foo=bar)
all these arguments are optional
--prefix string
This is used to prefix output message to stderr with the specified
string. this is used in onegit.sh because more than one instance of
lo_git_rewrite is running in parallel, and this allow to link a
message to a specific lo_git_rewrite instance
the default is an empty string.
--exclude-module module_name
This tell lo_git_rewrite to filter-out any files whose name start with
module_name/. This is used in onegit.sh to filter-out a module from a
given repo, like binfilter or dictionaries.
--fitter-module module_name
This tell lo_git_rewrite to filter-out any files whose name does _not_
start with module_name/. This is used in onegit.sh to extract a given
module from an existing repo, like binfilter or dictionaries.
--exclude-suffix string
This tell lo_git_rewrite to exclude any files whose name end with
string. This is used in onegit.sh to eliminated obsolete .tar.gz
file out of libs-extern-sys and libs-extern history.
--buffer-size nnn
This tell lo_git_rewrite to allocate a working buffer of nnn MB.
