On pátek 9. prosince 2016 19:05:48 CET Emilian Bold wrote: > Martin, I have just pushed https://github.com/emilianbold/main-silver You > may experiment with that.
Hello Emilian, I managed to fork & use your repository and everything seems great. I have a functional job that executes $ ant build-platform for each pull request. I plan to add a call to "ant test-platform" once it is stable enough[1]. Great work! I believe we shall use your Git repository as a base (somehow) when donating code to Apache once my Oracle peers finish review of the code to donate. > To https://github.com/emilianbold/main-silver.git > * [new branch] master -> master > Branch master set up to track remote branch master from origin. Could you synchronize https://hg.netbeans.org/releases/ instead? It contains history of all the NetBeans releases (in branches like release82, etc.) and it is the repository that is currently under the review. Btw. The releases repository contains everything that is available in the main-silver - just more. It would be fantastic, if you could create a complete mirror of the releases repository. Thanks again for your great work! -jt [1] http://deadlock.netbeans.org/job/prototypes-MavenDownload269264/ - still 12 test failures remaining > git push -u origin master > Counting objects: 3951610, done. > Delta compression using up to 8 threads. > Compressing objects: 100% (732965/732965), done. > Writing objects: 100% (3951610/3951610), 674.94 MiB | 717.00 KiB/s, done. > Total 3951610 (delta 2068729), reused 3951610 (delta 2068729) > remote: Resolving deltas: 100% (2068729/2068729), done. > remote: Checking connectivity: 3951610, done. > remote: warning: GH001: Large files detected. You may want to try Git Large > File Storage - https://git-lfs.github.com. > remote: warning: See http://git.io/iEPt8g for more information. > remote: warning: File dlight.util/test/manual/DLight_Simple_Tests/core is > 51.88 MB; this is larger than GitHub's recommended maximum file size of > 50.00 MB > To https://github.com/emilianbold/main-silver.git > * [new branch] master -> master > Branch master set up to track remote branch master from origin. > > > --emi > > On Wed, Dec 7, 2016 at 4:42 PM, Martin Balin <[email protected]> > > wrote: > > Hello Emilian, > > I'm working at Oracle on NetBeans development and we would like to start > > fixing build scripts to use Git instead of HG. > > This could be done earlier on your Git repo if you agree to as it will > > take time. Does not need to wait for final official donation of sources. > > Can you please send me the URL,... > > Thank you Martin Balin > > > > On 24.11.2016 20:07, Emilian Bold wrote: > >> At under 1GB the repository size is not an issue anymore. > >> > >> It's sad to see we will still have migration problems due to legal > >> considerations. > >> > >> Could you provide an estimate how long it would take to verify and > >> whitelist the entire codebase Oracle plans on donating? > >> > >> It's unclear to me how history would be preserved with an incremental > >> approach. > >> > >> I would prefer we migrate the whole thing in one piece with history and > >> all. > >> > >> > >> --emi > >> > >> On Thu, Nov 24, 2016 at 5:22 PM, Jaroslav Tulach < > >> [email protected] > >> > >>> wrote: > >>> Emilian, Jan, Mark, great work. > >>> > >>> Smooth migration from Hg to Git is essential for successful migration to > >>> Apache. Thanks a lot for investigating how to do that. > >>> > >>> My plan (as described in another email) is to prepare the code donation > >>> in > >>> Hg > >>> and update it incrementally with code integrated into Hg. > >>> > >>> Are your conversions methods ready for incremental updates or do they > >>> only > >>> work as a one-time batch conversion? > >>> > >>> -jt > >>> > >>> On čtvrtek 24. listopadu 2016 10:41:50 CET Jan Lahoda wrote: > >>>> Interesting. I tried "git gc --aggressive" on the Mark's converted > >>>> repository, and the result is: > >>>> netbeans-import/.git$ du -hs . > >>>> 792M . > >>>> > >>>> The original was: > >>>> netbeans-import.git $ du -hs . > >>>> 3,5G . > >>>> > >>>> (IIRC Mark was converting http://hg.netbeans.org/main, not releases, so > >>> > >>> the > >>> > >>>> repository is a little bit smaller than the releases one.) > >>>> > >>>> I tried: > >>>> $ git log -p | sha1sum > >>>> > >>>> on both repositories, and the hashes appear to be the same. I also > >>>> tried > >>> > >>> to > >>> > >>>> clone the gc-ed repository using git clone --bare --no-local, and the > >>>> resulting repository is still about the same size. So, this seems good > >>>> to > >>>> me, unless there is some downside I don't know about. > >>>> > >>>> Jan > >>>> > >>>> > >>>> On Wed, Nov 23, 2016 at 8:26 PM, Emilian Bold <[email protected]> > >>>> > >>>> wrote: > >>>>> Actually I don't believe the data loss is that large. (There may also > >>>> > >>>> be > >>>> > >>>> mercurial commits that are intentionally ignored by the conversion > >>>> > >>>> script, > >>>> > >>>> like commits that only add tags?) > >>>> > >>>>> hg log | grep '^changeset:' | wc -l > >>>>> > >>>>> 313209 > >>>>> > >>>>> git log | grep '^commit ' | wc -l > >>>>> > >>>>> 301478 > >>>>> > >>>>> So there is a difference of 11731 commits (about 4%) but those > >>>>> couldn't > >>>>> have such a large impact on repository size. > >>>>> > >>>>> I hope somebody else is willing to work with me on this so we document > >>>>> everything and do a reproducible repository conversion. > >>>>> > >>>>> > >>>>> > >>>>> --emi > >>>>> > >>>>> On Wed, Nov 23, 2016 at 9:10 PM, Emilian Bold <[email protected]> > >>>>> > >>>>> wrote: > >>>>>> Well, I dunno what black magic `gc --aggressive` does but the > >>>>> > >>>>> repository > >>>> > >>>> is 0.85GB now! > >>>> > >>>>>> I also ran `git reflog expire` first but it didn't change the size at > >>>>> > >>>>> all. > >>>>> > >>>>> One thing to keep in mind is that I used --force although I had 6 > >>>>> > >>>>>> commits > >>>>>> with the warning "repository has at least one unnamed head". Which > >>>>> > >>>>> were > >>>> > >>>> probably all close branch commits (hg commit --close-branch). > >>>> > >>>>>> So I might have have data loss(!) since I believe I read > >>>>> > >>>>> hg-fast-export.sh > >>>>> > >>>>> picks only one unnamed head as the migration winner. I wonder if the > >>>>> > >>>>> gc > >>>> > >>>> command didn't just purge a lot of valid commits from such an unnamed > >>>> > >>>>> head > >>>>> > >>>>> and that's why the repository became so small. > >>>>> > >>>>>> Could somebody else try a test repository conversion and validate my > >>>>>> numbers? > >>>>>> > >>>>>> git gc --aggressive --prune=now > >>>>>> Counting objects: 4085031, done. > >>>>>> Delta compression using up to 8 threads. > >>>>>> Compressing objects: 100% (2909203/2909203), done. > >>>>>> Writing objects: 100% (4085031/4085031), done. > >>>>>> Total 4085031 (delta 2150468), reused 1585934 (delta 0) > >>>>>> Checking connectivity: 4085031, done. > >>>>>> > >>>>>> > >>>>>> > >>>>>> --emi > >>>>>> > >>>>>> On Wed, Nov 23, 2016 at 7:59 PM, Paul Merlin <[email protected]> > >>>>>> > >>>>>> wrote: > >>>>>>> Hi Emilian, > >>>>>>> > >>>>>>> I see hg-fast-export.sh finished at some point. > >>>>>>> > >>>>>>>> As expected though, git does not have any of the disk space gains. > >>>>>>>> The > >>>>>>>> converted git releases/ repository is 3.6GB. > >>>>>>> > >>>>>>> Just a thought. > >>>>>>> Did you try some git cleanups after the conversion? > >>>>>>> > >>>>>>> git reflog expire --expire=now --all > >>>>>>> git gc --aggressive --prune=now > >>>>>>> > >>>>>>> Cheers > >>>>>>> > >>>>>>> In case these statistics mean something: > >>>>>>>> git-fast-import statistics: > >>>>>>>> ------------------------------------------------------------ > >>>>>>> > >>>>>>> --------- > >>>> > >>>> Alloc'd objects: 4090000 > >>>> > >>>>>>>> Total objects: 4085509 ( 40220100 duplicates > >>>>>>>> > >>>>>>> ) > >>>>>>> > >>>> blobs : 1036365 ( 28386238 duplicates 858087 > >>>>>>> > >>>>>>> deltas > >>>> > >>>> of > >>>> > >>>>> 969684 attempts) > >>>>> > >>>>>>>> trees : 2735935 ( 11833862 duplicates 1370606 > >>>>>>> > >>>>>>> deltas > >>>> > >>>> of > >>>> > >>>>> 2613480 attempts) > >>>>> > >>>>>>>> commits: 313209 ( 0 duplicates 0 > >>>>>>> > >>>>>>> deltas > >>>> > >>>> of > >>>> > >>>>> 0 attempts) > >>>>> > >>>>>>>> tags : 0 ( 0 duplicates 0 > >>>>>>> > >>>>>>> deltas > >>>> > >>>> of > >>>> > >>>>> 0 attempts) > >>>>>>>> > >>>>>>>> Total branches: 1283 ( 346 loads ) > >>>>>>>> > >>>>>>>> marks: 1048576 ( 313209 unique ) > >>>>>>>> atoms: 124011 > >>>>>>>> > >>>>>>>> Memory total: 218429 KiB > >>>>>>>> > >>>>>>>> pools: 26711 KiB > >>>>>>>> > >>>>>>>> objects: 191718 KiB > >>>>>>>> > >>>>>>>> ------------------------------------------------------------ > >>>>>>> > >>>>>>> --------- > >>>> > >>>> pack_report: getpagesize() = 4096 > >>>> > >>>>>>>> pack_report: core.packedGitWindowSize = 1073741824 > >>>>>>>> pack_report: core.packedGitLimit = 8589934592 > >>>>>>>> pack_report: pack_used_ctr = 39000045 > >>>>>>>> pack_report: pack_mmap_calls = 733040 > >>>>>>>> pack_report: pack_open_windows = 4 / 7 > >>>>>>>> pack_report: pack_mapped = 4280730006 / 6950823920 > >>>>>>>> ------------------------------------------------------------ > >>>>>>> > >>>>>>> --------- > >>>>>>> > >>>>>>>> --emi > >>>>>>>> > >>>>>>>> On Fri, Nov 18, 2016 at 1:32 PM, Emilian Bold < > >>>>>>> > >>>>>>> [email protected] > >>>> > >>>> wrote: > >>>>>>>>> A releases/ clone which on my system takes 3.8GB is reduced to > >>>>>>>> > >>>>>>>> 1.6GB > >>>> > >>>> with > >>>> > >>>>>>> the generaldelta and aggressivemergedeltas flags (took about 14 > >>>>>>> > >>>>>>>> hours). > >>>>> > >>>>> Pretty impressive! > >>>>> > >>>>>>>>> Converting to git with hg-fast-export.sh complains that > >>>>>>>> > >>>>>>>> "repository > >>>> > >>>> has at > >>>> > >>>>>>> least one unnamed head" for about 6 revisions. With --force I'm > >>>>>>> > >>>>>>>> able > >>>> > >>>> to > >>>> > >>>>> start the conversion but it hasn't finished yet. > >>>>> > >>>>>>>>> The git conversion is about 35% done and already using 1.3GB. > >>>>>>>>> > >>>>>>>>> So... I assume it's going to need just like the original > >>>>>>>> > >>>>>>>> repository > >>>> > >>>> about > >>>> > >>>>>>> 3.8GB. > >>>>>>> > >>>>>>>>> I wonder if git has similar space-saving tricks? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> --emi > >>>>>>>>> > >>>>>>>>> On Thu, Nov 17, 2016 at 8:46 AM, Emilian Bold < > >>>>>>>> > >>>>>>>> [email protected]> > >>>>> > >>>>> wrote: > >>>>>>>>>> Forgot about this. I've just started the Mercurial repository > >>>>>>>>> > >>>>>>>>> conversion > >>>>>>> > >>>>>>> which will take a few hours. > >>>>>>> > >>>>>>>>>> Will report tomorrow or when it's done. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> --emi > >>>>>>>>>> > >>>>>>>>>> On Wed, Nov 16, 2016 at 11:18 PM, cowwoc < > >>>>>>>>> > >>>>>>>>> [email protected]> > >>>> > >>>> wrote: > >>>>>>>> Hi Emilian, > >>>>>>>> > >>>>>>>>>>> Any update on this? > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Gili > >>>>>>>>>>> > >>>>>>>>>>> On 2016-11-11 01:33 (-0500), Emilian Bold <[email protected]> > >>>>>>>>>> > >>>>>>>>>> wrote: > >>>> Thank you for following through with this after we talked on > >>>> > >>>>>>>>>>>> IRC.> > >>>>>>>>>>>> > >>>>>>>>>>>> I will check later the size reduction for the releases/ repo.>
