On Mon, Jun 13, 2011 at 8:05 AM, Jeffrey Naisbitt <[email protected]>wrote:
> When I checkout the yahoo-merge branch, I see these svn externals warnings: > svn: warning: Error handling externals definition for > 'yahoo-merge/hdfs/src/test/bin': > svn: warning: URL > 'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at > revision 1135120 doesn't exist > svn: warning: Error handling externals definition for > 'yahoo-merge/mapreduce/src/test/bin': > svn: warning: URL > 'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at > revision 1135120 doesn't exist > Oops, sorry about that one. I will take care of that in about 30 minutes (just headed out the door now to catch a train). If someone else with commit access wants to, you just need to propset the externals to point to th new common/trunk/common/src/test/bin instead of the old location. > Also, the ant eclipse targets seem to be broken now. It seems like various > parts of the eclipse target need to be commonized now (the > .eclipse-templates stuff and .classpath, .launches, etc.) > > Will look into this as well. -Todd > > On 6/12/11 6:38 PM, "Todd Lipcon" <[email protected]> wrote: > > > OK, this seems to have succeeded without any big problems! > > > > I've re-enabled the git mirrors and the hudson builds. Feel free to > commit > > to the new trees. > > > > Here are some instructions for the migration: > > > > === SVN users === > > > > Next time you "svn up" in your "common" working directory you'll end up > > seeing the combined tree - ie a mapreduce/, hdfs/, and common/ > subdirectory. > > This is probably the easiest place from which to work, now. The URLs for > the > > combined SVN trees are: > > > > trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/ > > branch-0.22: > > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22 > > branch-0.21: > > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 > > yahoo-merge: > > http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge > > (this one has the yahoo-merge branches from common, hdfs, and mapred) > > MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 > > (this one has the yahoo-merge common and hdfs, and the MR-279 mapred) > > > > The same kind of thing happened for HDFS-1073 and branch-0.21-old. > > Pre-project-split branches like branch-0.20 should have remained > untouched. > > > > You can proceed to delete your checkouts of the individual mapred and > hdfs > > trees, since they exist within the combined trees above. If for some > reason > > you prefer to 'svn switch' an old MR or HDFS-specific checkout to point > to > > its new location, you can use the following incantation: > > svn sw $(svn info | grep URL | awk '{print $2}' | sed > > 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,') > > > > === Git Users === > > The git mirrors of the above 7 branches should now have a set of 4 > commits > > near the top that look like this: > > > > Merge: 928d485 cd66945 77f628f > > Author: Todd Lipcon <[email protected]> > > Date: Sun Jun 12 22:53:28 2011 +0000 > > > > HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in > a > > single tree (project unsplit) > > > > git-svn-id: > > > https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310 > > -9956-ffa450edef68 > > > > commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b > > Author: Todd Lipcon <[email protected]> > > Date: Sun Jun 12 22:53:27 2011 +0000 > > > > Relocate mapreduce into mapreduce/ > > > > commit cd66945f62635f589ff93468e94c0039684a8b6d > > Author: Todd Lipcon <[email protected]> > > Date: Sun Jun 12 22:53:26 2011 +0000 > > > > Relocate hdfs into hdfs/ > > > > commit 928d485e2743115fe37f9d123ce9a635c5afb91a > > Author: Todd Lipcon <[email protected]> > > Date: Sun Jun 12 22:53:25 2011 +0000 > > > > Relocate common into common/ > > > > The first of these 4 is a 3-parent "octopus" merge commit of the > > pre-project-unsplit branches. In theory, git is smart enough to track > > changes through this merge, so long as you pass the right flags (eg > > --follow). For example: > > > > todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline > --abbrev-commit > > --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | > head > > -10 > > 77f628f Relocate mapreduce into mapreduce/ > > 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of > > JobTrackerStatus. > > ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to > > aid diagnosis of related issues. Contributed by Jonathan Eagles > > 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated > options. > > Contributed by Ari Rabkin. > > > > If you want to be able to have git follow renames all the way through the > > project split back to the beginning of time, put the following in > > hadoop-common/.git/info/grafts: > > 5128a9a453d64bfe1ed978cf9ffed27985eeef36 > > 6c16dc8cf2b28818c852e95302920a278d07ad0c > > 6a3ac690e493c7da45bbf2ae2054768c427fd0e1 > > 6c16dc8cf2b28818c852e95302920a278d07ad0c > > 546d96754ffee3142bcbbf4563c624c053d0ed0d > > 6c16dc8cf2b28818c852e95302920a278d07ad0c > > > > In terms of rebasing git branches, git is actually pretty smart. For > > example, I have a local "HDFS-1073" branch in my hdfs repo. To transition > it > > to the new combined repo, I did the following: > > > > # Add my project-split hdfs git repo as a remote: > > git remote add splithdfs /home/todd/git/hadoop-hdfs/ > > git fetch splithdfs > > > > # Checkout a branch in my combined repo > > git checkout -b HDFS-1073 splithdfs/HDFS-1073 > > > > # Rebase it on the combined 1073 branch > > git rebase origin/HDFS-1073 > > > > ...and it actually applies my patches inside the appropriate subdirectory > (I > > was surprised and impressed by this!) > > If the branch you're rebasing has added or moved files, it might not be > > smart enough and you'll have to manually rename them in your branch > inside > > of the appropriate subtree.. but for simple patches this seems to work. > For > > less simple things, the best bet may be to use "git filter-branch" on the > > patch series to relocate it inside a subdirectory, and then try to > rebase. > > Let me know if you need a hand with any git cleanup, happy to help. > > > > > > == Outstanding issues == > > > > The one outstanding issue I'm aware of is that the test-patch builds > should > > be smart enough to be able to deal with patches that are relative to the > > combined root instead of the original project. Right now, if you export a > > diff from git, it will include "hdfs/" or "mapreduce/" in the changed > file > > names, and the QA bot won't know how to apply it. The workaround for this > is > > to change directory into the relative subproject dir, and then pass > > "--relative" to "git diff" or "git show", for example: > > > > todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative > > --no-prefix > > diff --git CHANGES.txt CHANGES.txt > > ... > > > > > > I imagine there are probably some other things that fell through the > cracks. > > Please get in touch if there's anything that seems amiss. > > > > -Todd > > > > > > On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <[email protected]> wrote: > > > >> All of the nits I ran into should be resolved and we should be good to > go. > >> I will start this in just about 10 minutes (3pm PST). > >> > >> ***Please hold all commits until further notice!*** I anticipate that > this > >> should take under an hour, but if there are any bumps along the way it > might > >> stretch into the evening. I'll send out an "all clear" email when things > are > >> ready to go on the new layout. > >> > >> I've disabled all of the Hudson builds for now and will be re-enabling > them > >> one by one after reconfiguring their SVN URLs. > >> > >> -Todd > >> > >> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <[email protected]> wrote: > >> > >>> Hi all, > >>> > >>> I'm figuring out one more small nit I noticed in my testing this > evening. > >>> Hopefully I will figure out what's going wrong and be ready to press > the big > >>> button tomorrow. > >>> > >>> Assuming I don't have to "abort mission", my hope is to do this at > around > >>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to > please > >>> hold commits to all branches while the move is in progress. > >>> > >>> Thanks > >>> -Todd > >>> > >>> > >>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <[email protected]> > wrote: > >>> > >>>> Hi all, > >>>> > >>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106 > >>>> this weekend. I have the credentials from Jukka to take care of the > git > >>>> trees as well, and have done a "practice" move several times on a > local > >>>> mirror of the svn. > >>>> > >>>> I'll send out an announcement of the exact time in advance of when I > >>>> actually do the commit. > >>>> > >>>> Thanks > >>>> -Todd > >>>> -- > >>>> Todd Lipcon > >>>> Software Engineer, Cloudera > >>>> > >>> > >>> > >>> > >>> -- > >>> Todd Lipcon > >>> Software Engineer, Cloudera > >>> > >> > >> > >> > >> -- > >> Todd Lipcon > >> Software Engineer, Cloudera > >> > > > > > > -- Todd Lipcon Software Engineer, Cloudera
