Mike, I just cloned your repo and all of the tests in 'ocw' passed on my machine.
Cheers, Cameron On Mar 14, 2014 12:25 PM, "Michael Joyce" <[email protected]> wrote: > Update on this. I've tested this on a fork to make sure that it works > properly. Everything seems fine. I've taken the old repository and removed > all the old large binaries and leftover JPL related things. I've gone > through and run some tests, including a sample evaluation through the UI > and the backend, and everything seems kosher. > > The repo is down to ~13 MB now. > > It would be a big help if everyone could go pull down the new code [1] and > make sure everything looks alright. I'll wait a bit before pushing any > changes to the ASF so everyone has adequate time to test. > > [1] https://github.com/MJJoyce/climate > > > > > -- Joyce > > > On Wed, Mar 12, 2014 at 2:49 PM, Michael Joyce <[email protected]> wrote: > > > Awesome, I'm glad we got this sorted out. Thanks for all the hard work! > > > > > > -- Joyce > > > > > > On Wed, Mar 12, 2014 at 2:45 PM, denis.nadeau <[email protected] > >wrote: > > > >> Happy to be your project guinea pig ! :-> > >> > >> After recompiling "git" with "libcurl" (./configure --with-curl) I was > >> able to push the changes. You need curl to get access to https (seems > >> like). > >> Your git remote command was very useful. I did not have to clone the > >> repo and copy my files over and redo the "git commands". > >> > >> I made quite some changes to obs4MIPs and need to "diff" and push > changes > >> to the repository. So it seems that, I am good to go with "git". :-) > >> > >> Great work and thanks for your help! > >> Denis > >> > >> > >> On 3/12/14 5:01 PM, Michael Joyce wrote: > >> > >>> Ah good, we're getting close! We'd be even closer if I hadn't messed up > >>> in > >>> a previous git related email! > >>> > >>> Our git://git.apache.org/climate.git mirror is our read only git > mirror. > >>> That would explain why you aren't able to write to it. > >>> > >>> We need to use: > >>> https://git-wip-us.apache.org/repos/asf/climate.git > >>> > >>> If we didn't have a commit bit we would instead use (http vs https) > >>> http://git-wip-us.apache.org/repos/asf/reponame.git > >>> > >>> I misread some documentation at [1] and [2] and confused myself. I > >>> thought > >>> the "WIP" or "Work in Progress" label was for migration only. Silly me. > >>> > >>> We can fix this fairly easily by running > >>> $ git remote set-url origin > >>> https://git-wip-us.apache.org/repos/asf/climate.gi > >>> > >>> Then, you should see updated URLs with > >>> $ git remote -v > >>> > >>> At that point you should be able to push successfully. > >>> > >>> Sorry that you've turned into our project guinea pig Denis! I had hoped > >>> to > >>> smooth out some of these rough edges this last weekend/early this week > >>> but > >>> unfortunately I haven't been able to do so. We'll get there though! > >>> > >>> [1] https://www.apache.org/dev/writable-git > >>> [2] https://git-wip-us.apache.org/ > >>> > >>> > >>> -- Joyce > >>> > >>> > >>> On Wed, Mar 12, 2014 at 12:44 PM, denis.nadeau <[email protected] > >>> >wrote: > >>> > >>> Joyce, > >>>> > >>>> This is great introduction and will help other SVN/CVS developers. > (I > >>>> did not know you had to "git add" every changes.) > >>>> > >>>> Right now, I just can't push to github. I think it might be a > >>>> configuration issue. Do you need my ssh keys or something for me to > >>>> 'push'? > >>>> > >>>> git push origin master > >>>> > >>>> fatal: The remote end hung up unexpectedly > >>>> > >>>> > >>>> git status > >>>> > >>>> # On branch master > >>>> # Your branch is ahead of 'origin/master' by 4 commits. > >>>> # > >>>> nothing to commit (working directory clean) > >>>> > >>>> git remote -v > >>>> > >>>> origin git://git.apache.org/climate.git (fetch) > >>>> origin git://git.apache.org/climate.git (push) > >>>> > >>>> Thanks for your help. (almost there...) > >>>> Denis > >>>> > >>>> > >>>> On 3/12/14 12:37 PM, Michael Joyce wrote: > >>>> > >>>> Ah, let me explain since git is just a bit different from SVN. > >>>>> > >>>>> When you commit in git you aren't actually committing to the primary > >>>>> server > >>>>> like you are in SVN. You're committing to your local working copy. In > >>>>> order > >>>>> to mirror those changes to the ASF you will need to run "git push". > So > >>>>> "git > >>>>> status" is telling you that you've committed 4 times and those > changes > >>>>> aren't mirrored on the server that you ran "git clone" from. > >>>>> > >>>>> To be safe, you might want to checkout a clean copy of the repo from > >>>>> the > >>>>> ASF (which should only take forever =) and then try again. We could > go > >>>>> through each of the commits and make sure they're the way you want > >>>>> them to > >>>>> be, but that might end up being more trouble than it's worth if we > try > >>>>> to > >>>>> do it via email. This is the workflow that I would probably follow: > >>>>> > >>>>> # Remove the files that you don't want anymore. I'm going to say that > >>>>> we're > >>>>> # sitting in the root of our repo and the files are in > >>>>> '/obs4MIPs/examples' > >>>>> $ git rm -r obs4MIPs/examples > >>>>> $ git status > >>>>> # You should now see a number of files being marked as "staged for > >>>>> commit". > >>>>> # Go ahead commit these removals > >>>>> $ git commit -m "Removing obs4MIPs example .nc files" > >>>>> > >>>>> # Now if you run git status you shouldn't see any files listed, but > it > >>>>> will > >>>>> say > >>>>> # that you're ahead of origin/master by 1 commit > >>>>> > >>>>> # Now add the readme/or update any other files > >>>>> > >>>>> $ git add . > >>>>> # It's important to note that "add" in git is not the same as "add" > in > >>>>> svn. > >>>>> Add in git means > >>>>> # "add/stage these changes for the next commit". If you're used to > svn > >>>>> this > >>>>> can be a bit > >>>>> # confusing. In git you need to add changes every time you want to > >>>>> commit, > >>>>> as opposed > >>>>> # to svn where you only "add" the file to the repo once. > >>>>> > >>>>> $ git status > >>>>> # You should see all the files that you changed present and "staged > for > >>>>> commit". When > >>>>> # something is "staged for commit" that means that it will be > committed > >>>>> next time we > >>>>> # run git commit. > >>>>> $ git commit -m "Update blah blah blah" > >>>>> > >>>>> # Now you should see that you're ahead by a few commits depending on > >>>>> how > >>>>> many times you've committed. > >>>>> # At this point you probably want to share all your changes with > >>>>> everyone, > >>>>> so we'll push the changes up to the server. > >>>>> > >>>>> # You really can abbreviate this to just 'git push' or 'git push > >>>>> origin'. > >>>>> We're going to play it safe and be super explicit. > >>>>> # This is telling git to push all the changes that you've committed > in > >>>>> your > >>>>> 'master' branch > >>>>> # (which is the default one that you've been working in) to the > remote > >>>>> named "origin". By default, > >>>>> # the repo that you cloned from is named "origin". > >>>>> $ git push origin master > >>>>> > >>>>> At this point we should get some emails saying you committed. > >>>>> > >>>>> Hopefully that helps a bit. If you have more questions let me know. > It > >>>>> can > >>>>> certainly be a bit jarring of a change moving to git from svn. I'm > >>>>> working > >>>>> on writing up a brief "intro to git" that I will send around to the > >>>>> mailing > >>>>> list once it's in a useful state. It should hopefully help clear up > >>>>> some > >>>>> confusion for everyone. > >>>>> > >>>>> > >>>>> -- Joyce > >>>>> > >>>>> > >>>>> On Wed, Mar 12, 2014 at 9:04 AM, denis.nadeau <[email protected] > > > >>>>> wrote: > >>>>> > >>>>> Joyce, > >>>>> > >>>>>> I did commit the change and also remove "rm" the .nc files. I did > not > >>>>>> see > >>>>>> an email either. Here are the 3 commands I used > >>>>>> > >>>>>> 1. git add > >>>>>> 2. git commit > >>>>>> 3. git rm > >>>>>> > >>>>>> > >>>>>> I guess "git rm" does not need a commit command. > >>>>>> When I run "git status" I get this message. I am not sure what > "ahead > >>>>>> of > >>>>>> 'origin/master' by 4 commits" means! > >>>>>> > >>>>>> git status > >>>>>> # On branch master > >>>>>> # Your branch is ahead of 'origin/master' by 4 commits. > >>>>>> # > >>>>>> nothing to commit (working directory clean) > >>>>>> > >>>>>> > >>>>>> Denis > >>>>>> > >>>>>> On 3/12/14 11:19 AM, Michael Joyce wrote: > >>>>>> > >>>>>> Awesome Denis thanks much. I will play around with this more soon > >>>>>> and > >>>>>> > >>>>>>> see > >>>>>>> if I can't strip out some more files. Did you push your changes up > to > >>>>>>> the > >>>>>>> repo? I didn't see a commit email come through, but I'm not certain > >>>>>>> my > >>>>>>> filters are working correctly with the mailing list migrations. > >>>>>>> > >>>>>>> > >>>>>>> -- Joyce > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Mar 12, 2014 at 7:20 AM, denis.nadeau < > [email protected] > >>>>>>> > > >>>>>>> wrote: > >>>>>>> > >>>>>>> Joyce, > >>>>>>> > >>>>>>> I deleted the .nc files found in my example directory for TRMM and > >>>>>>>> ECMWF. > >>>>>>>> I have installed a README file and explain users how to > >>>>>>>> retrieve the > >>>>>>>> data > >>>>>>>> from the original data provider. TRMM is pretty > straightforward, > >>>>>>>> but > >>>>>>>> for > >>>>>>>> ECMWF you need to register, obtain a key and download their Python > >>>>>>>> package. > >>>>>>>> > >>>>>>>> It works pretty well on my machine, let see what users say. > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Denis > >>>>>>>> > >>>>>>>> > >>>>>>>> On 3/10/14 3:53 PM, Michael Joyce wrote: > >>>>>>>> > >>>>>>>> I think that would be great Denis! I can go ahead and look at > >>>>>>>> doing > >>>>>>>> > >>>>>>>> something similar for the other ocw/ocw-ui components as well. > I'm > >>>>>>>>> sure > >>>>>>>>> this will help us out a good bit. > >>>>>>>>> > >>>>>>>>> Thanks! > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- Joyce > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Mar 10, 2014 at 11:20 AM, denis.nadeau < > >>>>>>>>> [email protected] > >>>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Michael, > >>>>>>>>>> > >>>>>>>>> I like the idea of having the NetCDF files in a external > >>>>>>>>> repository. > >>>>>>>>> > >>>>>>>>>> I was thinking that it might be better to point the people to > >>>>>>>>>> satellite > >>>>>>>>>> data at the different DAACs so that they can download the files > >>>>>>>>>> directly. > >>>>>>>>>> That would work for the "obs4MIPs" program. I would feel > >>>>>>>>>> better > >>>>>>>>>> about > >>>>>>>>>> it as well, I have been worried to be told by some data > >>>>>>>>>> providers > >>>>>>>>>> (ECMWF) > >>>>>>>>>> that we are not authorized to distribute their original data. > I > >>>>>>>>>> initially > >>>>>>>>>> did not think about this when I checked in my original code. > >>>>>>>>>> > >>>>>>>>>> I just found out that ECMWF now allows people to download their > >>>>>>>>>> data > >>>>>>>>>> in > >>>>>>>>>> "NetCDF" instead of "GRIB" using Python [1]. I tried it > before, > >>>>>>>>>> but > >>>>>>>>>> could > >>>>>>>>>> only retrieve GRIB data and did not want to mess with "Grads" > ctl > >>>>>>>>>> files > >>>>>>>>>> and > >>>>>>>>>> CDMS2/CDAT package. So now, I could just create a script to > >>>>>>>>>> download > >>>>>>>>>> the > >>>>>>>>>> right files and rename them to the appropriate filenames for > >>>>>>>>>> obs4MIPs > >>>>>>>>>> examples. > >>>>>>>>>> > >>>>>>>>>> I would feel much better about this. Let me know what you > think. > >>>>>>>>>> > >>>>>>>>>> [1] https://software.ecmwf.int/wiki/display/WEBAPI/Accessing+ > >>>>>>>>>> ECMWF+data+servers+in+batch > >>>>>>>>>> > >>>>>>>>>> Denis > >>>>>>>>>> > >>>>>>>>>> On 3/10/14 1:06 PM, Michael Joyce wrote: > >>>>>>>>>> > >>>>>>>>>> Hi guys, > >>>>>>>>>> > >>>>>>>>>> An unfortunate side effect of our export from SVN to Git is > that > >>>>>>>>>> > >>>>>>>>>>> we've > >>>>>>>>>>> ended up with a rather bloated repository. We've had a large > >>>>>>>>>>> number > >>>>>>>>>>> of > >>>>>>>>>>> binary files in our repo in the past and all of this has been > >>>>>>>>>>> rolled > >>>>>>>>>>> up > >>>>>>>>>>> into a obnoxious ~500 MB pack file. I've been completely unable > >>>>>>>>>>> to > >>>>>>>>>>> clone > >>>>>>>>>>> the repo on my home internet because it constantly times out > and > >>>>>>>>>>> it's > >>>>>>>>>>> painfully slow on my faster work connection. > >>>>>>>>>>> > >>>>>>>>>>> To fix this problem I suggest we do the following: > >>>>>>>>>>> - Remove all binary files from our repo and host them > externally. > >>>>>>>>>>> For > >>>>>>>>>>> example, NetCDF files can be downloaded when they're needed and > >>>>>>>>>>> cleaned > >>>>>>>>>>> up > >>>>>>>>>>> afterwards (for tests or examples). > >>>>>>>>>>> - Remove all the bloat from our pack file. I was digging > through > >>>>>>>>>>> stuff > >>>>>>>>>>> earlier and found a number of very large and outdated files in > >>>>>>>>>>> our > >>>>>>>>>>> pack > >>>>>>>>>>> file (~300 MB NC file, internal JPL presentations/files from a > >>>>>>>>>>> long > >>>>>>>>>>> time > >>>>>>>>>>> ago, etc.). We should be able to use [1] to help automate this > >>>>>>>>>>> for > >>>>>>>>>>> us, > >>>>>>>>>>> although we can also take care of it on our own if need be. > >>>>>>>>>>> > >>>>>>>>>>> Let me know what you guys think the best course of action is. > >>>>>>>>>>> That > >>>>>>>>>>> being > >>>>>>>>>>> said, dealing with this sooner rather than later would be nice > =D > >>>>>>>>>>> > >>>>>>>>>>> [1] https://github.com/cmaitchison/git_diet > >>>>>>>>>>> > >>>>>>>>>>> -- Joyce > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> > >>>>>>>>>>> ----------------------------------------------------- > >>>>>>>>>>> > >>>>>>>>>> Denis Nadeau, (CSC) > >>>>>>>>>> NCCS (NASA Center for Climate Simulation) > >>>>>>>>>> NASA Goddard Space Flight Center > >>>>>>>>>> Mailcode 606.2 > >>>>>>>>>> 8800 Greenbelt Road > >>>>>>>>>> Greenbelt, MD 20771 > >>>>>>>>>> Email: [email protected] > >>>>>>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 > >>>>>>>>>> ----------------------------------------------------- > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> > >>>>>>>>>> ----------------------------------------------------- > >>>>>>>>> > >>>>>>>> Denis Nadeau, (CSC) > >>>>>>>> NCCS (NASA Center for Climate Simulation) > >>>>>>>> NASA Goddard Space Flight Center > >>>>>>>> Mailcode 606.2 > >>>>>>>> 8800 Greenbelt Road > >>>>>>>> Greenbelt, MD 20771 > >>>>>>>> Email: [email protected] > >>>>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 > >>>>>>>> ----------------------------------------------------- > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> > >>>>>>> ----------------------------------------------------- > >>>>>> Denis Nadeau, (CSC) > >>>>>> NCCS (NASA Center for Climate Simulation) > >>>>>> NASA Goddard Space Flight Center > >>>>>> Mailcode 606.2 > >>>>>> 8800 Greenbelt Road > >>>>>> Greenbelt, MD 20771 > >>>>>> Email: [email protected] > >>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 > >>>>>> ----------------------------------------------------- > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>> ----------------------------------------------------- > >>>> Denis Nadeau, (CSC) > >>>> NCCS (NASA Center for Climate Simulation) > >>>> NASA Goddard Space Flight Center > >>>> Mailcode 606.2 > >>>> 8800 Greenbelt Road > >>>> Greenbelt, MD 20771 > >>>> Email: [email protected] > >>>> Phone: (301) 286-7286 Fax: 301.286.1634 > >>>> ----------------------------------------------------- > >>>> > >>>> > >>>> > >> > >> -- > >> ----------------------------------------------------- > >> Denis Nadeau, (CSC) > >> NCCS (NASA Center for Climate Simulation) > >> NASA Goddard Space Flight Center > >> Mailcode 606.2 > >> 8800 Greenbelt Road > >> Greenbelt, MD 20771 > >> Email: [email protected] > >> Phone: (301) 286-7286 Fax: 301.286.1634 > >> ----------------------------------------------------- > >> > >> > > >
