All, I'm going to push these changes now. Checkout CLIMATE-384 for updates
-- Joyce On Fri, Mar 14, 2014 at 2:59 PM, Cameron Goodale <[email protected]> wrote: > Mike, > > I just cloned your repo and all of the tests in 'ocw' passed on my machine. > > Cheers, > > Cameron > On Mar 14, 2014 12:25 PM, "Michael Joyce" <[email protected]> wrote: > > > Update on this. I've tested this on a fork to make sure that it works > > properly. Everything seems fine. I've taken the old repository and > removed > > all the old large binaries and leftover JPL related things. I've gone > > through and run some tests, including a sample evaluation through the UI > > and the backend, and everything seems kosher. > > > > The repo is down to ~13 MB now. > > > > It would be a big help if everyone could go pull down the new code [1] > and > > make sure everything looks alright. I'll wait a bit before pushing any > > changes to the ASF so everyone has adequate time to test. > > > > [1] https://github.com/MJJoyce/climate > > > > > > > > > > -- Joyce > > > > > > On Wed, Mar 12, 2014 at 2:49 PM, Michael Joyce <[email protected]> wrote: > > > > > Awesome, I'm glad we got this sorted out. Thanks for all the hard work! > > > > > > > > > -- Joyce > > > > > > > > > On Wed, Mar 12, 2014 at 2:45 PM, denis.nadeau <[email protected] > > >wrote: > > > > > >> Happy to be your project guinea pig ! :-> > > >> > > >> After recompiling "git" with "libcurl" (./configure --with-curl) I was > > >> able to push the changes. You need curl to get access to https (seems > > >> like). > > >> Your git remote command was very useful. I did not have to clone the > > >> repo and copy my files over and redo the "git commands". > > >> > > >> I made quite some changes to obs4MIPs and need to "diff" and push > > changes > > >> to the repository. So it seems that, I am good to go with "git". > :-) > > >> > > >> Great work and thanks for your help! > > >> Denis > > >> > > >> > > >> On 3/12/14 5:01 PM, Michael Joyce wrote: > > >> > > >>> Ah good, we're getting close! We'd be even closer if I hadn't messed > up > > >>> in > > >>> a previous git related email! > > >>> > > >>> Our git://git.apache.org/climate.git mirror is our read only git > > mirror. > > >>> That would explain why you aren't able to write to it. > > >>> > > >>> We need to use: > > >>> https://git-wip-us.apache.org/repos/asf/climate.git > > >>> > > >>> If we didn't have a commit bit we would instead use (http vs https) > > >>> http://git-wip-us.apache.org/repos/asf/reponame.git > > >>> > > >>> I misread some documentation at [1] and [2] and confused myself. I > > >>> thought > > >>> the "WIP" or "Work in Progress" label was for migration only. Silly > me. > > >>> > > >>> We can fix this fairly easily by running > > >>> $ git remote set-url origin > > >>> https://git-wip-us.apache.org/repos/asf/climate.gi > > >>> > > >>> Then, you should see updated URLs with > > >>> $ git remote -v > > >>> > > >>> At that point you should be able to push successfully. > > >>> > > >>> Sorry that you've turned into our project guinea pig Denis! I had > hoped > > >>> to > > >>> smooth out some of these rough edges this last weekend/early this > week > > >>> but > > >>> unfortunately I haven't been able to do so. We'll get there though! > > >>> > > >>> [1] https://www.apache.org/dev/writable-git > > >>> [2] https://git-wip-us.apache.org/ > > >>> > > >>> > > >>> -- Joyce > > >>> > > >>> > > >>> On Wed, Mar 12, 2014 at 12:44 PM, denis.nadeau < > [email protected] > > >>> >wrote: > > >>> > > >>> Joyce, > > >>>> > > >>>> This is great introduction and will help other SVN/CVS developers. > > (I > > >>>> did not know you had to "git add" every changes.) > > >>>> > > >>>> Right now, I just can't push to github. I think it might be a > > >>>> configuration issue. Do you need my ssh keys or something for me to > > >>>> 'push'? > > >>>> > > >>>> git push origin master > > >>>> > > >>>> fatal: The remote end hung up unexpectedly > > >>>> > > >>>> > > >>>> git status > > >>>> > > >>>> # On branch master > > >>>> # Your branch is ahead of 'origin/master' by 4 commits. > > >>>> # > > >>>> nothing to commit (working directory clean) > > >>>> > > >>>> git remote -v > > >>>> > > >>>> origin git://git.apache.org/climate.git (fetch) > > >>>> origin git://git.apache.org/climate.git (push) > > >>>> > > >>>> Thanks for your help. (almost there...) > > >>>> Denis > > >>>> > > >>>> > > >>>> On 3/12/14 12:37 PM, Michael Joyce wrote: > > >>>> > > >>>> Ah, let me explain since git is just a bit different from SVN. > > >>>>> > > >>>>> When you commit in git you aren't actually committing to the > primary > > >>>>> server > > >>>>> like you are in SVN. You're committing to your local working copy. > In > > >>>>> order > > >>>>> to mirror those changes to the ASF you will need to run "git push". > > So > > >>>>> "git > > >>>>> status" is telling you that you've committed 4 times and those > > changes > > >>>>> aren't mirrored on the server that you ran "git clone" from. > > >>>>> > > >>>>> To be safe, you might want to checkout a clean copy of the repo > from > > >>>>> the > > >>>>> ASF (which should only take forever =) and then try again. We could > > go > > >>>>> through each of the commits and make sure they're the way you want > > >>>>> them to > > >>>>> be, but that might end up being more trouble than it's worth if we > > try > > >>>>> to > > >>>>> do it via email. This is the workflow that I would probably follow: > > >>>>> > > >>>>> # Remove the files that you don't want anymore. I'm going to say > that > > >>>>> we're > > >>>>> # sitting in the root of our repo and the files are in > > >>>>> '/obs4MIPs/examples' > > >>>>> $ git rm -r obs4MIPs/examples > > >>>>> $ git status > > >>>>> # You should now see a number of files being marked as "staged for > > >>>>> commit". > > >>>>> # Go ahead commit these removals > > >>>>> $ git commit -m "Removing obs4MIPs example .nc files" > > >>>>> > > >>>>> # Now if you run git status you shouldn't see any files listed, but > > it > > >>>>> will > > >>>>> say > > >>>>> # that you're ahead of origin/master by 1 commit > > >>>>> > > >>>>> # Now add the readme/or update any other files > > >>>>> > > >>>>> $ git add . > > >>>>> # It's important to note that "add" in git is not the same as "add" > > in > > >>>>> svn. > > >>>>> Add in git means > > >>>>> # "add/stage these changes for the next commit". If you're used to > > svn > > >>>>> this > > >>>>> can be a bit > > >>>>> # confusing. In git you need to add changes every time you want to > > >>>>> commit, > > >>>>> as opposed > > >>>>> # to svn where you only "add" the file to the repo once. > > >>>>> > > >>>>> $ git status > > >>>>> # You should see all the files that you changed present and "staged > > for > > >>>>> commit". When > > >>>>> # something is "staged for commit" that means that it will be > > committed > > >>>>> next time we > > >>>>> # run git commit. > > >>>>> $ git commit -m "Update blah blah blah" > > >>>>> > > >>>>> # Now you should see that you're ahead by a few commits depending > on > > >>>>> how > > >>>>> many times you've committed. > > >>>>> # At this point you probably want to share all your changes with > > >>>>> everyone, > > >>>>> so we'll push the changes up to the server. > > >>>>> > > >>>>> # You really can abbreviate this to just 'git push' or 'git push > > >>>>> origin'. > > >>>>> We're going to play it safe and be super explicit. > > >>>>> # This is telling git to push all the changes that you've committed > > in > > >>>>> your > > >>>>> 'master' branch > > >>>>> # (which is the default one that you've been working in) to the > > remote > > >>>>> named "origin". By default, > > >>>>> # the repo that you cloned from is named "origin". > > >>>>> $ git push origin master > > >>>>> > > >>>>> At this point we should get some emails saying you committed. > > >>>>> > > >>>>> Hopefully that helps a bit. If you have more questions let me know. > > It > > >>>>> can > > >>>>> certainly be a bit jarring of a change moving to git from svn. I'm > > >>>>> working > > >>>>> on writing up a brief "intro to git" that I will send around to the > > >>>>> mailing > > >>>>> list once it's in a useful state. It should hopefully help clear up > > >>>>> some > > >>>>> confusion for everyone. > > >>>>> > > >>>>> > > >>>>> -- Joyce > > >>>>> > > >>>>> > > >>>>> On Wed, Mar 12, 2014 at 9:04 AM, denis.nadeau < > [email protected] > > > > > >>>>> wrote: > > >>>>> > > >>>>> Joyce, > > >>>>> > > >>>>>> I did commit the change and also remove "rm" the .nc files. I did > > not > > >>>>>> see > > >>>>>> an email either. Here are the 3 commands I used > > >>>>>> > > >>>>>> 1. git add > > >>>>>> 2. git commit > > >>>>>> 3. git rm > > >>>>>> > > >>>>>> > > >>>>>> I guess "git rm" does not need a commit command. > > >>>>>> When I run "git status" I get this message. I am not sure what > > "ahead > > >>>>>> of > > >>>>>> 'origin/master' by 4 commits" means! > > >>>>>> > > >>>>>> git status > > >>>>>> # On branch master > > >>>>>> # Your branch is ahead of 'origin/master' by 4 commits. > > >>>>>> # > > >>>>>> nothing to commit (working directory clean) > > >>>>>> > > >>>>>> > > >>>>>> Denis > > >>>>>> > > >>>>>> On 3/12/14 11:19 AM, Michael Joyce wrote: > > >>>>>> > > >>>>>> Awesome Denis thanks much. I will play around with this more > soon > > >>>>>> and > > >>>>>> > > >>>>>>> see > > >>>>>>> if I can't strip out some more files. Did you push your changes > up > > to > > >>>>>>> the > > >>>>>>> repo? I didn't see a commit email come through, but I'm not > certain > > >>>>>>> my > > >>>>>>> filters are working correctly with the mailing list migrations. > > >>>>>>> > > >>>>>>> > > >>>>>>> -- Joyce > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Mar 12, 2014 at 7:20 AM, denis.nadeau < > > [email protected] > > >>>>>>> > > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>> Joyce, > > >>>>>>> > > >>>>>>> I deleted the .nc files found in my example directory for TRMM > and > > >>>>>>>> ECMWF. > > >>>>>>>> I have installed a README file and explain users how to > > >>>>>>>> retrieve the > > >>>>>>>> data > > >>>>>>>> from the original data provider. TRMM is pretty > > straightforward, > > >>>>>>>> but > > >>>>>>>> for > > >>>>>>>> ECMWF you need to register, obtain a key and download their > Python > > >>>>>>>> package. > > >>>>>>>> > > >>>>>>>> It works pretty well on my machine, let see what users say. > > >>>>>>>> > > >>>>>>>> Regards, > > >>>>>>>> Denis > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On 3/10/14 3:53 PM, Michael Joyce wrote: > > >>>>>>>> > > >>>>>>>> I think that would be great Denis! I can go ahead and look at > > >>>>>>>> doing > > >>>>>>>> > > >>>>>>>> something similar for the other ocw/ocw-ui components as well. > > I'm > > >>>>>>>>> sure > > >>>>>>>>> this will help us out a good bit. > > >>>>>>>>> > > >>>>>>>>> Thanks! > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- Joyce > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Mon, Mar 10, 2014 at 11:20 AM, denis.nadeau < > > >>>>>>>>> [email protected] > > >>>>>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Michael, > > >>>>>>>>>> > > >>>>>>>>> I like the idea of having the NetCDF files in a external > > >>>>>>>>> repository. > > >>>>>>>>> > > >>>>>>>>>> I was thinking that it might be better to point the people to > > >>>>>>>>>> satellite > > >>>>>>>>>> data at the different DAACs so that they can download the > files > > >>>>>>>>>> directly. > > >>>>>>>>>> That would work for the "obs4MIPs" program. I would feel > > >>>>>>>>>> better > > >>>>>>>>>> about > > >>>>>>>>>> it as well, I have been worried to be told by some data > > >>>>>>>>>> providers > > >>>>>>>>>> (ECMWF) > > >>>>>>>>>> that we are not authorized to distribute their original data. > > I > > >>>>>>>>>> initially > > >>>>>>>>>> did not think about this when I checked in my original code. > > >>>>>>>>>> > > >>>>>>>>>> I just found out that ECMWF now allows people to download > their > > >>>>>>>>>> data > > >>>>>>>>>> in > > >>>>>>>>>> "NetCDF" instead of "GRIB" using Python [1]. I tried it > > before, > > >>>>>>>>>> but > > >>>>>>>>>> could > > >>>>>>>>>> only retrieve GRIB data and did not want to mess with "Grads" > > ctl > > >>>>>>>>>> files > > >>>>>>>>>> and > > >>>>>>>>>> CDMS2/CDAT package. So now, I could just create a script to > > >>>>>>>>>> download > > >>>>>>>>>> the > > >>>>>>>>>> right files and rename them to the appropriate filenames for > > >>>>>>>>>> obs4MIPs > > >>>>>>>>>> examples. > > >>>>>>>>>> > > >>>>>>>>>> I would feel much better about this. Let me know what you > > think. > > >>>>>>>>>> > > >>>>>>>>>> [1] https://software.ecmwf.int/wiki/display/WEBAPI/Accessing+ > > >>>>>>>>>> ECMWF+data+servers+in+batch > > >>>>>>>>>> > > >>>>>>>>>> Denis > > >>>>>>>>>> > > >>>>>>>>>> On 3/10/14 1:06 PM, Michael Joyce wrote: > > >>>>>>>>>> > > >>>>>>>>>> Hi guys, > > >>>>>>>>>> > > >>>>>>>>>> An unfortunate side effect of our export from SVN to Git is > > that > > >>>>>>>>>> > > >>>>>>>>>>> we've > > >>>>>>>>>>> ended up with a rather bloated repository. We've had a large > > >>>>>>>>>>> number > > >>>>>>>>>>> of > > >>>>>>>>>>> binary files in our repo in the past and all of this has been > > >>>>>>>>>>> rolled > > >>>>>>>>>>> up > > >>>>>>>>>>> into a obnoxious ~500 MB pack file. I've been completely > unable > > >>>>>>>>>>> to > > >>>>>>>>>>> clone > > >>>>>>>>>>> the repo on my home internet because it constantly times out > > and > > >>>>>>>>>>> it's > > >>>>>>>>>>> painfully slow on my faster work connection. > > >>>>>>>>>>> > > >>>>>>>>>>> To fix this problem I suggest we do the following: > > >>>>>>>>>>> - Remove all binary files from our repo and host them > > externally. > > >>>>>>>>>>> For > > >>>>>>>>>>> example, NetCDF files can be downloaded when they're needed > and > > >>>>>>>>>>> cleaned > > >>>>>>>>>>> up > > >>>>>>>>>>> afterwards (for tests or examples). > > >>>>>>>>>>> - Remove all the bloat from our pack file. I was digging > > through > > >>>>>>>>>>> stuff > > >>>>>>>>>>> earlier and found a number of very large and outdated files > in > > >>>>>>>>>>> our > > >>>>>>>>>>> pack > > >>>>>>>>>>> file (~300 MB NC file, internal JPL presentations/files from > a > > >>>>>>>>>>> long > > >>>>>>>>>>> time > > >>>>>>>>>>> ago, etc.). We should be able to use [1] to help automate > this > > >>>>>>>>>>> for > > >>>>>>>>>>> us, > > >>>>>>>>>>> although we can also take care of it on our own if need be. > > >>>>>>>>>>> > > >>>>>>>>>>> Let me know what you guys think the best course of action is. > > >>>>>>>>>>> That > > >>>>>>>>>>> being > > >>>>>>>>>>> said, dealing with this sooner rather than later would be > nice > > =D > > >>>>>>>>>>> > > >>>>>>>>>>> [1] https://github.com/cmaitchison/git_diet > > >>>>>>>>>>> > > >>>>>>>>>>> -- Joyce > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> -- > > >>>>>>>>>>> > > >>>>>>>>>>> ----------------------------------------------------- > > >>>>>>>>>>> > > >>>>>>>>>> Denis Nadeau, (CSC) > > >>>>>>>>>> NCCS (NASA Center for Climate Simulation) > > >>>>>>>>>> NASA Goddard Space Flight Center > > >>>>>>>>>> Mailcode 606.2 > > >>>>>>>>>> 8800 Greenbelt Road > > >>>>>>>>>> Greenbelt, MD 20771 > > >>>>>>>>>> Email: [email protected] > > >>>>>>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 > > >>>>>>>>>> ----------------------------------------------------- > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> -- > > >>>>>>>>>> > > >>>>>>>>>> ----------------------------------------------------- > > >>>>>>>>> > > >>>>>>>> Denis Nadeau, (CSC) > > >>>>>>>> NCCS (NASA Center for Climate Simulation) > > >>>>>>>> NASA Goddard Space Flight Center > > >>>>>>>> Mailcode 606.2 > > >>>>>>>> 8800 Greenbelt Road > > >>>>>>>> Greenbelt, MD 20771 > > >>>>>>>> Email: [email protected] > > >>>>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 > > >>>>>>>> ----------------------------------------------------- > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -- > > >>>>>>>> > > >>>>>>> ----------------------------------------------------- > > >>>>>> Denis Nadeau, (CSC) > > >>>>>> NCCS (NASA Center for Climate Simulation) > > >>>>>> NASA Goddard Space Flight Center > > >>>>>> Mailcode 606.2 > > >>>>>> 8800 Greenbelt Road > > >>>>>> Greenbelt, MD 20771 > > >>>>>> Email: [email protected] > > >>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 > > >>>>>> ----------------------------------------------------- > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>> ----------------------------------------------------- > > >>>> Denis Nadeau, (CSC) > > >>>> NCCS (NASA Center for Climate Simulation) > > >>>> NASA Goddard Space Flight Center > > >>>> Mailcode 606.2 > > >>>> 8800 Greenbelt Road > > >>>> Greenbelt, MD 20771 > > >>>> Email: [email protected] > > >>>> Phone: (301) 286-7286 Fax: 301.286.1634 > > >>>> ----------------------------------------------------- > > >>>> > > >>>> > > >>>> > > >> > > >> -- > > >> ----------------------------------------------------- > > >> Denis Nadeau, (CSC) > > >> NCCS (NASA Center for Climate Simulation) > > >> NASA Goddard Space Flight Center > > >> Mailcode 606.2 > > >> 8800 Greenbelt Road > > >> Greenbelt, MD 20771 > > >> Email: [email protected] > > >> Phone: (301) 286-7286 Fax: 301.286.1634 > > >> ----------------------------------------------------- > > >> > > >> > > > > > >
