Update on this. I've tested this on a fork to make sure that it works properly. Everything seems fine. I've taken the old repository and removed all the old large binaries and leftover JPL related things. I've gone through and run some tests, including a sample evaluation through the UI and the backend, and everything seems kosher.
The repo is down to ~13 MB now. It would be a big help if everyone could go pull down the new code [1] and make sure everything looks alright. I'll wait a bit before pushing any changes to the ASF so everyone has adequate time to test. [1] https://github.com/MJJoyce/climate -- Joyce On Wed, Mar 12, 2014 at 2:49 PM, Michael Joyce <[email protected]> wrote: > Awesome, I'm glad we got this sorted out. Thanks for all the hard work! > > > -- Joyce > > > On Wed, Mar 12, 2014 at 2:45 PM, denis.nadeau <[email protected]>wrote: > >> Happy to be your project guinea pig ! :-> >> >> After recompiling "git" with "libcurl" (./configure --with-curl) I was >> able to push the changes. You need curl to get access to https (seems >> like). >> Your git remote command was very useful. I did not have to clone the >> repo and copy my files over and redo the "git commands". >> >> I made quite some changes to obs4MIPs and need to "diff" and push changes >> to the repository. So it seems that, I am good to go with "git". :-) >> >> Great work and thanks for your help! >> Denis >> >> >> On 3/12/14 5:01 PM, Michael Joyce wrote: >> >>> Ah good, we're getting close! We'd be even closer if I hadn't messed up >>> in >>> a previous git related email! >>> >>> Our git://git.apache.org/climate.git mirror is our read only git mirror. >>> That would explain why you aren't able to write to it. >>> >>> We need to use: >>> https://git-wip-us.apache.org/repos/asf/climate.git >>> >>> If we didn't have a commit bit we would instead use (http vs https) >>> http://git-wip-us.apache.org/repos/asf/reponame.git >>> >>> I misread some documentation at [1] and [2] and confused myself. I >>> thought >>> the "WIP" or "Work in Progress" label was for migration only. Silly me. >>> >>> We can fix this fairly easily by running >>> $ git remote set-url origin >>> https://git-wip-us.apache.org/repos/asf/climate.gi >>> >>> Then, you should see updated URLs with >>> $ git remote -v >>> >>> At that point you should be able to push successfully. >>> >>> Sorry that you've turned into our project guinea pig Denis! I had hoped >>> to >>> smooth out some of these rough edges this last weekend/early this week >>> but >>> unfortunately I haven't been able to do so. We'll get there though! >>> >>> [1] https://www.apache.org/dev/writable-git >>> [2] https://git-wip-us.apache.org/ >>> >>> >>> -- Joyce >>> >>> >>> On Wed, Mar 12, 2014 at 12:44 PM, denis.nadeau <[email protected] >>> >wrote: >>> >>> Joyce, >>>> >>>> This is great introduction and will help other SVN/CVS developers. (I >>>> did not know you had to "git add" every changes.) >>>> >>>> Right now, I just can't push to github. I think it might be a >>>> configuration issue. Do you need my ssh keys or something for me to >>>> 'push'? >>>> >>>> git push origin master >>>> >>>> fatal: The remote end hung up unexpectedly >>>> >>>> >>>> git status >>>> >>>> # On branch master >>>> # Your branch is ahead of 'origin/master' by 4 commits. >>>> # >>>> nothing to commit (working directory clean) >>>> >>>> git remote -v >>>> >>>> origin git://git.apache.org/climate.git (fetch) >>>> origin git://git.apache.org/climate.git (push) >>>> >>>> Thanks for your help. (almost there...) >>>> Denis >>>> >>>> >>>> On 3/12/14 12:37 PM, Michael Joyce wrote: >>>> >>>> Ah, let me explain since git is just a bit different from SVN. >>>>> >>>>> When you commit in git you aren't actually committing to the primary >>>>> server >>>>> like you are in SVN. You're committing to your local working copy. In >>>>> order >>>>> to mirror those changes to the ASF you will need to run "git push". So >>>>> "git >>>>> status" is telling you that you've committed 4 times and those changes >>>>> aren't mirrored on the server that you ran "git clone" from. >>>>> >>>>> To be safe, you might want to checkout a clean copy of the repo from >>>>> the >>>>> ASF (which should only take forever =) and then try again. We could go >>>>> through each of the commits and make sure they're the way you want >>>>> them to >>>>> be, but that might end up being more trouble than it's worth if we try >>>>> to >>>>> do it via email. This is the workflow that I would probably follow: >>>>> >>>>> # Remove the files that you don't want anymore. I'm going to say that >>>>> we're >>>>> # sitting in the root of our repo and the files are in >>>>> '/obs4MIPs/examples' >>>>> $ git rm -r obs4MIPs/examples >>>>> $ git status >>>>> # You should now see a number of files being marked as "staged for >>>>> commit". >>>>> # Go ahead commit these removals >>>>> $ git commit -m "Removing obs4MIPs example .nc files" >>>>> >>>>> # Now if you run git status you shouldn't see any files listed, but it >>>>> will >>>>> say >>>>> # that you're ahead of origin/master by 1 commit >>>>> >>>>> # Now add the readme/or update any other files >>>>> >>>>> $ git add . >>>>> # It's important to note that "add" in git is not the same as "add" in >>>>> svn. >>>>> Add in git means >>>>> # "add/stage these changes for the next commit". If you're used to svn >>>>> this >>>>> can be a bit >>>>> # confusing. In git you need to add changes every time you want to >>>>> commit, >>>>> as opposed >>>>> # to svn where you only "add" the file to the repo once. >>>>> >>>>> $ git status >>>>> # You should see all the files that you changed present and "staged for >>>>> commit". When >>>>> # something is "staged for commit" that means that it will be committed >>>>> next time we >>>>> # run git commit. >>>>> $ git commit -m "Update blah blah blah" >>>>> >>>>> # Now you should see that you're ahead by a few commits depending on >>>>> how >>>>> many times you've committed. >>>>> # At this point you probably want to share all your changes with >>>>> everyone, >>>>> so we'll push the changes up to the server. >>>>> >>>>> # You really can abbreviate this to just 'git push' or 'git push >>>>> origin'. >>>>> We're going to play it safe and be super explicit. >>>>> # This is telling git to push all the changes that you've committed in >>>>> your >>>>> 'master' branch >>>>> # (which is the default one that you've been working in) to the remote >>>>> named "origin". By default, >>>>> # the repo that you cloned from is named "origin". >>>>> $ git push origin master >>>>> >>>>> At this point we should get some emails saying you committed. >>>>> >>>>> Hopefully that helps a bit. If you have more questions let me know. It >>>>> can >>>>> certainly be a bit jarring of a change moving to git from svn. I'm >>>>> working >>>>> on writing up a brief "intro to git" that I will send around to the >>>>> mailing >>>>> list once it's in a useful state. It should hopefully help clear up >>>>> some >>>>> confusion for everyone. >>>>> >>>>> >>>>> -- Joyce >>>>> >>>>> >>>>> On Wed, Mar 12, 2014 at 9:04 AM, denis.nadeau <[email protected]> >>>>> wrote: >>>>> >>>>> Joyce, >>>>> >>>>>> I did commit the change and also remove "rm" the .nc files. I did not >>>>>> see >>>>>> an email either. Here are the 3 commands I used >>>>>> >>>>>> 1. git add >>>>>> 2. git commit >>>>>> 3. git rm >>>>>> >>>>>> >>>>>> I guess "git rm" does not need a commit command. >>>>>> When I run "git status" I get this message. I am not sure what "ahead >>>>>> of >>>>>> 'origin/master' by 4 commits" means! >>>>>> >>>>>> git status >>>>>> # On branch master >>>>>> # Your branch is ahead of 'origin/master' by 4 commits. >>>>>> # >>>>>> nothing to commit (working directory clean) >>>>>> >>>>>> >>>>>> Denis >>>>>> >>>>>> On 3/12/14 11:19 AM, Michael Joyce wrote: >>>>>> >>>>>> Awesome Denis thanks much. I will play around with this more soon >>>>>> and >>>>>> >>>>>>> see >>>>>>> if I can't strip out some more files. Did you push your changes up to >>>>>>> the >>>>>>> repo? I didn't see a commit email come through, but I'm not certain >>>>>>> my >>>>>>> filters are working correctly with the mailing list migrations. >>>>>>> >>>>>>> >>>>>>> -- Joyce >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 12, 2014 at 7:20 AM, denis.nadeau <[email protected] >>>>>>> > >>>>>>> wrote: >>>>>>> >>>>>>> Joyce, >>>>>>> >>>>>>> I deleted the .nc files found in my example directory for TRMM and >>>>>>>> ECMWF. >>>>>>>> I have installed a README file and explain users how to >>>>>>>> retrieve the >>>>>>>> data >>>>>>>> from the original data provider. TRMM is pretty straightforward, >>>>>>>> but >>>>>>>> for >>>>>>>> ECMWF you need to register, obtain a key and download their Python >>>>>>>> package. >>>>>>>> >>>>>>>> It works pretty well on my machine, let see what users say. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Denis >>>>>>>> >>>>>>>> >>>>>>>> On 3/10/14 3:53 PM, Michael Joyce wrote: >>>>>>>> >>>>>>>> I think that would be great Denis! I can go ahead and look at >>>>>>>> doing >>>>>>>> >>>>>>>> something similar for the other ocw/ocw-ui components as well. I'm >>>>>>>>> sure >>>>>>>>> this will help us out a good bit. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> >>>>>>>>> -- Joyce >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 10, 2014 at 11:20 AM, denis.nadeau < >>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Michael, >>>>>>>>>> >>>>>>>>> I like the idea of having the NetCDF files in a external >>>>>>>>> repository. >>>>>>>>> >>>>>>>>>> I was thinking that it might be better to point the people to >>>>>>>>>> satellite >>>>>>>>>> data at the different DAACs so that they can download the files >>>>>>>>>> directly. >>>>>>>>>> That would work for the "obs4MIPs" program. I would feel >>>>>>>>>> better >>>>>>>>>> about >>>>>>>>>> it as well, I have been worried to be told by some data >>>>>>>>>> providers >>>>>>>>>> (ECMWF) >>>>>>>>>> that we are not authorized to distribute their original data. I >>>>>>>>>> initially >>>>>>>>>> did not think about this when I checked in my original code. >>>>>>>>>> >>>>>>>>>> I just found out that ECMWF now allows people to download their >>>>>>>>>> data >>>>>>>>>> in >>>>>>>>>> "NetCDF" instead of "GRIB" using Python [1]. I tried it before, >>>>>>>>>> but >>>>>>>>>> could >>>>>>>>>> only retrieve GRIB data and did not want to mess with "Grads" ctl >>>>>>>>>> files >>>>>>>>>> and >>>>>>>>>> CDMS2/CDAT package. So now, I could just create a script to >>>>>>>>>> download >>>>>>>>>> the >>>>>>>>>> right files and rename them to the appropriate filenames for >>>>>>>>>> obs4MIPs >>>>>>>>>> examples. >>>>>>>>>> >>>>>>>>>> I would feel much better about this. Let me know what you think. >>>>>>>>>> >>>>>>>>>> [1] https://software.ecmwf.int/wiki/display/WEBAPI/Accessing+ >>>>>>>>>> ECMWF+data+servers+in+batch >>>>>>>>>> >>>>>>>>>> Denis >>>>>>>>>> >>>>>>>>>> On 3/10/14 1:06 PM, Michael Joyce wrote: >>>>>>>>>> >>>>>>>>>> Hi guys, >>>>>>>>>> >>>>>>>>>> An unfortunate side effect of our export from SVN to Git is that >>>>>>>>>> >>>>>>>>>>> we've >>>>>>>>>>> ended up with a rather bloated repository. We've had a large >>>>>>>>>>> number >>>>>>>>>>> of >>>>>>>>>>> binary files in our repo in the past and all of this has been >>>>>>>>>>> rolled >>>>>>>>>>> up >>>>>>>>>>> into a obnoxious ~500 MB pack file. I've been completely unable >>>>>>>>>>> to >>>>>>>>>>> clone >>>>>>>>>>> the repo on my home internet because it constantly times out and >>>>>>>>>>> it's >>>>>>>>>>> painfully slow on my faster work connection. >>>>>>>>>>> >>>>>>>>>>> To fix this problem I suggest we do the following: >>>>>>>>>>> - Remove all binary files from our repo and host them externally. >>>>>>>>>>> For >>>>>>>>>>> example, NetCDF files can be downloaded when they're needed and >>>>>>>>>>> cleaned >>>>>>>>>>> up >>>>>>>>>>> afterwards (for tests or examples). >>>>>>>>>>> - Remove all the bloat from our pack file. I was digging through >>>>>>>>>>> stuff >>>>>>>>>>> earlier and found a number of very large and outdated files in >>>>>>>>>>> our >>>>>>>>>>> pack >>>>>>>>>>> file (~300 MB NC file, internal JPL presentations/files from a >>>>>>>>>>> long >>>>>>>>>>> time >>>>>>>>>>> ago, etc.). We should be able to use [1] to help automate this >>>>>>>>>>> for >>>>>>>>>>> us, >>>>>>>>>>> although we can also take care of it on our own if need be. >>>>>>>>>>> >>>>>>>>>>> Let me know what you guys think the best course of action is. >>>>>>>>>>> That >>>>>>>>>>> being >>>>>>>>>>> said, dealing with this sooner rather than later would be nice =D >>>>>>>>>>> >>>>>>>>>>> [1] https://github.com/cmaitchison/git_diet >>>>>>>>>>> >>>>>>>>>>> -- Joyce >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> ----------------------------------------------------- >>>>>>>>>>> >>>>>>>>>> Denis Nadeau, (CSC) >>>>>>>>>> NCCS (NASA Center for Climate Simulation) >>>>>>>>>> NASA Goddard Space Flight Center >>>>>>>>>> Mailcode 606.2 >>>>>>>>>> 8800 Greenbelt Road >>>>>>>>>> Greenbelt, MD 20771 >>>>>>>>>> Email: [email protected] >>>>>>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 >>>>>>>>>> ----------------------------------------------------- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> ----------------------------------------------------- >>>>>>>>> >>>>>>>> Denis Nadeau, (CSC) >>>>>>>> NCCS (NASA Center for Climate Simulation) >>>>>>>> NASA Goddard Space Flight Center >>>>>>>> Mailcode 606.2 >>>>>>>> 8800 Greenbelt Road >>>>>>>> Greenbelt, MD 20771 >>>>>>>> Email: [email protected] >>>>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 >>>>>>>> ----------------------------------------------------- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>> ----------------------------------------------------- >>>>>> Denis Nadeau, (CSC) >>>>>> NCCS (NASA Center for Climate Simulation) >>>>>> NASA Goddard Space Flight Center >>>>>> Mailcode 606.2 >>>>>> 8800 Greenbelt Road >>>>>> Greenbelt, MD 20771 >>>>>> Email: [email protected] >>>>>> Phone: (301) 286-7286 Fax: 301.286.1634 >>>>>> ----------------------------------------------------- >>>>>> >>>>>> >>>>>> >>>>>> -- >>>> ----------------------------------------------------- >>>> Denis Nadeau, (CSC) >>>> NCCS (NASA Center for Climate Simulation) >>>> NASA Goddard Space Flight Center >>>> Mailcode 606.2 >>>> 8800 Greenbelt Road >>>> Greenbelt, MD 20771 >>>> Email: [email protected] >>>> Phone: (301) 286-7286 Fax: 301.286.1634 >>>> ----------------------------------------------------- >>>> >>>> >>>> >> >> -- >> ----------------------------------------------------- >> Denis Nadeau, (CSC) >> NCCS (NASA Center for Climate Simulation) >> NASA Goddard Space Flight Center >> Mailcode 606.2 >> 8800 Greenbelt Road >> Greenbelt, MD 20771 >> Email: [email protected] >> Phone: (301) 286-7286 Fax: 301.286.1634 >> ----------------------------------------------------- >> >> >
