Gump Actions (was re: brutus config -> lsd config)
Sam wrote: > It appears to me that you are looking at Gump as as monolithic tool. I don't think I do, other than I believe that 90% of the community are more interested in Gump the service, than Gump the code -- so I focus on that. I think (for us/Gumpmeisters and for the eventual growing millions ... ;-) we ensure we can run things in whatever parts we want, to optimize our time. I don't think we differ here. We do have one monolith script (integrate.pt for the 'server' = cron run) but we also have individual script (update/build/debug) and the GUI (I almost got it working again), all of which I think ought be able to do all or any small parts they wish. For the record, I really like that we document to xdocs -- external files -- 'cos if forrest fails to publish I can go in and tweak and re-run. I like that we can do updates as we see fit, builds as we see fit, tight and separate -- I just don't do them much myself, I don't have the network resources. The only problem I have are the in-memory structures that represent the run information [options/list], state, annotations, etc. I've thought about serializing that to XML, but not had the time/energy. I also fear it is overkill. > I see it as a series of actions: Generate, Update, Make, Publish... I agree. Have you looked in engine.py? I've tried to break things into individual named tasks with there being only the minimal mandatory dependencies (i.e. code would crash without). There is no dependency upon 'update' for 'build', not on either of these two for documentation. Again, the in memory tree is the main common aspect between these tasks. As such, I try to use the 'GumpRun' as the common interfacing information between aspects. BTW: Generate no longer exists, we "generate into memory". That said, I could value that as "generate to a merge file (we have that) and re-read" -- the only reason I didn't do that is that I don't see much time saving from loading/merging on demand. I've thought about it. > My suggestion is that we should decompose Gump. There is no reason that > everything needs to funnel through a single entry point. I think it is decomposed, with 'engine' being my attempt at extracting logic from model classes, so we don't have it duplicated. I'm no expert designer, I'd welcome feedback/improvements. > Focusing initially on the cron job, it needs to run a script. That > script can be written entirely in Python (including reading the > configuration, setting environment variables, etc). It can do more than > what one typically does from the command line (e.g., copying of > directories, nagging, etc). That is gumpy.py. > > Key points: > > 1) when it does things typically done by "testing", it calls into the > exact same code. > I've tried in the unit tests. [You, tinkered with them yet?] I find that when I crash (in public) it is in any code I've not covered in tests (the joys of a large runtime interpretted language w/ huge metadata inputs). As such I'm striving to get as much coverage in the tests as possible. Engine.py is one weak spot, I just don't want to get too fancy trying to make that unit testable (Against some mythincall in-memory workspace/filesystem/etc.). BTW: See the test workspace in unit tests? That allows acces to the file system, and such. I often run gumpy.py against that test workspace to get better test coverage. BTW: I want a 'check in script that does: 1) Run all unit test 2) Run the test workspace 3) Do a pychecker run on all code I tried writting it, but struggled with the pychecker part and gave up. If I had the discipline to do that I know it'd help reduce my error rate... > 2) official, completed outputs are served from a different URL than > incomplete or testing outputs. Yeah, I hear you. Right now I've been doing --text for testing when I don't want to dork the site, or I just dork it and be damned. You'll find there is some complexity in what gets copied to log (due to forrest) but feel free to dig in. Again (from TODO list) a historical RDBMS is a nice thought for saving some offical result history. > 3) official runs start by cleaning up the work area. This was done by > rsync in "classic" gump. We moved away from rsync in part because of (perceived) bugs [which in hindsight maybe weren't] and portability (and we didn't need the 'r'). The folks on M$ were struggling with it (see archives). Basically Antoine gave us a Python port of is (see utils/sync.py). We use this to do as you say, and it works well. Since we own the code, we could even extend it to report on 'what files changed for this build' (some folks would like that). Right now when we update we sync, which you might wish to break that apart. regards, Adam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: brutus config -> lsd config
Adam R. B. Jack wrote: I think we need to ensure we have a 'test' flavour [or test workspace perhaps] on every box, one that we expect folks to tinker with (when the main build isn't running). A test workspace would work [other than the fact it could mask 'updated'] because we'd only run it at odd times, and we'd not crap on the same output tree (like I often do testing.) Can I suggest a different approach? What I used to do with classic gump was to capture and datestamp the outputs of the "official" runs, keeping a fixed number live. This is a simple matter of copying a few directories at the end of the run. I've tinkered with this on and off, but not had a separate flag for 'official'. I think it needs to be introduced. (There are mutiple places where we check isFull() [i.e. 'all' projects not some] -- and we have the integrate script different (in what it does) than build or update or debug. There are even comments in the code about 'if official then nag and/or do rss/atom', but much has been removed since I like to test the whole path (quickly) on subsets of a profile.) This brings me onto configuration, see next... It appears to me that you are looking at Gump as as monolithic tool. I see it as a series of actions: Generate, Update, Make, Publish... My suggestion is that we should decompose Gump. There is no reason that everything needs to funnel through a single entry point. There are a number of discrete actions that users may wish to trigger. These actions may be triggered from the command line, a gui, a webapp, or a cron job. Focusing initially on the cron job, it needs to run a script. That script can be written entirely in Python (including reading the configuration, setting environment variables, etc). It can do more than what one typically does from the command line (e.g., copying of directories, nagging, etc). Key points: 1) when it does things typically done by "testing", it calls into the exact same code. 2) official, completed outputs are served from a different URL than incomplete or testing outputs. 3) official runs start by cleaning up the work area. This was done by rsync in "classic" gump. - Sam Ruby - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: brutus config -> lsd config
> >>Are we happy with the gump config on brutus? Is that setup reasonably > >>stable? > > > > Unless we decide to move to tomcat/forrest, I'd say so. > > How should that change things? What's the next step? Brutus is a good master to clone, unless tomcat/forrest add more dependencies, that was all. We just haven't tried it yet, and I was just trying to be exact. > Apparently running a forrest server on a different port is as easy as a > executing a command. So I would think that the next step would be to > get gump to produce xdocs during the run in addition to the static > documents at the end of the run. Once this was done, forrest can be > used to dynamically produce content from these xdocs. Does this sound > about right? So forrest bundles tomcat or similar? Huh. I guess I did see that with the 'forrest run' command, I just filtered it out, thinking we'd not want users (like gump) doing such stuff in a multi-user environment. Translation -- I was waiting for 'admin' to install something. :-) > > If there is anything I am not sure I like about the set-up is that we > > mirrored the /usr/local/gump from Moof, and I'm not sure if that directory > > is the best root one on all. I like things like /var/gump and > > /opt/gump/packages and things like that, and (maybe) /home/gump. But, so > > long as we can find choices (or a choice) that suits norms on these > > platforms, I don't really care. > > I already did a few symbolic links from /home/gump to strategic points > in /usr/local/gump; we certainly can do a few more from /var and /opt. Yup, I saw. I'm not stuck on these, I am just trying to raise the awareness that this was more an OSX decision, not a Gump community one. Packages in /opt seem good 'cos they are static. Workspaces in /var seem good, etc. I really don't know if there are standards or conventions on the various flavours and OSs, but if there are, I'd like to follow them. > > Oh yes, and I'm still not sure about .bash_profile including one flavours' > > local-env-py.sh. I didn't take Sam up on his posting (I forget when/werre) > > that said, don't expect these settings in cronjob. As such, we are still > > using gumpy (that works, but is ugly since it needs three more env variables > > & doesn't read the workspace to get the values) not gumpy.py. > > Why isn't gumpy simply: > >. local-env-py.sh >python gump.py > > ? Err, good question. ;-) History? I wrote gump.sh (for traditional), then gumpy.sh, then (sometimes I'm slow to see things) it dawned on me a gumpy.py would be a nice idea (to avoid the need for .bat, and more folks were testing on M$). I started work on gumpy.py being careful not to use gump.* packages so we could wipe *.pyc [not done in gumpy.py yet, see archives for TODOs] and also CVS update Gump. I added some nice features, like reading the workspace file (using minidom) to extract certain variables -- even mailing failures. I made great progress, and it sorta works, I just never cracked the environment issues (and hup also caused me some grief). Daft, but I got 'stuck' on thinking that if I were shooting for a *.py I ought not have a *.sh. > For that matter, the cron command can look something like this (omitting > paths): > >bash -c "(. local-env-py.sh; python gump.py)" Don't ya love how one persons block is another persons simplicity? Sure, when not. Better a one liner *.sh using gumpy.py than not using gumpy.py... Let me take one last look into gumpy.sh and gumpy.py and we do exactly what you just said. > > I think we need to ensure we have a 'test' flavour [or test workspace > > perhaps] on every box, one that we expect folks to tinker with (when the > > main build isn't running). A test workspace would work [other than the fact > > it could mask 'updated'] because we'd only run it at odd times, and we'd not > > crap on the same output tree (like I often do testing.) > > Can I suggest a different approach? What I used to do with classic gump > was to capture and datestamp the outputs of the "official" runs, keeping > a fixed number live. This is a simple matter of copying a few > directories at the end of the run. I've tinkered with this on and off, but not had a separate flag for 'official'. I think it needs to be introduced. (There are mutiple places where we check isFull() [i.e. 'all' projects not some] -- and we have the integrate script different (in what it does) than build or update or debug. There are even comments in the code about 'if official then nag and/or do rss/atom', but much has been removed since I like to test the whole path (quickly) on subsets of a profile.) This brings me onto configuration, see next... regards Adam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: brutus config -> lsd config
Adam R. B. Jack wrote: Are we happy with the gump config on brutus? Is that setup reasonably stable? Unless we decide to move to tomcat/forrest, I'd say so. How should that change things? What's the next step? Apparently running a forrest server on a different port is as easy as a executing a command. So I would think that the next step would be to get gump to produce xdocs during the run in addition to the static documents at the end of the run. Once this was done, forrest can be used to dynamically produce content from these xdocs. Does this sound about right? If there is anything I am not sure I like about the set-up is that we mirrored the /usr/local/gump from Moof, and I'm not sure if that directory is the best root one on all. I like things like /var/gump and /opt/gump/packages and things like that, and (maybe) /home/gump. But, so long as we can find choices (or a choice) that suits norms on these platforms, I don't really care. I already did a few symbolic links from /home/gump to strategic points in /usr/local/gump; we certainly can do a few more from /var and /opt. Oh yes, and I'm still not sure about .bash_profile including one flavours' local-env-py.sh. I didn't take Sam up on his posting (I forget when/werre) that said, don't expect these settings in cronjob. As such, we are still using gumpy (that works, but is ugly since it needs three more env variables & doesn't read the workspace to get the values) not gumpy.py. Why isn't gumpy simply: . local-env-py.sh python gump.py ? For that matter, the cron command can look something like this (omitting paths): bash -c "(. local-env-py.sh; python gump.py)" I think we need to ensure we have a 'test' flavour [or test workspace perhaps] on every box, one that we expect folks to tinker with (when the main build isn't running). A test workspace would work [other than the fact it could mask 'updated'] because we'd only run it at odd times, and we'd not crap on the same output tree (like I often do testing.) Can I suggest a different approach? What I used to do with classic gump was to capture and datestamp the outputs of the "official" runs, keeping a fixed number live. This is a simple matter of copying a few directories at the end of the run. - Sam Ruby - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: brutus config -> lsd config
Leo Simons wrote: Are we happy with the gump config on brutus? Is that setup reasonably stable? I'd like to mirror on lsd to the maximum extent possible. Since brutus is debian and lsd is fedora, the mirroring won't be complete, but we should get pretty far. Not that it is particularly hard, but if you give me an account, I would be willing to help set things up. The differences between debian and fedora primarily involve how you install of apache, cvs, python, and subversion. Oh, and java can be installed as an RPM. - Sam Ruby - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: brutus config -> lsd config
> Are we happy with the gump config on brutus? Is that setup reasonably > stable? Unless we decide to move to tomcat/forrest, I'd say so. If there is anything I am not sure I like about the set-up is that we mirrored the /usr/local/gump from Moof, and I'm not sure if that directory is the best root one on all. I like things like /var/gump and /opt/gump/packages and things like that, and (maybe) /home/gump. But, so long as we can find choices (or a choice) that suits norms on these platforms, I don't really care. I think the ideas of shared packages, and shared forrest/maven/whatever are good. Not sure they are installed in the best place, but they work. The idea of multiple installs is nice. In a perfect world we might want one install, multiple workspaces, but we aren't there yet. Oh yes, and I'm still not sure about .bash_profile including one flavours' local-env-py.sh. I didn't take Sam up on his posting (I forget when/werre) that said, don't expect these settings in cronjob. As such, we are still using gumpy (that works, but is ugly since it needs three more env variables & doesn't read the workspace to get the values) not gumpy.py. I think we need to ensure we have a 'test' flavour [or test workspace perhaps] on every box, one that we expect folks to tinker with (when the main build isn't running). A test workspace would work [other than the fact it could mask 'updated'] because we'd only run it at odd times, and we'd not crap on the same output tree (like I often do testing.) In short, we have improvements we can make, but Brutus (the breakthrough was using a shared 'gump') is a good un. > I'd like to mirror on lsd to the maximum extent possible. Since brutus > is debian and lsd is fedora, the mirroring won't be complete, but we > should get pretty far. Great idea. Perhaps re-install from scratch (save packages) into a gump account. regards, Adam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
brutus config -> lsd config
Are we happy with the gump config on brutus? Is that setup reasonably stable? I'd like to mirror on lsd to the maximum extent possible. Since brutus is debian and lsd is fedora, the mirroring won't be complete, but we should get pretty far. -- cheers, - Leo Simons --- Weblog -- http://leosimons.com/ IoC Component Glue -- http://jicarilla.org/ Articles & Opinions -- http://articles.leosimons.com/ --- "We started off trying to set up a small anarchist community, but people wouldn't obey the rules." -- Alan Bennett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
