Lane Schwartz <[email protected]> writes: > I'm trying to figure out a way to define and launch jobs programmatically. > > The idea is that I am writing a computationally intensive program in > some programming language (most typically Java or Scala, but sometimes > C, C++, Ruby). I would like to be able, within my programming > language, to define some unit of computation that is then launched as > an SGE job, with the results returned to the original program. > > It appears that something like this sort of thing exists in one form > or another (e.g. hadoop, jppf) for various programming languages, but > as far as I have been able to determine, these frameworks tend to > require their own grid setup. Meaning, I can set up a hadoop or jppf > cluster, but that cluster won't interact with my SGE cluster in terms > of scheduling jobs.
There are ways of having SGE manage Hadoop instances, if that's what you want. > I am aware of DRMAA, but as far as I can tell it isn't what I am > looking for. I have successfully used DRMAA to launch (from within > Java) SGE jobs defined as external scripts. But what I'm looking for > is a mechanism for defining the job itself as Java code to be > executed. Or, if I'm writing in C++, as C++ code. I'm afraid, like Reuti, I don't understand the requirement. Is this effectively wanting mobile code, or something else? If you want to execute your source, why isn't it a question of compiling and executing as normal, maybe both in the job? In case "Java code" means byte code, I don't see why you couldn't run that directly as a job with a suitable "shell" or starter method defined. If it's a question of effectively running a managed server and sending stuff to it, that should also be possible within any constraints of the cluster it runs on. > I won't be surprised if nothing like this exists, but with all of the > work that so many people have done in grid environments over the > years, Ho, ho. > I'm hoping that something like this exists and I just haven't > heard about it. Well, we used to compile a domain-specific language (concocted graphically) on the fly and execute it remotely on a grid -- except that grids had been invented then -- but I suspect that's not what this is about. (It was a bit less direct doing a similar thing on the parallel blade system In The Cloud that also hadn't been invented...) -- Community Grid Engine: http://arc.liv.ac.uk/SGE/ _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
