Hi Derek, That's great. I guess the best place I can refer for you to look is this [3]. That's how we keep track of the configurations of the job types. If you follow the trail, you should be able to implement a new HTCondorConfiguration implementation. Then we can write a groovy template like this [4] and attach to the configuration.
If you haven't already, you can set up Airavata in an IDE for development purposes. Please follow [5]. [3] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L107 [4] https://github.com/apache/airavata/blob/master/modules/configuration/server/src/main/resources/SLURM_Groovy.template [5] https://github.com/apache/airavata/tree/master/modules/ide-integration Thanks Dimuthu On Wed, Sep 16, 2020 at 3:26 PM Derek Weitzel <[email protected]> wrote: > Hi, > > For HTCondor, it is very similar submission process as SLURM. First, > create the submission file describing the job. Then, submit that job using > the command “condor_submit <job_file>”. It returns success or failure, as > well as the job id (though, you will have to use some string parsing to > extract the job id) > > “condor_q <jobid>” is used to check the status. You can craft the command > to only get a job status number. For example: > $ condor_q <jobid> -format ‘%i’ ‘JobStatus’ > (or something like this) JobStatus numbers have very good definitions. > > When the job is done, condor_q <jobid> will not return anything (it has > left the queue). You can get completed job statistics with condor_history > <jobid> -l > > > > - Derek > > > > > On Sep 16, 2020, at 12:55 PM, Daniel Shchur < > [email protected]> wrote: > > > > Hi Dimuthu, > > > > I haven’t had a lot of experience with Condor. One of my colleagues, > Derek, has more experience with this so I’ll have to ask him to help me a > little. The way he explained it however is that there are equivalent > commands to all of the SLURM commands. The submission script isn’t too > different from what SLURM uses as well. This link should provide some > insight: > https://htcondor.readthedocs.io/en/latest/users-manual/submitting-a-job.html > > > > I’ll write up a step by step shortly. > > > > - Daniel Shchur > > > > From: DImuthu Upeksha > > Sent: Wednesday, September 16, 2020 1:45 PM > > To: Daniel Shchur > > Cc: Airavata Users > > Subject: Re: Submitting jobs on Open Science Grid > > > > Hi Daniel, > > > > Good thing about using a Groovy script and using cli to submit jobs is > that you don't have to re-invent anything. You can simply follow what we > have done for other schedulers and replicate them. I would also favor that > approach than the python one. I just wasn't aware that HTCondor has that > capability (CLI). Can you please summarize the steps (commands) that you > normally follow to submit a job to HTCondor through the CLI? > > > > Thanks > > Dimuthu > > > > On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur < > [email protected]> wrote: > > Hi Dimuthu, > > > > Something we have been discussing is going the route of SLURM and > implementing a Groovy template to submit jobs via CLI, as well as ping > Condor to get status. Would it be preferable to try and go through python > bindings instead to call the API directly? > > > > - Daniel Shchur > > > > From: DImuthu Upeksha > > Sent: Wednesday, September 16, 2020 1:30 PM > > To: Airavata Users > > Cc: Daniel Shchur > > Subject: Re: Submitting jobs on Open Science Grid > > > > Hi Daniel, > > > > If you follow the paper that Suresh has sent, you might notice that > there is a Job submission task in the pre workflow manager. You have to > implement a job submission task for HTCondor like we do for standard job > schedulers like slurm and pbs. Have a look at [1] that implements a ssh > based submission to schedulers. However, as I have heard, there is a python > client for HTCondor submissions. So you don't need to worry about all these > SSH based communications. You can use any of these [2] ways to talk to the > python client through java. > > > > Implementing a job management extension for Airavata includes multiple > steps so I would suggest you to work on that step by step. As the first > step, try to come up with a sample java code that accepts all the > parameters and submits a HTCondor job through the python client. Once you > have a working code, I'll help you on how to generalize it and migrate it > to Airavata. > > > > [1] > https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java > > [2] https://www.baeldung.com/java-working-with-python > > > > Thanks > > Dimuthu > > > > On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <[email protected]> wrote: > > Hi Daniel, > > > > As we discussed offline, it will be good to get a contribution from you > on the HTCondor extension. Please refer to this attached paper to get an > idea about the job management framework. > > > > Suresh > > > > > > > > > > On Sep 8, 2020, at 5:59 PM, Daniel Shchur <[email protected]> > wrote: > > > > Hi all, > > > > I’m Daniel Shchur and working on a programming team at the University of > Nebraska—Lincoln. We are working on creating a science gateway for a > specific application that would allow users to run the application on UNL’s > Holland Computing Center super computer as well as free resources on the > Open Science Grid. > > > > We were wondering if it is at all possible to submit jobs to different > job management workflows such as HTCondor, the workflow used by Open > Science Grid. We were not able to find this in the documentation. Any help > with this issue would be greatly appreciated. > > > > Thank you! > > > > - Daniel Shchur > >
