Hi Daniel, Good thing about using a Groovy script and using cli to submit jobs is that you don't have to re-invent anything. You can simply follow what we have done for other schedulers and replicate them. I would also favor that approach than the python one. I just wasn't aware that HTCondor has that capability (CLI). Can you please summarize the steps (commands) that you normally follow to submit a job to HTCondor through the CLI?
Thanks Dimuthu On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur <[email protected]> wrote: > Hi Dimuthu, > > > > Something we have been discussing is going the route of SLURM and > implementing a Groovy template to submit jobs via CLI, as well as ping > Condor to get status. Would it be preferable to try and go through python > bindings instead to call the API directly? > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <[email protected]> > *Sent: *Wednesday, September 16, 2020 1:30 PM > *To: *Airavata Users <[email protected]> > *Cc: *Daniel Shchur <[email protected]> > *Subject: *Re: Submitting jobs on Open Science Grid > > > > Hi Daniel, > > > > If you follow the paper that Suresh has sent, you might notice that there > is a Job submission task in the pre workflow manager. You have to implement > a job submission task for HTCondor like we do for standard job schedulers > like slurm and pbs. Have a look at [1] that implements a ssh based > submission to schedulers. However, as I have heard, there is a python > client for HTCondor submissions. So you don't need to worry about all these > SSH based communications. You can use any of these [2] ways to talk to the > python client through java. > > > > Implementing a job management extension for Airavata includes multiple > steps so I would suggest you to work on that step by step. As the first > step, try to come up with a sample java code that accepts all the > parameters and submits a HTCondor job through the python client. Once you > have a working code, I'll help you on how to generalize it and migrate it > to Airavata. > > > > [1] > https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=zN0maF1ewq6hAC6VMBZKPhZWRuwXE70h5w7A5NASKTc&e=> > > [2] https://www.baeldung.com/java-working-with-python > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=tJVmrIlkB6rca7GLYAle5AVZcIJY6Z6SkbajdJMEVgQ&e=> > > > > Thanks > > Dimuthu > > > > On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <[email protected]> wrote: > > Hi Daniel, > > > > As we discussed offline, it will be good to get a contribution from you on > the HTCondor extension. Please refer to this attached paper to get an idea > about the job management framework. > > > > Suresh > > > > > > > > On Sep 8, 2020, at 5:59 PM, Daniel Shchur <[email protected]> > wrote: > > > > Hi all, > > > > I’m Daniel Shchur and working on a programming team at the University of > Nebraska—Lincoln. We are working on creating a science gateway for a > specific application that would allow users to run the application on UNL’s > Holland Computing Center super computer as well as free resources on the > Open Science Grid. > > We were wondering if it is at all possible to submit jobs to different job > management workflows such as HTCondor, the workflow used by Open Science > Grid. We were not able to find this in the documentation. Any help with > this issue would be greatly appreciated. > > Thank you! > > > > - Daniel Shchur > > > > >
