Hi Dimuthu, I haven’t had a lot of experience with Condor. One of my colleagues, Derek, has more experience with this so I’ll have to ask him to help me a little. The way he explained it however is that there are equivalent commands to all of the SLURM commands. The submission script isn’t too different from what SLURM uses as well. This link should provide some insight: https://htcondor.readthedocs.io/en/latest/users-manual/submitting-a-job.html
I’ll write up a step by step shortly. - Daniel Shchur From: DImuthu Upeksha<mailto:[email protected]> Sent: Wednesday, September 16, 2020 1:45 PM To: Daniel Shchur<mailto:[email protected]> Cc: Airavata Users<mailto:[email protected]> Subject: Re: Submitting jobs on Open Science Grid Hi Daniel, Good thing about using a Groovy script and using cli to submit jobs is that you don't have to re-invent anything. You can simply follow what we have done for other schedulers and replicate them. I would also favor that approach than the python one. I just wasn't aware that HTCondor has that capability (CLI). Can you please summarize the steps (commands) that you normally follow to submit a job to HTCondor through the CLI? Thanks Dimuthu On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur <[email protected]<mailto:[email protected]>> wrote: Hi Dimuthu, Something we have been discussing is going the route of SLURM and implementing a Groovy template to submit jobs via CLI, as well as ping Condor to get status. Would it be preferable to try and go through python bindings instead to call the API directly? - Daniel Shchur From: DImuthu Upeksha<mailto:[email protected]> Sent: Wednesday, September 16, 2020 1:30 PM To: Airavata Users<mailto:[email protected]> Cc: Daniel Shchur<mailto:[email protected]> Subject: Re: Submitting jobs on Open Science Grid Hi Daniel, If you follow the paper that Suresh has sent, you might notice that there is a Job submission task in the pre workflow manager. You have to implement a job submission task for HTCondor like we do for standard job schedulers like slurm and pbs. Have a look at [1] that implements a ssh based submission to schedulers. However, as I have heard, there is a python client for HTCondor submissions. So you don't need to worry about all these SSH based communications. You can use any of these [2] ways to talk to the python client through java. Implementing a job management extension for Airavata includes multiple steps so I would suggest you to work on that step by step. As the first step, try to come up with a sample java code that accepts all the parameters and submits a HTCondor job through the python client. Once you have a working code, I'll help you on how to generalize it and migrate it to Airavata. [1] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=zN0maF1ewq6hAC6VMBZKPhZWRuwXE70h5w7A5NASKTc&e=> [2] https://www.baeldung.com/java-working-with-python<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=tJVmrIlkB6rca7GLYAle5AVZcIJY6Z6SkbajdJMEVgQ&e=> Thanks Dimuthu On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <[email protected]<mailto:[email protected]>> wrote: Hi Daniel, As we discussed offline, it will be good to get a contribution from you on the HTCondor extension. Please refer to this attached paper to get an idea about the job management framework. Suresh On Sep 8, 2020, at 5:59 PM, Daniel Shchur <[email protected]<mailto:[email protected]>> wrote: Hi all, I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid. We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated. Thank you! - Daniel Shchur
