Hi Dimuthu, Something we have been discussing is going the route of SLURM and implementing a Groovy template to submit jobs via CLI, as well as ping Condor to get status. Would it be preferable to try and go through python bindings instead to call the API directly?
- Daniel Shchur From: DImuthu Upeksha<mailto:[email protected]> Sent: Wednesday, September 16, 2020 1:30 PM To: Airavata Users<mailto:[email protected]> Cc: Daniel Shchur<mailto:[email protected]> Subject: Re: Submitting jobs on Open Science Grid Hi Daniel, If you follow the paper that Suresh has sent, you might notice that there is a Job submission task in the pre workflow manager. You have to implement a job submission task for HTCondor like we do for standard job schedulers like slurm and pbs. Have a look at [1] that implements a ssh based submission to schedulers. However, as I have heard, there is a python client for HTCondor submissions. So you don't need to worry about all these SSH based communications. You can use any of these [2] ways to talk to the python client through java. Implementing a job management extension for Airavata includes multiple steps so I would suggest you to work on that step by step. As the first step, try to come up with a sample java code that accepts all the parameters and submits a HTCondor job through the python client. Once you have a working code, I'll help you on how to generalize it and migrate it to Airavata. [1] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=zN0maF1ewq6hAC6VMBZKPhZWRuwXE70h5w7A5NASKTc&e=> [2] https://www.baeldung.com/java-working-with-python<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=tJVmrIlkB6rca7GLYAle5AVZcIJY6Z6SkbajdJMEVgQ&e=> Thanks Dimuthu On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <[email protected]<mailto:[email protected]>> wrote: Hi Daniel, As we discussed offline, it will be good to get a contribution from you on the HTCondor extension. Please refer to this attached paper to get an idea about the job management framework. Suresh On Sep 8, 2020, at 5:59 PM, Daniel Shchur <[email protected]<mailto:[email protected]>> wrote: Hi all, I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid. We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated. Thank you! - Daniel Shchur
