Hi Daniel, If you look at [2], it's where the protocol specific configuration object is created. Each implementation is a subclass of the JobManagerConfiguration interface [3]. So each implementation might have different execution commands and those are stored in the database and you can register them through PGA. Please look at the screenshot I sent earlier. Once helix figured out the right Configuration implementation, it invokes it at the [4]. Please let me know if this is clear
[2] https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106 [3] https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java [4] https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78 Thanks Dimuthu On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur <[email protected]> wrote: > Hi all, > > > > We have been trying to figure out where the helix system actually > specifies job submission protocol commands (e.g. sbatch, squeue, scancel). > The only direct references we could find are part of the client-sdk > “registerSampleApplication”. Everything else, when referencing > “ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end > files that may not be used anymore. > > > > Question: is there a direct command mapping for the > “Map<JobManagerCommand, string>” that is called within the code, or is this > set elsewhere? We haven’t found any references to where Airavata would know > how to submit a SLURM job so that we can let it know how to submit an > HTCondor job. > > Any help on this would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <[email protected]> > *Sent: *Monday, October 5, 2020 9:50 AM > *To: *Airavata Dev <[email protected]> > *Cc: *[email protected]; Evan Palmer > <[email protected]> > *Subject: *Re: HTCondor Extension Help > > > > Sorry I didn't send the image I mentioned earlier > > > > > > > > On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha < > [email protected]> wrote: > > Hi Daniel, > > > > Your PR looks good to me. I think the best place for you to start is by > setting up the PGA portal (the old portal we used before the django one) > and adding a resource manager type for the compute resource as you can see > in the attached image. You can set up the pga locally using this [1] guide. > Once you can register the resource manager type using the pga portal, you > can use the django portal to submit jobs. I believe that you don't need to > change any code in the django portal. > > > > [1] > https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_ide-2Dintegration-23optional-2Dstarting-2Dsuper-2Dadmin-2Dportal-2Dpga&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw&s=8A5-Br28vywXcU42WH9bl0Ad0b5YMnXNNIplSl3r9J8&e=> > > > > Thanks > > Dimuthu > > > > On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur < > [email protected]> wrote: > > Hi Dimuthu and Suresh, > > > > Thanks for getting back to us. We went ahead and made a work in progress pull > request > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_pull_263&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw&s=axraS0UJlcrcJlRwGIeswZKPC0cHEDu13ADapD1Xz5A&e=> > for our changes we have created so far. We also took a look at the > job-monitor section and will work towards creating additions to support > HTCondor. > > > > Do any of you have any suggestions on what the best way is to test > submissions to HTCondor? We looked into using the Airavata API and the > Django portal and are curious what you would recommend, and how to do it > using your recommendation. > > > > Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <[email protected]> > *Sent: *Friday, October 2, 2020 5:22 PM > *To: *Airavata Dev <[email protected]> > *Cc: *[email protected]; Evan Palmer > <[email protected]> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > This is great progress. I think you are on the right track. Did you manage > to set up the ide integration for the development of Airavata. If you have > already done that, you can test these features. However I recommend you to > send pull requests to a separate branch (htcondor?) and then I can provide > you feedback on the next steps. > > > > Thanks > > Dimuthu > > > > > > On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru <[email protected]> wrote: > > Hi Daniel, > > > > Thanks for this nice summary for keeping the dev list apprised of your > progress. > > > > One additional item is to discuss and plan on job monitoring approaches, > you can look at current references here - > https://github.com/apache/airavata/tree/master/modules/job-monitor > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_job-2Dmonitor&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=zpJDdIRgv9hm0QkGZuHhDOYvc1_zMhUmH9TF7mx9A-0&e=> > > > > > Suresh > > > > On Oct 1, 2020, at 4:52 PM, Daniel Shchur <[email protected]> > wrote: > > > > Hi all, > > > > We are working on the HTCondor extension for Airavata. We have gone > through the code and started to implement additions based on the other job > workflow systems such as SLURM. We forked > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_docquantum_airavata_tree_htcondor&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=14ShokmbcQHBZlmV67fTrNoK8ARktUQERu6wRQQTlWc&e=> > the > project and created a new branch (htcondor) to host our changes. We will > make a pull request into the main Airavata repository for it once we have > finished the implementation. A relevant issue has also been created here > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_AIRAVATA-2D3377&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=ZjE4skBTqEHdJqLBat2-uLoS1w6_Ksh_c5d4VPKjnS8&e=> > . > > > > We would like help to know exactly where additions need to be made in case > we missed a few spots. At this point, we know that: > > - A groovy template needs to be made (finished) > - A job configuration class needs to be made (WIP) > - A job parser class needs to be made (WIP) > - And enums need to be modified throughout the project so that systems > know of HTCondor as an option. (finished) > > > > If any of you could direct us to the right places, that would be greatly > appreciated. > > > > Also, if there’s more documentation or information about the code, such as > how the command map should work or should be implemented (configuration > class?), that would be appreciated. > > > > Thank you! > > > > - Daniel Shchur > > > > > > >
