Hi Daniel, We only have tested ide integration on MacOS and Ubuntu. However I just managed to reproduce your issue in Windows 10 with WSL2 enabled and commits [5-9] done to the develop branch should fix those issues. You might have to merge those changes to your fork to make it effective and I recommend getting a fresh clone to avoid unexpected behaviors due to temp files. Let us know if it works for you.
[5] https://github.com/apache/airavata/commit/f9b9e203fc268db468977680ab511f88a3eadc99 [6] https://github.com/apache/airavata/commit/3794421f01fdb4ef5c0199fdfc6cfc15c25d2e7a [7] https://github.com/apache/airavata/commit/f581b61d44eb9a848b866456acd63f29e2bbfc8e [8] https://github.com/apache/airavata/commit/9e30529d7545a1df0b9d94c2783d7d11c32d301d [9] https://github.com/apache/airavata/commit/b11d81182f915491187dcf9f3ede5beeb5ea94a2 Thanks Dimuthu On Tue, Oct 13, 2020 at 6:37 PM Daniel Shchur <daniel.shc...@huskers.unl.edu> wrote: > Hi Dimuthu, > > > > Windows 10 2004 19041.508 > > Docker v19.03.13 running on WSL2 - Ubuntu 20.04 > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Tuesday, October 13, 2020 5:34 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > I'm trying to reproduce the error locally. Can you please send me the > following information of your local setup. > > > > OS / version > > Docker version > > > > Thanks > > Dimuthu > > > > On Tue, Oct 13, 2020 at 5:32 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimunth, > > > > We have been trying to get the PGA running; however, it seems to fail > building currently during the optimize portion. This is after having > started the resources stack and starting the JobEngineStarter and > AiravataAPI. Those parts start up without a hitch. I also added in the > hosts file changes. > > > > The error is: > > > > > php artisan optimize > > Generating optimized class loader > > Compiling common classes > > Compiling views > > > > > > > [ErrorException] > > > > file_put_contents(/var/www/portals/default/app/storage/views/16ac8ea2e3f88487e87226a07d300af0): > failed to open stream: No such file or directory > > > > > > > optimize [--force] [--psr] > > > > Script php artisan optimize handling the post-install-cmd event returned > with error code 1 > > Any help would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Friday, October 9, 2020 8:46 AM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > Please send you PR to airavata:htcondor branch which I created yesterday. > Then I also can help you to speed up the development process. > > > > Thanks > > Dimuthu > > > > On Thu, Oct 8, 2020 at 5:59 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu, > > > > That clears up a lot of confusion. Thank you! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Thursday, October 8, 2020 4:58 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > That's correct, we assume all job manager configurations have unique set > of command types like job submission, cancellation, status checking etc. > It's up to the end user to specify the exact command for each command type. > For example, SLURM job submission command is sbatch. So the user has to > specify that though the portal which will eventually be stored in the > database. > > > > Thanks > > Dimuthu > > > > > > On Thu, Oct 8, 2020 at 5:14 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu, > > > > So, to be clear, > > > > The relevant job submission commands are registered at the client-side? As > in, the front end interface and are not hardcoded into the backend? > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Thursday, October 8, 2020 4:12 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > If you look at [2], it's where the protocol specific configuration object > is created. Each implementation is a subclass of the > JobManagerConfiguration interface [3]. So each implementation might have > different execution commands and those are stored in the database and you > can register them through PGA. Please look at the screenshot I sent > earlier. Once helix figured out the right Configuration implementation, it > invokes it at the [4]. Please let me know if this is clear > > > > [2] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobFactory.java-23L106&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU&s=jOokB292No_5nhnM9IQCJ3f4HgYCBysW7cYy7r_iwns&e=> > > [3] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobManagerConfiguration.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU&s=E9HjTKXLL3YhkdLhA98G3r0fH7lpD5WKlJldHoFI32E&e=> > > [4] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_JobSubmissionTask.java-23L78&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU&s=yuHKbjJZT93zcEiQHAIPBNZMq6aKiKLmzkSep5i2-j8&e=> > > > > Thanks > > Dimuthu > > > > On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi all, > > > > We have been trying to figure out where the helix system actually > specifies job submission protocol commands (e.g. sbatch, squeue, scancel). > The only direct references we could find are part of the client-sdk > “registerSampleApplication”. Everything else, when referencing > “ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end > files that may not be used anymore. > > > > Question: is there a direct command mapping for the > “Map<JobManagerCommand, string>” that is called within the code, or is this > set elsewhere? We haven’t found any references to where Airavata would know > how to submit a SLURM job so that we can let it know how to submit an > HTCondor job. > > Any help on this would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Monday, October 5, 2020 9:50 AM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Sorry I didn't send the image I mentioned earlier > > > > > > > > On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > > Hi Daniel, > > > > Your PR looks good to me. I think the best place for you to start is by > setting up the PGA portal (the old portal we used before the django one) > and adding a resource manager type for the compute resource as you can see > in the attached image. You can set up the pga locally using this [1] guide. > Once you can register the resource manager type using the pga portal, you > can use the django portal to submit jobs. I believe that you don't need to > change any code in the django portal. > > > > [1] > https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_ide-2Dintegration-23optional-2Dstarting-2Dsuper-2Dadmin-2Dportal-2Dpga&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw&s=8A5-Br28vywXcU42WH9bl0Ad0b5YMnXNNIplSl3r9J8&e=> > > > > Thanks > > Dimuthu > > > > On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu and Suresh, > > > > Thanks for getting back to us. We went ahead and made a work in progress pull > request > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_pull_263&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw&s=axraS0UJlcrcJlRwGIeswZKPC0cHEDu13ADapD1Xz5A&e=> > for our changes we have created so far. We also took a look at the > job-monitor section and will work towards creating additions to support > HTCondor. > > > > Do any of you have any suggestions on what the best way is to test > submissions to HTCondor? We looked into using the Airavata API and the > Django portal and are curious what you would recommend, and how to do it > using your recommendation. > > > > Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Friday, October 2, 2020 5:22 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > This is great progress. I think you are on the right track. Did you manage > to set up the ide integration for the development of Airavata. If you have > already done that, you can test these features. However I recommend you to > send pull requests to a separate branch (htcondor?) and then I can provide > you feedback on the next steps. > > > > Thanks > > Dimuthu > > > > > > On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru <sma...@apache.org> wrote: > > Hi Daniel, > > > > Thanks for this nice summary for keeping the dev list apprised of your > progress. > > > > One additional item is to discuss and plan on job monitoring approaches, > you can look at current references here - > https://github.com/apache/airavata/tree/master/modules/job-monitor > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_job-2Dmonitor&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=zpJDdIRgv9hm0QkGZuHhDOYvc1_zMhUmH9TF7mx9A-0&e=> > > > > > Suresh > > > > On Oct 1, 2020, at 4:52 PM, Daniel Shchur <daniel.shc...@huskers.unl.edu> > wrote: > > > > Hi all, > > > > We are working on the HTCondor extension for Airavata. We have gone > through the code and started to implement additions based on the other job > workflow systems such as SLURM. We forked > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_docquantum_airavata_tree_htcondor&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=14ShokmbcQHBZlmV67fTrNoK8ARktUQERu6wRQQTlWc&e=> > the > project and created a new branch (htcondor) to host our changes. We will > make a pull request into the main Airavata repository for it once we have > finished the implementation. A relevant issue has also been created here > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_AIRAVATA-2D3377&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=ZjE4skBTqEHdJqLBat2-uLoS1w6_Ksh_c5d4VPKjnS8&e=> > . > > > > We would like help to know exactly where additions need to be made in case > we missed a few spots. At this point, we know that: > > - A groovy template needs to be made (finished) > - A job configuration class needs to be made (WIP) > - A job parser class needs to be made (WIP) > - And enums need to be modified throughout the project so that systems > know of HTCondor as an option. (finished) > > > > If any of you could direct us to the right places, that would be greatly > appreciated. > > > > Also, if there’s more documentation or information about the code, such as > how the command map should work or should be implemented (configuration > class?), that would be appreciated. > > > > Thank you! > > > > - Daniel Shchur > > > > > > > > > > > > > > >