Hi Daniel, Sorry for the late reply. Can you send me the output of following commands
docker-compose exec pga getent hosts host.docker.internal docker-compose exec pga cat /etc/hosts ipconfig And the last few lines of the APIServerStarter console output. Thanks Dimuthu On Tue, Oct 20, 2020 at 6:29 PM Daniel Shchur <daniel.shc...@huskers.unl.edu> wrote: > Here’s and image for reference > > - Daniel Shchur > > > > *From: *Daniel Shchur <daniel.shc...@huskers.unl.edu> > *Sent: *Tuesday, October 20, 2020 3:36 PM > *To: *dev@airavata.apache.org > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *RE: HTCondor Extension Help > > > > Hi Dimuthu, > > > > I must have done something wrong because a colleague of mine was able to > get into the login page without the extra network changes with docker. > However, he was still experiencing the issue where the login page would go > back to the home page and note “The Airavata servers are currently down” > when trying to login with “default-admin.” We are currently trying to go > through the logs to see why this might be happening. > > > > Thanks! > > > > - Daniel Shchur > > > > *From: *Daniel Shchur <daniel.shc...@huskers.unl.edu> > *Sent: *Monday, October 19, 2020 9:02 PM > *To: *dev@airavata.apache.org > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *RE: HTCondor Extension Help > > > > Hi Dimuthu, > > > > I did try the steps mentioned in [10]. I forgot to mention that when doing > so, I wouldn’t get any connection at all. However, if I were to simply > leave it as: > > > > docker-compose exec pga getent hosts > > > > I could then copy the last address (which wasn’t a loopback) and use the > next `exec` function to set the container’s hosts file, allowing me to > connect to the web interface presented by the container. I should note that > we have one developer doing this on native Ubuntu Linux and he faced the > same issue. > > > > Thank you! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Monday, October 19, 2020 5:29 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > Did you follow the steps mentioned in [10]?. I recently added couple of > instructions that is specific to windows deployments > > docker-compose exec pga getent hosts host.docker.internal > > docker-compose exec pga /bin/sh -c "echo '<host-machine ip> airavata.host' >> > /etc/hosts" > > > > [10] > https://github.com/apache/airavata/tree/develop/modules/ide-integration#optional-starting-super-admin-portal-pga > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_develop_modules_ide-2Dintegration-23optional-2Dstarting-2Dsuper-2Dadmin-2Dportal-2Dpga&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=Ypkf_DmUlrtRhSp_fDiIv6WQo6FaPTI9Qqf6Dutchw0&s=t9cv4PIElFkkZUmTp4QOg_j6bhsIQasdi6gKCIxrmzM&e=> > > > > Thanks > > Dimuthu > > > > On Sun, Oct 18, 2020 at 7:26 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu, > > > > We had some trouble initially getting the changes pulled in, but we > eventually did and were able to run the portal. However, when running as > recommended, the PGA portal is not able to communicate with the database > stack with Keystore and so on, so we weren’t able to get to the login. The > error we receive is: > > > > curl_exec() failed. Error1: Failed to connect to airavata.host port 8443: > Connection refused > > > > One theory I had was that the containers aren’t communicating with each > other so I added an external network within their respective docker-compose > files might mitigate the issue, and it did. I was able to get into the > login portal without the issue above given the resources stack and pga > container were all on the same docker internal network. However, now when > trying to log in, the portal redirects to the home page noting that the > “Airavata servers are currently down”. > > Any help with this would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Wednesday, October 14, 2020 12:33 AM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > We only have tested ide integration on MacOS and Ubuntu. However I just > managed to reproduce your issue in Windows 10 with WSL2 enabled and > commits [5-9] done to the develop branch should fix those issues. You might > have to merge those changes to your fork to make it effective and I > recommend getting a fresh clone to avoid unexpected behaviors due to temp > files. Let us know if it works for you. > > > > [5] > https://github.com/apache/airavata/commit/f9b9e203fc268db468977680ab511f88a3eadc99 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_f9b9e203fc268db468977680ab511f88a3eadc99&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8&s=jXtAEOkym95skPFsXd9YW4PxKg8ko3fmd2lHgphKA2M&e=> > > [6] > https://github.com/apache/airavata/commit/3794421f01fdb4ef5c0199fdfc6cfc15c25d2e7a > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_3794421f01fdb4ef5c0199fdfc6cfc15c25d2e7a&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8&s=avw7ECyJ3UUo0YYlphKm8LWxDrf_wHmaoUsMSSx_CyU&e=> > > [7] > https://github.com/apache/airavata/commit/f581b61d44eb9a848b866456acd63f29e2bbfc8e > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_f581b61d44eb9a848b866456acd63f29e2bbfc8e&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8&s=_LrHNm7vMt75JMADyCyHs7VWKutPjKZCjw8YLnZF0xQ&e=> > > [8] > https://github.com/apache/airavata/commit/9e30529d7545a1df0b9d94c2783d7d11c32d301d > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_9e30529d7545a1df0b9d94c2783d7d11c32d301d&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8&s=aO43YTRpZ8PLuZ84xcB_yR-5yT8GOKXtfwHKxAgdkak&e=> > > [9] > https://github.com/apache/airavata/commit/b11d81182f915491187dcf9f3ede5beeb5ea94a2 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_b11d81182f915491187dcf9f3ede5beeb5ea94a2&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8&s=k_GXTpaj3wcjCgsjj8PIzxxFISaGmYekMoTg2rk5aM0&e=> > > > > Thanks > > Dimuthu > > > > On Tue, Oct 13, 2020 at 6:37 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu, > > > > Windows 10 2004 19041.508 > > Docker v19.03.13 running on WSL2 - Ubuntu 20.04 > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Tuesday, October 13, 2020 5:34 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > I'm trying to reproduce the error locally. Can you please send me the > following information of your local setup. > > > > OS / version > > Docker version > > > > Thanks > > Dimuthu > > > > On Tue, Oct 13, 2020 at 5:32 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimunth, > > > > We have been trying to get the PGA running; however, it seems to fail > building currently during the optimize portion. This is after having > started the resources stack and starting the JobEngineStarter and > AiravataAPI. Those parts start up without a hitch. I also added in the > hosts file changes. > > > > The error is: > > > > > php artisan optimize > > Generating optimized class loader > > Compiling common classes > > Compiling views > > > > > > > [ErrorException] > > > > file_put_contents(/var/www/portals/default/app/storage/views/16ac8ea2e3f88487e87226a07d300af0): > failed to open stream: No such file or directory > > > > > > > optimize [--force] [--psr] > > > > Script php artisan optimize handling the post-install-cmd event returned > with error code 1 > > Any help would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Friday, October 9, 2020 8:46 AM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > Please send you PR to airavata:htcondor branch which I created yesterday. > Then I also can help you to speed up the development process. > > > > Thanks > > Dimuthu > > > > On Thu, Oct 8, 2020 at 5:59 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu, > > > > That clears up a lot of confusion. Thank you! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Thursday, October 8, 2020 4:58 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > That's correct, we assume all job manager configurations have unique set > of command types like job submission, cancellation, status checking etc. > It's up to the end user to specify the exact command for each command type. > For example, SLURM job submission command is sbatch. So the user has to > specify that though the portal which will eventually be stored in the > database. > > > > Thanks > > Dimuthu > > > > > > On Thu, Oct 8, 2020 at 5:14 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu, > > > > So, to be clear, > > > > The relevant job submission commands are registered at the client-side? As > in, the front end interface and are not hardcoded into the backend? > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Thursday, October 8, 2020 4:12 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > If you look at [2], it's where the protocol specific configuration object > is created. Each implementation is a subclass of the > JobManagerConfiguration interface [3]. So each implementation might have > different execution commands and those are stored in the database and you > can register them through PGA. Please look at the screenshot I sent > earlier. Once helix figured out the right Configuration implementation, it > invokes it at the [4]. Please let me know if this is clear > > > > [2] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobFactory.java-23L106&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU&s=jOokB292No_5nhnM9IQCJ3f4HgYCBysW7cYy7r_iwns&e=> > > [3] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobManagerConfiguration.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU&s=E9HjTKXLL3YhkdLhA98G3r0fH7lpD5WKlJldHoFI32E&e=> > > [4] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_JobSubmissionTask.java-23L78&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU&s=yuHKbjJZT93zcEiQHAIPBNZMq6aKiKLmzkSep5i2-j8&e=> > > > > Thanks > > Dimuthu > > > > On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi all, > > > > We have been trying to figure out where the helix system actually > specifies job submission protocol commands (e.g. sbatch, squeue, scancel). > The only direct references we could find are part of the client-sdk > “registerSampleApplication”. Everything else, when referencing > “ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end > files that may not be used anymore. > > > > Question: is there a direct command mapping for the > “Map<JobManagerCommand, string>” that is called within the code, or is this > set elsewhere? We haven’t found any references to where Airavata would know > how to submit a SLURM job so that we can let it know how to submit an > HTCondor job. > > Any help on this would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Monday, October 5, 2020 9:50 AM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Sorry I didn't send the image I mentioned earlier > > > > > > > > On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > > Hi Daniel, > > > > Your PR looks good to me. I think the best place for you to start is by > setting up the PGA portal (the old portal we used before the django one) > and adding a resource manager type for the compute resource as you can see > in the attached image. You can set up the pga locally using this [1] guide. > Once you can register the resource manager type using the pga portal, you > can use the django portal to submit jobs. I believe that you don't need to > change any code in the django portal. > > > > [1] > https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_ide-2Dintegration-23optional-2Dstarting-2Dsuper-2Dadmin-2Dportal-2Dpga&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw&s=8A5-Br28vywXcU42WH9bl0Ad0b5YMnXNNIplSl3r9J8&e=> > > > > Thanks > > Dimuthu > > > > On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur < > daniel.shc...@huskers.unl.edu> wrote: > > Hi Dimuthu and Suresh, > > > > Thanks for getting back to us. We went ahead and made a work in progress pull > request > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_pull_263&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw&s=axraS0UJlcrcJlRwGIeswZKPC0cHEDu13ADapD1Xz5A&e=> > for our changes we have created so far. We also took a look at the > job-monitor section and will work towards creating additions to support > HTCondor. > > > > Do any of you have any suggestions on what the best way is to test > submissions to HTCondor? We looked into using the Airavata API and the > Django portal and are curious what you would recommend, and how to do it > using your recommendation. > > > > Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Sent: *Friday, October 2, 2020 5:22 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Cc: *daniellthibod...@gmail.com; Evan Palmer > <evan.pal...@huskers.unl.edu> > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > > > > This is great progress. I think you are on the right track. Did you manage > to set up the ide integration for the development of Airavata. If you have > already done that, you can test these features. However I recommend you to > send pull requests to a separate branch (htcondor?) and then I can provide > you feedback on the next steps. > > > > Thanks > > Dimuthu > > > > > > On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru <sma...@apache.org> wrote: > > Hi Daniel, > > > > Thanks for this nice summary for keeping the dev list apprised of your > progress. > > > > One additional item is to discuss and plan on job monitoring approaches, > you can look at current references here - > https://github.com/apache/airavata/tree/master/modules/job-monitor > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_job-2Dmonitor&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=zpJDdIRgv9hm0QkGZuHhDOYvc1_zMhUmH9TF7mx9A-0&e=> > > > > > Suresh > > > > On Oct 1, 2020, at 4:52 PM, Daniel Shchur <daniel.shc...@huskers.unl.edu> > wrote: > > > > Hi all, > > > > We are working on the HTCondor extension for Airavata. We have gone > through the code and started to implement additions based on the other job > workflow systems such as SLURM. We forked > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_docquantum_airavata_tree_htcondor&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=14ShokmbcQHBZlmV67fTrNoK8ARktUQERu6wRQQTlWc&e=> > the > project and created a new branch (htcondor) to host our changes. We will > make a pull request into the main Airavata repository for it once we have > finished the implementation. A relevant issue has also been created here > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_AIRAVATA-2D3377&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo&s=ZjE4skBTqEHdJqLBat2-uLoS1w6_Ksh_c5d4VPKjnS8&e=> > . > > > > We would like help to know exactly where additions need to be made in case > we missed a few spots. At this point, we know that: > > · A groovy template needs to be made (finished) > > · A job configuration class needs to be made (WIP) > > · A job parser class needs to be made (WIP) > > · And enums need to be modified throughout the project so that > systems know of HTCondor as an option. (finished) > > > > If any of you could direct us to the right places, that would be greatly > appreciated. > > > > Also, if there’s more documentation or information about the code, such as > how the command map should work or should be implemented (configuration > class?), that would be appreciated. > > > > Thank you! > > > > - Daniel Shchur > > > > > > > > > > > > > > > > > > > > > > >
