Hi Lahiru, I’ve managed to run all the jobs that were stalled.
Thank you. Aaron From: Lahiru Jayathilake <[email protected]> Date: Sunday, December 1, 2024 at 7:15 PM To: Emre Brookes <[email protected]> Cc: emre brookes <[email protected]>, [email protected] <[email protected]>, Eroma Abeysinghe <[email protected]>, [email protected] <[email protected]> Subject: Re: Airavata jobs failing Hi Emre, This issue has been resolved. Could you please verify if everything is in order? Additionally, there’s no limitation on the number of jobs that can be submitted. Best, Lahiru On Sun, Dec 1, 2024 at 1:09 PM Emre Brookes <[email protected]<mailto:[email protected]>> wrote: Hi Lahiru, Issues again, we submitted many jobs and it seems the Airavata endpoint is failing, worse than normal... Error trying to initialize service account user profile: Required field 'accessToken' is unset! Struct:AuthzToken(accessToken:null, claimsMap:{userName=service-account-lims, gatewayID=ultrascan})string(1074) "#0 /srv/www/htdocs/common/class/ultrascan-airavata-bridge/lib/Airavata/API/Airavata.php(5947): Airavata\API\AiravataClient->recv_searchProjects() #1 /srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataUtils.php(64): Airavata\API\AiravataClient->searchProjects(Object(Airavata\Model\Security\AuthzToken), 'ultrascan', 'service-account...', Array, -1, 0) #2 /srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataWrapper.php(135): fetch_projectid(Object(Airavata\API\AiravataClient), Object(Airavata\Model\Security\AuthzToken), 'ultrascan', 'Emre_Brookes_c3...') #3 /srv/www/htdocs/common/class/submit_airavata.php(213): SCIGAP\AiravataWrapper->launch_airavata_experiment('uslims.uleth.ca<http://uslims.uleth.ca>', 'Emre_Brookes_c3...', 'US3-AIRA', 'uslims3_Worksho...', 'expanse.sdsc.ed...', 'shared', 32, 1, 1, 114, 'us3', 'NONE', 'NONE', '/srv/www/htdocs...', 'ab1fd003-7c82-0...', 64000) #4 /srv/www/htdocs/common/class/submit_airavata.php(20): submit_airavata->createExperiment() #5 /srv/www/htdocs/uslims3/uslims3_Workshop/2DSA_2.php(222): submit_airavata->submit() #6 {main}" We will try to limit the number of simultaneous active jobs for further testing. Thanks, Emre emre brookes wrote: > Hi Lahiru, > > Thanks for the quick work! > > I see many seem to have finished now. > Will check again in the morning to see what happens with the last few. > > Is there a practical limit to the number of jobs that can be submitted > to the metascheduler? > > Best regards, > Emre > > > Lahiru Jayathilake wrote: >> *EXTERNAL EMAIL* >> >> Hi Emre, >> >> I hope you're doing well! >> I’ve checked the issue, and the experiments are now being submitted. >> >> Let me know if you notice any further issues. >> >> Best, >> Lahiru >> >> On Sat, Nov 30, 2024 at 6:11 PM Emre Brookes >> <[email protected]<mailto:[email protected]> >> <mailto:[email protected]<mailto:[email protected]>>> wrote: >> >> Hi, >> >> Hope you had a good Thanksgiving & your grant submission is close >> to done. >> We submitted 100 jobs via the metascheduler and it seems to have >> choked >> further submission. >> No jobs, even from other servers seem to be going through. >> >> e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from >> uslims.uleth.ca<http://uslims.uleth.ca> >> <http://uslims.uleth.ca/> >> seems stuck >> >> & these (of many) from uslimstest.genapp.rocks also seem stuck: >> >> -------------------------------------------------------------------------------- >> status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e: >> "SCHEDULED" >> compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e: >> "Resource not yet assigned" >> -------------------------------------------------------------------------------- >> -------------------------------------------------------------------------------- >> status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c: >> "EXECUTING" >> compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c: >> "expanse.sdsc.edu<http://expanse.sdsc.edu> >> <http://expanse.sdsc.edu/>" >> -------------------------------------------------------------------------------- >> -------------------------------------------------------------------------------- >> >> Thanks, >> Emre >> >> >> CAUTION: This message originated outside of UT Health San Antonio. >> Please exercise caution when clicking on links or opening attachments. > > . >
