Hi Lahiru,
Issues again, we submitted many jobs and it seems the Airavata endpoint
is failing, worse than normal...
Error trying to initialize service account user profile: Required field
'accessToken' is unset! Struct:AuthzToken(accessToken:null,
claimsMap:{userName=service-account-lims,
gatewayID=ultrascan})string(1074) "#0
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/lib/Airavata/API/Airavata.php(5947):
Airavata\API\AiravataClient->recv_searchProjects() #1
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataUtils.php(64):
Airavata\API\AiravataClient->searchProjects(Object(Airavata\Model\Security\AuthzToken),
'ultrascan', 'service-account...', Array, -1, 0) #2
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataWrapper.php(135):
fetch_projectid(Object(Airavata\API\AiravataClient),
Object(Airavata\Model\Security\AuthzToken), 'ultrascan',
'Emre_Brookes_c3...') #3
/srv/www/htdocs/common/class/submit_airavata.php(213):
SCIGAP\AiravataWrapper->launch_airavata_experiment('uslims.uleth.ca',
'Emre_Brookes_c3...', 'US3-AIRA', 'uslims3_Worksho...',
'expanse.sdsc.ed...', 'shared', 32, 1, 1, 114, 'us3', 'NONE', 'NONE',
'/srv/www/htdocs...', 'ab1fd003-7c82-0...', 64000) #4
/srv/www/htdocs/common/class/submit_airavata.php(20):
submit_airavata->createExperiment() #5
/srv/www/htdocs/uslims3/uslims3_Workshop/2DSA_2.php(222):
submit_airavata->submit() #6 {main}"
We will try to limit the number of simultaneous active jobs for further
testing.
Thanks,
Emre
emre brookes wrote:
Hi Lahiru,
Thanks for the quick work!
I see many seem to have finished now.
Will check again in the morning to see what happens with the last few.
Is there a practical limit to the number of jobs that can be submitted
to the metascheduler?
Best regards,
Emre
Lahiru Jayathilake wrote:
*EXTERNAL EMAIL*
Hi Emre,
I hope you're doing well!
I’ve checked the issue, and the experiments are now being submitted.
Let me know if you notice any further issues.
Best,
Lahiru
On Sat, Nov 30, 2024 at 6:11 PM Emre Brookes
<[email protected] <mailto:[email protected]>> wrote:
Hi,
Hope you had a good Thanksgiving & your grant submission is close
to done.
We submitted 100 jobs via the metascheduler and it seems to have
choked
further submission.
No jobs, even from other servers seem to be going through.
e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from
uslims.uleth.ca
<http://uslims.uleth.ca/>
seems stuck
& these (of many) from uslimstest.genapp.rocks also seem stuck:
--------------------------------------------------------------------------------
status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
"SCHEDULED"
compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
"Resource not yet assigned"
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
"EXECUTING"
compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
"expanse.sdsc.edu
<http://expanse.sdsc.edu/>"
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Thanks,
Emre
CAUTION: This message originated outside of UT Health San Antonio.
Please exercise caution when clicking on links or opening attachments.
.
