Hi Jarett, Did you do a recent upgrade of airavata and pga? If not please do so with the latest production. By the information you have provided, it could be an issue with gfac server reading from the rabbitmq queue. But you said although the experiment is in LAUNCHED job is in submitted. So does your email contain unread emails for this job? When was the last time the experiment completed and any changes done to server machines, etc.. from then to now?
Hi Jeff, Yours is slightly different since its in EXECUTING. With the information you have provided, I think your issue could be with email monitoring. Do you have unread emails for the jobs in EXECUTING in your email box? If you do, then you need to check you gfac-config.yaml in airavata bin folder and make sure it processes emails from the comet. hope this info helps for further investigations. Thanks, Eroma On Fri, Jun 23, 2017 at 4:56 PM, Sale, Jeff <[email protected]> wrote: > I have a similar issue. I have been working with the Airavata support > folks, Eroma, Supun, and Marcus for the past few weeks trying to get > Gaussian jobs to run on Comet. They have been super helpful, and it appears > I am now able to run jobs to completion according to the Gaussian.log file > in the scratch directory on Comet, but when I browse to the Experiment on > the PGA the stdout and stderr files never appear as a link in Outputs and > the job status is perpetually in "EXECUTING". > > I seem to recall Supun saying this was something they were aware of and > are working to resolve, but I could be wrong about this. > > Jeff > > ________________________________________ > From: Jarett DeAngelis [[email protected]] > Sent: Friday, June 23, 2017 1:28 PM > To: [email protected] > Subject: Job stuck in "launched," "submitted" status > > Hi gang, > > Working on our Airavata deployment (still build 16) again and have > encountered an issue where after submitting a job to Slurm, it gets stuck > in the “LAUNCHED” state, appearing to have sent the job to Slurm because it > says “SUBMITTED” underneath, but it just stays that way forever. If you > look at RabbitMQ there is a message sitting in the queue. Our first thought > was that it was the email account we’re using for job tracking, but that is > functioning fine. Where should I be looking for answers? > > Thanks, > Jarett > -- Thank You, Best Regards, Eroma
