Hi Eroma, I have no emails in my Airavata account for this job at all. The last time an email went into/out of it was the end of 2016, which is also the last time this instance of Airavata was tested or used.
Jarett > On Jun 26, 2017, at 11:50 AM, Sale, Jeff <[email protected]> wrote: > > Thanks, Eroma. Yes, my gfac-config.yaml file appears to be configured > properly. I am still using the [email protected] > <mailto:[email protected]> email and the emails are being sent to it > correctly. My airavata-server.properties file appears to also be configured > properly with that email. > > However, I went ahead made one seemingly minor change and added the test > airavata gmail account to the airavata-server.properties file under **** > Monitoring module Configuration ******. It may have been previously set to > the default airavata user and xxx password by me when I sent you the file a > few weeks ago. Sorry about that, if that in fact was the key change. > > Once I made that change the job completed and the Gaussian.log, .stderr, and > .stdout files were successfully created locally. The latter two were empty, > but the .log file seems to imply that the job ran successfully, tho' I am not > a Gaussian person. > > Also, having made that change, my server is now getting a bunch of [ERROR] > outputs to the console from what appear to be previous failed jobs which used > the same test airavata email account. I'm guessing I can ignore these but I'm > not sure. Any thoughts? Also, the links in the Experiment view to the > Gaussian.stderr, .stdout, .log files are still broken, but I can view them if > I click the "Open" link for the Storage. > > Next, I will go ahead and change the email account to my own test gmail > account and see what happens. Thanks! > > Jeff > > > From: Eroma Abeysinghe [[email protected] > <mailto:[email protected]>] > Sent: Monday, June 26, 2017 6:35 AM > To: [email protected] <mailto:[email protected]> > Subject: Re: Job stuck in "launched," "submitted" status > > Hi Jarett, > > Did you do a recent upgrade of airavata and pga? If not please do so with the > latest production. By the information you have provided, it could be an issue > with gfac server reading from the rabbitmq queue. But you said although the > experiment is in LAUNCHED job is in submitted. So does your email contain > unread emails for this job? When was the last time the experiment completed > and any changes done to server machines, etc.. from then to now? > > Hi Jeff, > Yours is slightly different since its in EXECUTING. With the information you > have provided, I think your issue could be with email monitoring. Do you have > unread emails for the jobs in EXECUTING in your email box? If you do, then > you need to check you gfac-config.yaml in airavata bin folder and make sure > it processes emails from the comet. > > hope this info helps for further investigations. > > Thanks, > Eroma > > On Fri, Jun 23, 2017 at 4:56 PM, Sale, Jeff <[email protected] > <mailto:[email protected]>> wrote: > I have a similar issue. I have been working with the Airavata support folks, > Eroma, Supun, and Marcus for the past few weeks trying to get Gaussian jobs > to run on Comet. They have been super helpful, and it appears I am now able > to run jobs to completion according to the Gaussian.log file in the scratch > directory on Comet, but when I browse to the Experiment on the PGA the stdout > and stderr files never appear as a link in Outputs and the job status is > perpetually in "EXECUTING". > > I seem to recall Supun saying this was something they were aware of and are > working to resolve, but I could be wrong about this. > > Jeff > > ________________________________________ > From: Jarett DeAngelis [[email protected] <mailto:[email protected]>] > Sent: Friday, June 23, 2017 1:28 PM > To: [email protected] <mailto:[email protected]> > Subject: Job stuck in "launched," "submitted" status > > Hi gang, > > Working on our Airavata deployment (still build 16) again and have > encountered an issue where after submitting a job to Slurm, it gets stuck in > the “LAUNCHED” state, appearing to have sent the job to Slurm because it says > “SUBMITTED” underneath, but it just stays that way forever. If you look at > RabbitMQ there is a message sitting in the queue. Our first thought was that > it was the email account we’re using for job tracking, but that is > functioning fine. Where should I be looking for answers? > > Thanks, > Jarett > > > > -- > Thank You, > Best Regards, > Eroma
