Prentice, Will do. Thanks again!
--murph -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Prentice Bisbal Sent: Thursday, February 09, 2012 4:25 PM To: [email protected] Subject: Re: [gridengine users] Another MATLAB + SGE Question Awesome! Thanks, Murph! If you already have a ticket open about this bug with Mathworks, feel free to forward them my e-mails from this list, and keep us posted if that leads to anything. Prentice On 02/09/2012 03:32 PM, Murphy, Brian (E IT F PR ORL 2) wrote: > Prentice, > > It worked like a charm. > > From my user: > > "I added the pause(16) line to the top of destroyJobFcn.m and reran the MDCS > validation. It worked as it should... no errored SGE jobs. I will need to > look further into the length of the pause or think up another, more > deterministic approach to adding the delay." > > Thanks again! > > --murph > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Murphy, Brian (E IT F PR ORL 2) > Sent: Thursday, February 09, 2012 1:30 PM > To: [email protected] > Subject: Re: [gridengine users] Another MATLAB + SGE Question > > Prentice, > > Thank you. I see what you are saying. I will have the users test this. > > --murph > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Prentice Bisbal > Sent: Wednesday, February 08, 2012 4:11 PM > To: [email protected] > Subject: Re: [gridengine users] Another MATLAB + SGE Question > > Murph, > > I did more testing this afternoon. I'm not sure I'd call this problem a > race condition. More like a communications error, that behaves like a > race condition. > > Here's my latest understanding of what's going on (again posted to list > for others who might search the archives ) > > When MATLAB calls qsub, it uses the -o switch to redirect STDOUT to a > file in the Job directory named <JobID>.mpiexec.log. For Job1, this > would be > > Job1/Job1.mpiexec.log > > As soon as the job itself completes, MATLAB thinks it's done completely, > and if you called the destroy function, it will delete the job > directory. Now in reality, after the job itself completes, SGE still has > some housekeeping to do, like collecting the STDOUT from the job and > writing it to Job1/Job1.mpiexec.log. So that's exactly what SGE tries to > do, but it finds that file is no longer there, and panics, reporting an > error with the job. > > To fix this, I added a pause before the destroy, like this: > > pjob = createParallelJob(sched); > set(pjob, 'MinimumNumberOfWorkers', 1); > set(pjob, 'MaximumNumberOfWorkers', 16); > createTask(pjob, @sum, 1, {[1 2 3]}); > submit(pjob); > waitForState(pjob, 'finished'); > results = getAllOutputArguments(pjob); > disp(results); > pause(16); > destroy(pjob); > > This works, but keeps the job running a bit longer. I initially put the > pause right after waitForState(), but then moved it since I decided that > users copying my code would what to get their results ASAP, and wouldn't > care if there was a pause before calling destroy. From testing, I've > found that the pause should be about 1 second per process on my cluster. > So, for a job with 4 workers, it should pause for 4 seconds. Since I > have a 16 worker license, I set my pause duration to 16 seconds. > > I say this is a communications error and not a race condition because of > getJobStateFcn.m did it's job correctly, waitForState would block until > the job no longer shows up in the output of 'qstat -xml', but it's > clearly unblocking before then. > > Prentice > > On 02/08/2012 11:31 AM, Murphy, Brian (E IT F PR ORL 2) wrote: > > >> Prentice, >> >> No worries. I will post back here when (if) we have a solution. >> >> --murph >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On >> Behalf Of Prentice Bisbal >> Sent: Wednesday, February 08, 2012 11:27 AM >> To: [email protected] >> Subject: Re: [gridengine users] Another MATLAB + SGE Question >> >> Murph, >> >> Thanks for letting me know I'm not the only one! >> >> Prentice >> >> >> On 02/08/2012 11:19 AM, Murphy, Brian (E IT F PR ORL 2) wrote: >>> Prentice, >>> >>> From what we have determined, this is a race condition. We have >>> experienced the same problem for months. Some of my users are currently >>> working with Mathworks to find a solution. >>> >>> --murph >>> >>> Siemens Energy, Inc >>> Orlando, FL >>> >>> -----Original Message----- >>> From: [email protected] [mailto:[email protected]] On >>> Behalf Of Prentice Bisbal >>> Sent: Wednesday, February 08, 2012 10:37 AM >>> To: [email protected] Users >>> Subject: [gridengine users] Another MATLAB + SGE Question >>> >>> So I finally have MATLAB set up and working fine with SGE. I can submit >>> parallel and distributed jobs from MATLAB to SGE, and then SGE does its >>> thing. >>> I have one remaining problem, and I thought I'd ask here first before >>> talking to Mathworks, since I have more confidence in you guys. I'm >>> afraid this might be very MATLAB specific, though. >>> >>> When I submit a parallel job, the job executes, and appears to complete >>> without any issues (MATLAB prints the results), but then the running job >>> goes from status 'r' to 'Eqw'. Using 'qstat -j <jobid> -E explain' shows >>> this error: >>> >>> error reason 1: 02/08/2012 10:28:58 [103808:6171]: error: >>> can't open output file "/work/prentice/matlab/Job1/Job1.mp >>> >>> My working directory is /work/prentice/matlab, and the Job1 directory is >>> temporary directory used my MATLAB when the job is running. >>> >>> In my MATLAB submit script, it call destroy() to kill the job after it >>> gets the results, and this destroy function deletes the Job1 subdir >>> while SGE is still running leading to this result. If I don't call >>> destroy(), MATLAB leaves a bunch of files and dirs in my working >>> directory related to Job1, Job2, etc. >>> >>> Has anyone else seen this with MATLAB+SGE? I suspect I might need to >>> modify the destroyJobFcn.m file to not kill the jobs while they're still >>> running. >>> >> _______________________________________________ >> users mailing list >> [email protected] >> https://gridengine.org/mailman/listinfo/users >> >> _______________________________________________ >> users mailing list >> [email protected] >> https://gridengine.org/mailman/listinfo/users >> > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users > > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users > > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users > _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
