I've narrowed down the problem, and this is probably something I should
take up with Mathworks at this point. I'm sharing the info here for
future googlers.
When I'm done getting results from my job, I call the destry()function
in MATLAB:
% Add a task to the job. In this example, I'm calling MATLAB's Sum
% function to add 1+1. Replace @sum with a name your MATLAB program
createTask(pjob, @sum, 1, {[1 2 3]});
%createTask(pjob, @test_parfor, 1, {});
% Run the job
submit(pjob);
% Wait for the job to finish running, and retrieve the results
waitForState(pjob, 'finished');
results = getAllOutputArguments(pjob);
% Display the results
disp(results);
% Destroy parallel job
destroy(pjob);
If I comment out that destroy() statement, the jobs don't end up in an
Eqw state in SGE - they complete without error. I think the problem lies
in the waitForState function.
The waitForState function, is supposed to block the job until it's
finished. It determines the job is finished by looking at the output of
'qstat -xml'. If it can't find any jobs with the Job ID in that output,
it assumes the job is finished. If it can't determine the jobs status
from the scheduler, it uses MATLAB's job state, which, I assume, is kept
in the "Job1", "Job2", etc. files and subdirs it creates when running
the job. All this magic happens in a file called getJobStateFcn.m, which
Mathworks provides an SGE-specific version of.
I suspect MATLAB isn't getting the results it wants from SGE, so it's
falling back to it's own mechanisms for determining job state, declaring
the job finished when qstat shows that it's still in state 'r'. The
destroy function then deletes the 'Job] files and subdirs prematurely,
which is why the job exits prematurely with the error:
can't open output file "/work/prentice/matlab/Job1/Job1.mp
The Job1 file and dir have already been deleted directly by MATLAB.
I can safely omit the destroy function, but then it leaves all those Job# files
and subdirs behind, which accumulate very quickly.
Since this is a MATLAB problem, and not an SGE problem, I won't make anymore
noise with this issue here.
--
Prentice
On 02/08/2012 10:37 AM, Prentice Bisbal wrote:
> So I finally have MATLAB set up and working fine with SGE. I can submit
> parallel and distributed jobs from MATLAB to SGE, and then SGE does its
> thing.
> I have one remaining problem, and I thought I'd ask here first before
> talking to Mathworks, since I have more confidence in you guys. I'm
> afraid this might be very MATLAB specific, though.
>
> When I submit a parallel job, the job executes, and appears to complete
> without any issues (MATLAB prints the results), but then the running job
> goes from status 'r' to 'Eqw'. Using 'qstat -j <jobid> -E explain' shows
> this error:
>
> error reason 1: 02/08/2012 10:28:58 [103808:6171]: error:
> can't open output file "/work/prentice/matlab/Job1/Job1.mp
>
> My working directory is /work/prentice/matlab, and the Job1 directory is
> temporary directory used my MATLAB when the job is running.
>
> In my MATLAB submit script, it call destroy() to kill the job after it
> gets the results, and this destroy function deletes the Job1 subdir
> while SGE is still running leading to this result. If I don't call
> destroy(), MATLAB leaves a bunch of files and dirs in my working
> directory related to Job1, Job2, etc.
>
> Has anyone else seen this with MATLAB+SGE? I suspect I might need to
> modify the destroyJobFcn.m file to not kill the jobs while they're still
> running.
>
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users