Hi Ansink,
Yes, this is a known problem with Alchemi.
The issue is, currently Alchemi uses only remoting to transfer the files
as method "arguments". This causes these files to be in memory, and
eventually there are scalability problems.
We are trying to fix the problem by having a seperate data transfer
service, so that large-ish input files dont cause a problem.
Till then, I guess this is known "bug".
Regards
Krishna.
Krishna,
I have a problem using Alchemi and i try to explain my problem. Look at the
following code
GConnection gc = GConnection.FromConsole(hostname, "9000", "admin",
"admin");
// create a new grid application
_gridApplicatie = new GApplication(gc);
// subcribe to events
_gridApplicatie.ThreadFinish += new GThreadFinish(JobFinished);
_gridApplicatie.ThreadFailed += new GThreadFailed(JobFailed);
_gridApplicatie.ApplicationFinish += new
GApplicationFinish(ApplicationFinished);
Initialize(programProfile);
// Create jobs
if (Directory.Exists(_map))
{
string [] calcs = Directory.GetFiles(_map);
// Loop over the jobs
for (int jobNr = 0; jobNr < calcs.Length; jobNr++)
{
// Create a new Alchemi job
GJob alchemiJob = new GJob();
// Add Input files to the alchemiJob
for (int i = 0; i < _inputFiles.Length; i++)
{
alchemiJob.InputFiles.Add(new
EmbeddedFileDependency(Path.GetFileName(_inputFiles[i]), _inputFiles[i]));
}
// Add Output files to the alchemiJob
for (int i = 0; i < _outputFiles.Length; i++)
{
alchemiJob.OutputFiles.Add(new
EmbeddedFileDependency(_outputFiles[i]));
}
// Run command
alchemiJob.RunCommand = string.Format("{0} {1} ",
Path.GetFileName(_executable),
Path.GetFileName(programProfile.Name));
// Add job
_gridApplicatie.Threads.Add(alchemiJob);
}
// start the grid application
_gridApplicatie.Start();
Console.WriteLine("\nStarted Sobek grid application .. Waiting for jobs to
finish ..\n");
The input files that i add for my job as a embeddedfiledependency is about 20 mb of data.
So when i want to run a calculation of more than a dozen of jobs the memory allocated is too much, because all of the files are in memory until the start of the grid application.
Is this a known problem or am i doing someting wrong. I can run my grid application for only +- 12 jobs fine. Is it possible to read & encrypt the input files at a later moment??
Johan Ansink
senior programmer
HKV consultants.
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
alchemi-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/alchemi-users
