Nuno Guerreiro wrote:
Thank you for you comments Florian :)
I'm still trying to find out how to connect all the pieces, and most
important of all, which pieces should I choose. From what I've read,
in theory, I could use Condor-G to submit a job to one of several
Globus Gatekeepers, which would then submit that job to a local
scheduler like PBS or Condor. This local scheduler would choose a
machine, execute the job and send the results back to me.
Hello,
This works in practice, too.
I don't know if this is a realistic scenario but, here it goes:
- Using a data mining toolkit like Weka, I would develop an
application that performs some data analysis.
- I would submit that application, plus the Weka toolkit, plus the
data set to process to Condor-G.
- Condor-G would then choose a Globus Gatekeeper and forward my request/job.
- The Globus Gatekeeper would then forward my request/job to a local scheduler.
- The local scheduler would execute my application, using the Weka
toolkit, to process the data set.
- The results would be sent by the local scheduler to Globus which
would then forward them to Condor-G, making them available to me.
Can this be a feasible option?
I don't know much about the Weka toolkit, but it should work as
described, unless 1) it requires MPI or 2) cannot be statically compiled
and you can't ship the required dynamic libraries to your target site or
3) it's so huge that uploading it with each job is shockingly
inefficient. In the last case, you may consider using different kinds of
jobs with dependencies between them, see Condor DAGMan. Just keep in
mind that the GT4 host must be able to connect back per TCP to your
Condor-G client - this might cause trouble for firewalls. Also, you
should be prepared to implement some fault tolerance (retry) if the jobs
fail for inexplicable reasons (as Grid jobs like to do).
Is Condor-G able to use a service like the Globus Index Service and
choose a gatekeeper?
Condor-G has no built-in support for querying MDS or anything like that.
To achieve the intended setup, write a script which polls whatever
directory services you want to query and submits the appropriate
resource descriptions (ClassAds) to Condor. You can then reference any
attribute from such a custom resource description in your job
requirements. Maybe some prebuilt adapters exist for MDS, but I don't
know of any. Read Section 5.3.7 "Matchmaking in the Grid Universe" of
the Condor's manual for details.
One more thing, if your jobs can run or sit in the queue for a long time
(say >24h), you will need to throw MyProxy into the mix and use
Condor-G's proxy certificate refresh options.
Here is a set of tutorials to guide you:
https://bi.offis.de/wisent/tiki-index.php?page=Condor-GT4
Regards,
Jan Ploski
