Amila, Condor is a middleware to provision HTC resources. Following slides can provide basic information.
http://research.cs.wisc.edu/htcondor/tutorials/condor-g-dagman-talk.ppt Yan, If i understood it right, collecting batch job and parameter-sweep information to construct a DAG is a good idea and its a good use-case for airavata. Its going to be interesting for managing large number of jobs running in parallel. Thanks Raminder On Mar 2, 2013, at 9:56 PM, Amila Jayasekara wrote: > Hi Yan, > > This is a great idea. Please some inline comments below. > > Thanks > Amila > > On Sat, Mar 2, 2013 at 5:43 PM, Yan Liu <[email protected]> wrote: >> High-throughput computing (HTC) resources are available from national >> cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC >> resources are suitable for running serial or embarrassingly parallel user >> jobs. Unlike high-performance computing (HPC) resources, HTC environment is >> more distributed, loosely coupled, and managed individually by various >> resource contributors. For HTC users, a HTC environment presents a virtual >> resource pool with dynamically aggregated resources and can be accessed >> through a unified client software (e.g., OSG/VDT/Condor). >> >> Integrating HTC capabilities in Airavata is important for users to access >> HTC resources seamlessly as they access other kinds of computing >> environments supported by Airavata. At first glance, the integration may be >> straightforward by adding middleware support (e.g., Condor or BOSCO) into >> Airavata. However, I am proposing a user-oriented approach to the >> integration in order to fully leverage HTC client software's capabilities. >> >> An Airavata user does not care the underlying middleware when she/he >> composes a job, ideally. What the user cares is the computational capability >> provided by the underlying resources. A HTC environment, with the support >> from the Condor middleware, is desirable for running: >> - large batch jobs >> - parameter-sweeping jobs >> - stochastic jobs with the same configuration but requiring a large number >> of repeated runs in order to obtain statistically confident results >> - workflow jobs that can be represented as DAG (directed acyclic graph) > > I am +1 for this idea. So how do you precisely define a HTC > environment ? (In other words what parameters does user needs to > specify when configuring a HTC environment ?) > I was kind of thinking along the same path. I agree with you that user > does not need to care about underlying middleware where his/her job > should run. So Airavata should be able to figure out an appropriate > place (machine) to run the job based on provided configurations. I > think that configuration should be what you proposed as "HTC > environment". > > I am also curious to know whether you have thought about how > middleware related security (authentication, proxy certificates etc > ...) should be handle in this scenario. (My knowledge about Condor is > limited) > > Further I didnt quite understand how Condor middleware enables you to > run "workflow jobs that can be represented as DAG". Appreciate your > explanation on this. > > Also can Condor work with providers such as EC2 ? > >> >> Therefore, instead of presenting a raw Condor interface to Airavata users, >> tailored interfaces to aforementioned user job types will be more useful. >> Technically, Condor submmit script syntax supports all of the described jobs >> through job macros and DAG support. If Airavata can bridge user job >> requirements and the composition of the technical Condor submission script, >> HTC resources can be more effectively represented for and used by Airavata >> community. >> >> The development roadmap is upon Airavata team's design, I'm willing to >> contribute a disease mapping application for the testing and evaluation of >> the new components and capabilities developed in Airavata for this purpose. >> >> Thanks, >> Yan
