We can not expect users or applications to change behavior for Airavata. It is up to us enable applications and users as they are now. As you have seen several applications have system input parameters inside a master input file and they are used by the application and required to be used in scheduling. As I was suggesting the memory for scheduling should be higher than what is expected by the application. Similarly time for scheduling should also be higher than what is given in the input to accommodate cleanup and other post processing as well. Some schedulers allow soft and hard limits (admins may or may not enable them) and we can think of these pairs of system parameters as soft and hard memory and time limits.
Thanks, Sudhakar. On Dec 11, 2014, at 2:02 AM, Shameera Rathnayaka <shameerai...@gmail.com<mailto:shameerai...@gmail.com>> wrote: Hi Amila, According to my understanding, what this handle does is, read the user given configuration at run time. I have no idea this will effect to qsub or aprun or other parameter. It is better if someone explain it to me too. We already have a way to provide the these configuration parameters with the experiment itself by defining ComputeResourceScheduling. But there are some use cases like gaussian, where gaussian users provide these configurations with the input file it self. IMO here we have two options, either we can ask gaussian users to adopt to the airavata way but still those configuration in input file is required for gaussian application(I guess , correct me if I am wrong here) or use airavata extension points to support this scenario. Here the handler address the second option. Thanks, Shameera. On Thu, Dec 11, 2014 at 12:00 AM, Amila Jayasekara <thejaka.am...@gmail.com<mailto:thejaka.am...@gmail.com>> wrote: Also, regarding the handler that Shameera is working on ... I guess that handler is going to change mainly "qsub" parameters or "aprun" parameters (Correct me if i am wrong). I think it would be more useful to write a handler which changes any parameter in "qsub", "aprun" or "mpiexec". In implementation wise I would imagine there is an abstract handler with concrete implementation for each job scheduling command. Thanks -Amila On Wed, Dec 10, 2014 at 9:17 AM, Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>> wrote: +1 for more generalization. We are collecting more raw material for chemistry application use cases at https://cwiki.apache.org/confluence/display/AIRAVATA/Use+Cases. We'll review them (and bio apps that we also collected previously) in a wiki document to see if our API mappings are correct. Preliminarily, we see the command line arguments don't contain the full list of input and output files. Additional required inputs may be passed via control files, environment variables, etc. Examples include data libraries for basis functions, names of checkpoint files, names of output files, and so forth. So we need a way to say the application may take 4 inputs, but only 1 is needed to construct a valid command line, for example. On the other hand, I don't think we need the InputMetadataType that Chathuri introduces below. This overlaps with what is already in the compute resource description fields. Marlon On 12/8/14, 10:17 PM, Amila Jayasekara wrote: Hi Chathuri, I do not know anything about Gaussian. So its kind of hard for me to understand what exactly is the meaning of the structures you introduced and why you exactly need those structures. A more important question is how to come up with a more abstract and generic thrift IDLS so that you dont need to change it every time we add a new application. Going through many example applications is certainly a good way to understand broad requirements and helps to abstract out many features. Thanks -Thejaka On Mon, Dec 8, 2014 at 10:22 AM, Chathuri Wimalasena <kamalas...@gmail.com<mailto:kamalas...@gmail.com>> wrote: Hi Devs, We are trying to add Gaussian application using airavata-appcatalog. While doing that, we face some limitations of the current design. In Gaussian there are several input files, some input files should used when the job run command is generated, but some does not. Those which are not involved with job run command also need to be staged to working directory. Such flags are not supported in current design. Another interesting feature that in Gaussian is, in input file, we can specify the values for memory, cpu like options. If input file includes those parameters, we need to give priority to those values instead of the values specified in the request. To support these features, we need to slightly modify our thrift IDLS, specially to InputDataObjectType struct. Current struct is below. struct InputDataObjectType { 1: required string name, 2: optional string value, 3: optional DataType type, 4: optional string applicationArgument, 5: optional bool standardInput = 0, 6: optional string userFriendlyDescription, 7: optional string metaData } In order to support 1st requirement, we introduce 2 enums. enum InputValidityType{ REQUIRED, OPTIONAL } enum CommandLineType{ INCLUSIVE, EXCLUSIVE } Please excuse me for names. You are welcome to suggest better names. To support 2nd requirement, we change metaData field to a map with another enum where we define all the metadata types that can have. enum InputMetadataType { MEMORY, CPU } So the new InputDataObjectType would be as below. struct InputDataObjectType { 1: required string name, 2: optional string value, 3: optional DataType type, 4: optional string applicationArgument, 5: optional bool standardInput = 0, 6: optional string userFriendlyDescription, * 7: optional map<InputMetadataType, string> metaData,* * 8: optional InputValidityType inputValid;* * 9: optional CommandLineType addedToCommandLine;* * 10: optional bool dataStaged = 0;* } Suggestions are welcome. Thanks, Chathuri -- Best Regards, Shameera Rathnayaka. email: shameera AT apache.org<http://apache.org/> , shameerainfo AT gmail.com<http://gmail.com/> Blog : http://shameerarathnayaka.blogspot.com/
